Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-11-22 10:15:35 +01:00
Code Issues Projects Releases Wiki Activity

update kubernetes deployment

Browse Source
This commit is contained in:
Kira 2016-11-21 16:07:04 +08:00
parent e1b2ba0e23
commit 00259567a8
30 changed files with 1083 additions and 228 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

187
docs/kubernetes_deployment.md
View File

@ -1,65 +1,136 @@
## makeing Harbor on Kubernetes
To deploy Harbor on Kubernetes, it requires some additional steps because
1. When Harbor registry uses https, so we need cert or workaround to avoid errors like this:
```
Error response from daemon: invalid registry endpoint https://{HOST}/v0/: unable to ping registry endpoint https://{HOST}/v0/
v2 ping attempt failed with error: Get https://{HOST}/v2/: EOF
v1 ping attempt failed with error: Get https://{HOST}/v1/_ping: EOF. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry {HOST}` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/{HOST}/ca.crt
```
There is a workaround if you don't have a cert. The workaround is to add the host into the list of insecure registry by editting the ```/etc/default/docker``` file:
```
sudo vi /etc/default/docker
```
add the line at the end of file:
```
DOCKER_OPTS="$DOCKER_OPTS --insecure-registry={HOST}"
```
restart docker service
```
sudo service docker restart
```
2. The registry config file needs to have the IP (or DNS name) of the registry, but on Kubernetes, you don't know the IP before the service is created. There are several workarounds to solve this problem for now:
- Use DNS name and link the DNS name with the IP after the service is created.
- Rebuild the registry image with the service IP after the service is created and use ```kubectl rolling-update``` to update to the new image.
To start Harbor on Kubernetes, you first need to build the docker images. The docker images for deploying Harbor on Kubernetes depends on the docker images to deploy Harbor with docker-compose. So the first step is to build docker images with docker-compose. Before actually building the images, you need to first adjust the [configuration](https://github.com/vmware/harbor/blob/master/make/harbor.cfg):
- Change the [hostname](https://github.com/vmware/harbor/blob/master/make/harbor.cfg#L5) to ```localhost```
- Adjust the [email settings](https://github.com/vmware/harbor/blob/master/make/harbor.cfg#L11) according to your needs.
## Integration with Kubernetes
This Document decribes how to deploy Harbor on Kubernetes.
### Prerequisite
* You need to build docker images of Harbor. See [Installation Guide](./installation_guide.md)
* You should have domain knowledge about Kubernetes (Replication Controller, Service, Persistent Volume, Persistent Volume Claim, Config Map).
### Configuration
We provide a python script `make/kubernetes/prepare` to generate Kubernetes ConfigMap files.
The script is written in python, so you need a version of python in your deployment environment.
Also the script need `openssl` to generate private key and certification, make sure you have a workable `openssl`.
There are some args of the python script:
- -f: Default Value is `../harbor.cfg`. You can specify other config file of Harbor.
- -k: Path to https private key. This arg can overwrite the value of `ssl_cert_key` in `harbor.cfg`.
- -c: Path to https certification. This arg can overwrite the value of `ssl_cert` in `harbor.cfg`.
- -s: Path to secret key. Must be 16 characters. If you don't set it, the script will generate it automatically.
#### Basic Configuration
These Basic Configuration must be set. Otherwise you can't deploy Harbor on Kubernetes.
- `make/harbor.cfg`: Basic config of Harbor. Please refer to `harbor.cfg`.
- `make/kubernetes/**/*.rc.yaml`: Specify configs of containers.
You need to specify the path to your images in all `*.rc.yaml`. example:
```
containers:
- name: nginx-app
# it's very importent that you need modify the path of image.
image: harbor/nginx
```
- `make/kubernetes/pv/*.pvc.yaml`: Persistent Volume Claim.
You can set capacity of storage in these files. example:
```
resources:
requests:
# you can set another value to adapt to your needs
storage: 100Gi
```
- `make/kubernetes/pv/*.pv.yaml`: Persistent Volume. Be bound with `*.pvc.yaml`.
PVs and PVCs are one to one correspondence. If you changed capacity of PVC, you need to set capacity of PV together.
example:
```
capacity:
# same value with PVC
storage: 100Gi
```
In PV, you should set another way to store data rather than `hostPath`:
```
# it's default value, you should use others like nfs.
hostPath:
path: /data/registry
```
For more infomation about store ways, Please check [Kubernetes Document](http://kubernetes.io/docs/user-guide/persistent-volumes/)
Then you can generate ConfigMap files by :
Then you can run the following commends to build docker images:
```
cd make
./prepare
docker-compose build
docker build -f kubernetes/dockerfiles/proxy-dockerfile -t {your_account}/proxy .
docker build -f kubernetes/dockerfiles/registry-dockerfile -t {your_account}/registry .
docker build -f kubernetes/dockerfiles/ui-dockerfile -t {your_account}/deploy_ui .
docker tag deploy_mysql {your_account}/deploy_mysql
docker push {your_account}/proxy
docker push {your_account}/registry
docker push {your_account}/deploy_ui
docker push {your_account}/deploy_mysql
```
where "your_account" is your own registry. Then you need to update the "image" field in the ```*-rc.yaml``` files at:
```
make/kubernetes/mysql-rc.yaml
make/kubernetes/proxy-rc.yaml
make/kubernetes/registry-rc.yaml
make/kubernetes/ui-rc.yaml
python make/kubernetes/prepare
```
Further more, the following configuration could be changed according to your need:
- **harbor_admin_password**: The password for the administrator of Harbor, by default the password is Harbor12345. You can changed it [here](https://github.com/vmware/harbor/blob/master/make/kubernetes/ui-rc.yaml#L36).
- **auth_mode**: The authentication mode of Harbor. By default it is *db_auth*, i.e. the credentials are stored in a database. Please set it to *ldap_auth* if you want to verify user's credentials against an LDAP server. You can change the configuration [here](https://github.com/vmware/harbor/blob/master/make/kubernetes/ui-rc.yaml#L40).
- **ldap_url**: The URL for LDAP endpoint, for example ldaps://ldap.mydomain.com. It is only used when **auth_mode** is set to *ldap_auth*. It could be changed [here](https://github.com/vmware/harbor/blob/master/make/kubernetes/ui-rc.yaml#L42).
- **ldap_basedn**: The basedn template for verifying the user's credentials against LDAP, for example uid=%s,ou=people,dc=mydomain,dc=com. It is only used when **auth_mode** is set to *ldap_auth*. It could be changed [here](https://github.com/vmware/harbor/blob/master/make/kubernetes/ui-rc.yaml#L44).
- **db_password**: The password of root user of mySQL database. Change this password for any production use. You need to change both [here](https://github.com/vmware/harbor/blob/master/make/kubernetes/ui-rc.yaml#L28) and [here](https://github.com/vmware/harbor/blob/master/make/harbor.cfg#L32) to make the change. Please note, you need to change the ```harbor.cfg``` before building the docker images.
These files will be generated:
- make/kubernetes/jobservice/jobservice.cm.yaml
- make/kubernetes/mysql/mysql.cm.yaml
- make/kubernetes/nginx/nginx.cm.yaml
- make/kubernetes/registry/registry.cm.yaml
- make/kubernetes/ui/ui.cm.yaml
Finally you can start the jobs by running:
```
kubectl create -f make/kubernetes
```
#### Advanced Configuration
If Basic Configuration was not covering your requirements, you can read this section for more details.
`./prepare` has a specify format of placeholder:
- `{{key}}`: It means we should replace the placeholder with the value in `config.cfg` which name is `key`.
- `{{num key}}`: It's used for multiple lines text. It will add `num` spaces to the leading of every line in text.
You can find all configs of Harbor in `make/kubernetes/templates/`. There are specifications of these files:
- `jobservice.cm.yaml`: ENV and web config of jobservice
- `mysql.cm.yaml`: Root passowrd of MySQL
- `nginx.cm.yaml`: Https certification and nginx config. If you are fimiliar with nginx, you can modify it.
- `registry.cm.yaml`: Token service certification and registry config
Registry use filesystem to store data of images. You can find it like:
```
storage:
filesystem:
rootdirectory: /storage
```
If you want use another storage backend, please see [Docker Doc](https://docs.docker.com/datacenter/dtr/2.1/guides/configure/configure-storage/)
- `ui.cm.yaml`: Token service private key, ENV and web config of ui
`ui` and `jobservice` are powered by beego. If you are fimiliar with beego, you can modify configs in `jobservice.cm.yaml` and `ui.cm.yaml`.
### Running
When you finished your configuring and generated ConfigMap files, you can run Harbor on kubernetes with these commands:
```
# create pv & pvc
kubectl apply -f make/kubernetes/pv/log.pv.yaml
kubectl apply -f make/kubernetes/pv/registry.pv.yaml
kubectl apply -f make/kubernetes/pv/storage.pv.yaml
kubectl apply -f make/kubernetes/pv/log.pvc.yaml
kubectl apply -f make/kubernetes/pv/registry.pvc.yaml
kubectl apply -f make/kubernetes/pv/storage.pvc.yaml
# create config map
kubectl apply -f make/kubernetes/jobservice/jobservice.cm.yaml
kubectl apply -f make/kubernetes/mysql/mysql.cm.yaml
kubectl apply -f make/kubernetes/nginx/nginx.cm.yaml
kubectl apply -f make/kubernetes/registry/registry.cm.yaml
kubectl apply -f make/kubernetes/ui/ui.cm.yaml
# create service
kubectl apply -f make/kubernetes/jobservice/jobservice.svc.yaml
kubectl apply -f make/kubernetes/mysql/mysql.svc.yaml
kubectl apply -f make/kubernetes/nginx/nginx.svc.yaml
kubectl apply -f make/kubernetes/registry/registry.svc.yaml
kubectl apply -f make/kubernetes/ui/ui.svc.yaml
# create k8s rc
kubectl apply -f make/kubernetes/registry/registry.rc.yaml
kubectl apply -f make/kubernetes/mysql/mysql.rc.yaml
kubectl apply -f make/kubernetes/jobservice/jobservice.rc.yaml
kubectl apply -f make/kubernetes/ui/ui.rc.yaml
kubectl apply -f make/kubernetes/nginx/nginx.rc.yaml
```

3
make/kubernetes/dockerfiles/proxy-dockerfile
View File

@ -1,3 +0,0 @@
FROM library/nginx:1.11.5
ADD ./config/nginx /etc/nginx

33
make/kubernetes/dockerfiles/registry-config.yml
View File

@ -1,33 +0,0 @@
version: 0.1
log:
level: debug
fields:
service: registry
storage:
cache:
layerinfo: inmemory
filesystem:
rootdirectory: /storage
maintenance:
uploadpurging:
enabled: false
http:
addr: :5000
secret: placeholder
debug:
addr: localhost:5001
auth:
token:
issuer: registry-token-issuer
realm: http://harbor.caicloud.io/service/token
rootcertbundle: /etc/registry/root.crt
service: token-service
notifications:
endpoints:
- name: harbor
disabled: false
url: http://harbor.caicloud.io/service/notifications
timeout: 500
threshold: 5
backoff: 1000

6
make/kubernetes/dockerfiles/registry-dockerfile
View File

@ -1,6 +0,0 @@
FROM library/registry:2.3.0
ADD ./config/registry/ /etc/registry/
ADD ./kubernetes/dockerfiles/registry-config.yml /etc/registry/config.yml
CMD ["/etc/registry/config.yml"]

4
make/kubernetes/dockerfiles/ui-dockerfile
View File

@ -1,4 +0,0 @@
FROM deploy_ui
ADD ./config/ui/app.conf /etc/ui/app.conf
ADD ./config/ui/private_key.pem /etc/ui/private_key.pem

112
make/kubernetes/jobservice/jobservice.rc.yaml Normal file
View File

@ -0,0 +1,112 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: jobservice-rc
labels:
name: jobservice-rc
spec:
replicas: 1
selector:
name: jobservice-apps
template:
metadata:
labels:
name: jobservice-apps
spec:
containers:
- name: jobservice-app
image: harbor/jobservice
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_HOST
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_HOST
- name: MYSQL_PORT
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_PORT
- name: MYSQL_USR
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_USR
- name: MYSQL_PWD
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_PWD
- name: UI_SECRET
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: UI_SECRET
- name: SECRET_KEY
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: SECRET_KEY
- name: CONFIG_PATH
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: CONFIG_PATH
- name: REGISTRY_URL
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: REGISTRY_URL
- name: VERIFY_REMOTE_CERT
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: VERIFY_REMOTE_CERT
- name: MAX_JOB_WORKERS
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MAX_JOB_WORKERS
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: LOG_LEVEL
- name: LOG_DIR
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: LOG_DIR
- name: GODEBUG
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: GODEBUG
- name: EXT_ENDPOINT
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: EXT_ENDPOINT
- name: TOKEN_URL
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: TOKEN_URL
ports:
- containerPort: 80
volumeMounts:
- name: config
mountPath: /etc/jobservice
- name: logs
mountPath: /var/log/jobs
volumes:
- name: config
configMap:
name: harbor-jobservice-config
items:
- key: config
path: app.conf
- name: logs
persistentVolumeClaim:
claimName: log-pvc

9
make/kubernetes/jobservice/jobservice.svc.yaml Normal file
View File

@ -0,0 +1,9 @@
apiVersion: v1
kind: Service
metadata:
name: jobservice
spec:
ports:
- port: 80
selector:
name: jobservice-apps

22
make/kubernetes/mysql-rc.yaml → make/kubernetes/mysql/mysql.rc.yaml
View File

@ -1,30 +1,34 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
name: mysql-rc
labels:
name: mysql
name: mysql-rc
spec:
replicas: 1
selector:
name: mysql
name: mysql-apps
template:
metadata:
labels:
name: mysql
name: mysql-apps
spec:
containers:
- name: mysql
image: caicloud/harbor_deploy_mysql:latest
imagePullPolicy: Always
- name: mysql-app
image: harbor/mysql
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: root123
valueFrom:
configMapKeyRef:
name: harbor-mysql-config
key: MYSQL_ROOT_PASSWORD
volumeMounts:
- name: mysql-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-storage
emptyDir: {}
persistentVolumeClaim:
claimName: storage-pvc

4
make/kubernetes/mysql-svc.yaml → make/kubernetes/mysql/mysql.svc.yaml
View File

@ -2,10 +2,8 @@ apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
name: mysql
spec:
ports:
- port: 3306
selector:
name: mysql
name: mysql-apps

36
make/kubernetes/nginx/nginx.rc.yaml Normal file
View File

@ -0,0 +1,36 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-rc
labels:
name: nginx-rc
spec:
replicas: 1
selector:
name: nginx-apps
template:
metadata:
labels:
name: nginx-apps
spec:
containers:
- name: nginx-app
image: harbor/nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
- containerPort: 443
volumeMounts:
- name: config
mountPath: /etc/nginx
volumes:
- name: config
configMap:
name: harbor-nginx-config
items:
- key: config
path: nginx.conf
- key: pkey
path: https.key
- key: cert
path: https.crt

7
make/kubernetes/proxy-svc.yaml → make/kubernetes/nginx/nginx.svc.yaml
View File

@ -1,15 +1,12 @@
apiVersion: v1
kind: Service
metadata:
name: proxy
labels:
name: proxy
name: nginx
spec:
type: LoadBalancer
ports:
- name: http
port: 80
- name: https
port: 443
selector:
name: proxy
name: nginx-apps

201
make/kubernetes/prepare Normal file
View File

@ -0,0 +1,201 @@
#!/usr/bin/env python
from __future__ import print_function, unicode_literals # We require Python 2.6 or later
import sys
import argparse
import io
import os
import random
import re
import string
import subprocess
if sys.version_info[:3][0] == 2:
import ConfigParser as configparser
import StringIO as io
if sys.version_info[:3][0] == 3:
import configparser as configparser
import io as io
# prepare base dir
base_dir = os.path.dirname(os.path.abspath(__file__))
parser = argparse.ArgumentParser(description='Generate *.cm.yaml')
parser.add_argument('-f', default=os.path.join(base_dir, '../harbor.cfg'),
dest='config_file', help='[Optional] path of harbor config file')
parser.add_argument('-k', default='',
dest='private_key', help='[Optional] path of harbor https private key(pem)')
parser.add_argument('-c', default='',
dest='cert', help='[Optional] harbor path of https cert(pem)')
parser.add_argument('-s', default='',
dest='secret_key', help="[Optional] path of harbor secret key(16 characters)")
args = parser.parse_args()
# read config file
config_str = ''
if os.path.isfile(args.config_file):
with open(args.config_file) as conf:
config_str = conf.read()
else:
raise Exception('Error: No such file(' + args.config_file + ')')
config_str = '[harbor]\n' + config_str
fp = io.StringIO()
fp.write(config_str)
fp.seek(0, os.SEEK_SET)
config = configparser.RawConfigParser()
config.readfp(fp)
def get_config(key):
"""get value by key
"""
if config.has_option('harbor', key):
return config.get('harbor', key)
print('Warning: Key(' + key + ') is not existing. Use empty string as default')
return ''
def set_config(key, value):
"""set key & value
"""
config.set('harbor', key, value)
# relative path with config file
def rel_path(p):
if p[0] == '/':
return p
config_path = args.config_file
if config_path[0] != '/':
config_path = os.path.join(os.getcwd(), config_path)
return os.path.join(os.path.dirname(config_path), p)
# path of private key
pk_path = args.private_key
if pk_path == '':
pk_path = get_config('ssl_cert_key')
if pk_path != '':
pk_path = rel_path(pk_path)
# path of cert
cert_path = args.cert
if cert_path == '':
cert_path = get_config('ssl_cert')
if cert_path != '':
cert_path = rel_path(cert_path)
# validate
if get_config('ui_url_protocol') == 'https':
if pk_path == '':
raise Exception("Error: The protocol is https but attribute ssl_cert_key is not set")
if cert_path == '':
raise Exception("Error: The protocol is https but attribute ssl_cert is not set")
else:
pk_path = ''
cert_path = ''
# read secret key
if args.secret_key != '':
if os.path.isfile(args.secret_key):
key = ''
with open(args.secret_key, 'r') as skey:
key = skey.read()
if len(key) != 16:
raise Exception('Error: The length of secret key has to be 16 characters!')
set_config('secret_key', key)
else:
set_config('secret_key', ''.join(random.choice(
string.ascii_letters + string.digits) for i in range(16)))
# read https pkey & cert
if pk_path != '':
if os.path.isfile(pk_path):
with open(pk_path, 'r') as pkey:
set_config('https_pkey', pkey.read())
else:
raise Exception('Error: https private key is not existing')
else:
set_config('https_pkey', 'USE_HTTP')
if cert_path != '':
if os.path.isfile(cert_path):
with open(cert_path, 'r') as cert:
set_config('https_cert', cert.read())
else:
raise Exception('Error: https cert is not existing')
else:
set_config('https_cert', 'USE_HTTP')
# add configs
set_config('ui_url', get_config('ui_url_protocol') +
'://' + get_config('hostname'))
set_config('ui_secret', ''.join(random.choice(
string.ascii_letters + string.digits) for i in range(16)))
# generate auth pkey & cert
with open(os.devnull, 'w') as devnull:
openssl = subprocess.call(['which','openssl'], stdout=devnull, stderr=devnull)
if openssl == 0:
pkey = subprocess.check_output(['openssl','genrsa','4096'], stderr=devnull)
subj = '/C={0}/ST={1}/L={2}/O={3}/OU={4}/CN={5}/emailAddress={6}'.format(get_config('crt_country'),
get_config('crt_state'), get_config('crt_location'), get_config('crt_organization'),
get_config('crt_organizationalunit'), get_config('crt_commonname'), get_config('crt_email'))
openssl = subprocess.Popen(['openssl', 'req', '-new', '-x509', '-key', '/dev/stdin', '-days', '3650', '-subj', subj],
stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=devnull)
cert = openssl.communicate(input=pkey)[0]
set_config('auth_pkey', pkey.decode())
set_config('auth_cert', cert.decode())
else:
set_config('auth_pkey', 'NEED_SET')
set_config('auth_cert', 'NEED_SET')
print('Warning: auth_pkey and auth_cert cannot be generated automatically without openssl. Please set it manually')
variable = re.compile(r'{{.+?}}')
detail = re.compile(r'((\d+) )?([a-zA-Z_0-9-]+)')
def render_template(tmpl):
"""render template
replace {{(number of leading spaces)name}} with config
examples:
config:
hostname='test\ntest'
{{hostname}} -> 'test\ntest'
{{4 hostname}} -> 'test\n test'
"""
matches = variable.findall(tmpl)
for match in matches:
segs = detail.search(match)
if segs.group() == '':
raise Exception('Error: Invalid template item(' + match + ')')
value = get_config(segs.group(3))
spaces = segs.group(2)
if spaces != '' and spaces != None:
leading = ''.join(' ' for i in range(int(spaces)))
value = str(value).replace('\n', '\n' + leading)
tmpl = tmpl.replace(match, value)
return tmpl
def generate_template(tmpl, dest):
"""generate file
"""
with open(tmpl) as tmpl:
with open(dest, 'w') as dest:
dest.write(render_template(tmpl.read()))
template_dir = os.path.join(base_dir, 'templates')
output_dir = base_dir
generate_template(os.path.join(template_dir, 'ui.cm.yaml'), os.path.join(output_dir, 'ui/ui.cm.yaml'))
generate_template(os.path.join(template_dir, 'jobservice.cm.yaml'), os.path.join(output_dir, 'jobservice/jobservice.cm.yaml'))
generate_template(os.path.join(template_dir, 'mysql.cm.yaml'), os.path.join(output_dir, 'mysql/mysql.cm.yaml'))
generate_template(os.path.join(template_dir, 'nginx.cm.yaml'), os.path.join(output_dir, 'nginx/nginx.cm.yaml'))
generate_template(os.path.join(template_dir, 'registry.cm.yaml'), os.path.join(output_dir, 'registry/registry.cm.yaml'))

22
make/kubernetes/proxy-rc.yaml
View File

@ -1,22 +0,0 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: proxy
labels:
name: proxy
spec:
replicas: 1
selector:
name: proxy
template:
metadata:
labels:
name: proxy
spec:
containers:
- name: proxy
image: caicloud/harbor_proxy:latest
imagePullPolicy: Always
ports:
- containerPort: 80
- containerPort: 443

14
make/kubernetes/pv/log.pv.yaml Normal file
View File

@ -0,0 +1,14 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: log-pv
labels:
type: log
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/logs

13
make/kubernetes/pv/log.pvc.yaml Normal file
View File

@ -0,0 +1,13 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: log-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
selector:
matchLabels:
type: log

14
make/kubernetes/pv/registry.pv.yaml Normal file
View File

@ -0,0 +1,14 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: registry-pv
labels:
type: registry
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/registry

13
make/kubernetes/pv/registry.pvc.yaml Normal file
View File

@ -0,0 +1,13 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
selector:
matchLabels:
type: registry

14
make/kubernetes/pv/storage.pv.yaml Normal file
View File

@ -0,0 +1,14 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: storage-pv
labels:
type: storage
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/storage

13
make/kubernetes/pv/storage.pvc.yaml Normal file
View File

@ -0,0 +1,13 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: storage-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
selector:
matchLabels:
type: storage

28
make/kubernetes/registry-rc.yaml
View File

@ -1,28 +0,0 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: registry
labels:
name: registry
spec:
replicas: 1
selector:
name: registry
template:
metadata:
labels:
name: registry
spec:
containers:
- name: registry
image: caicloud/harbor_registry:2.3.0
imagePullPolicy: Always
ports:
- containerPort: 5000
- containerPort: 5001
volumeMounts:
- name: storage
mountPath: /storage
volumes:
- name: storage
emptyDir: {}

39
make/kubernetes/registry/registry.rc.yaml Normal file
View File

@ -0,0 +1,39 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: registry-rc
labels:
name: registry-rc
spec:
replicas: 1
selector:
name: registry-apps
template:
metadata:
labels:
name: registry-apps
spec:
containers:
- name: registry-app
image: harbor/registry
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5000
- containerPort: 5001
volumeMounts:
- name: config
mountPath: /etc/docker/registry
- name: storage
mountPath: /storage
volumes:
- name: config
configMap:
name: harbor-registry-config
items:
- key: config
path: config.yml
- key: cert
path: root.crt
- name: storage
persistentVolumeClaim:
claimName: registry-pvc

8
make/kubernetes/registry-svc.yaml → make/kubernetes/registry/registry.svc.yaml
View File

@ -2,13 +2,11 @@ apiVersion: v1
kind: Service
metadata:
name: registry
labels:
name: registry
spec:
ports:
- name: internal
- name: repo
port: 5000
- name: external
- name: debug
port: 5001
selector:
name: registry
name: registry-apps

25
make/kubernetes/templates/jobservice.cm.yaml Normal file
View File

@ -0,0 +1,25 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: harbor-jobservice-config
data:
MYSQL_HOST: mysql
MYSQL_PORT: "3306"
MYSQL_USR: root
MYSQL_PWD: "{{db_password}}"
UI_SECRET: "{{ui_secret}}"
SECRET_KEY: "{{secret_key}}"
CONFIG_PATH: /etc/jobservice/app.conf
REGISTRY_URL: http://registry:5000
VERIFY_REMOTE_CERT: "{{verify_remote_cert}}"
MAX_JOB_WORKERS: "{{max_job_workers}}"
LOG_LEVEL: debug
LOG_DIR: /var/log/jobs
GODEBUG: netdns=cgo
EXT_ENDPOINT: "{{ui_url}}"
TOKEN_URL: http://ui
config: |
appname = jobservice
runmode = dev
[dev]
httpport = 80

6
make/kubernetes/templates/mysql.cm.yaml Normal file
View File

@ -0,0 +1,6 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: harbor-mysql-config
data:
MYSQL_ROOT_PASSWORD: "{{db_password}}"

155
make/kubernetes/templates/nginx.cm.yaml Normal file
View File

@ -0,0 +1,155 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: harbor-nginx-config
data:
config: |
worker_processes auto;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
tcp_nodelay on;
# this is necessary for us to be able to disable request buffering in all cases
proxy_http_version 1.1;
upstream registry {
server registry:5000;
}
upstream ui {
server ui:80;
}
server {
listen 443 ssl;
server_name {{hostname}};
# SSL
ssl_certificate /etc/nginx/https.crt;
ssl_certificate_key /etc/nginx/https.key;
# Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location / {
proxy_pass http://ui/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /v1/ {
return 404;
}
location /v2/ {
proxy_pass http://registry/v2/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /service/ {
proxy_pass http://ui/service/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
}
server {
listen 80;
server_name {{hostname}};
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
# rewrite ^/(.*) https://$server_name:443/$1 permanent;
location / {
proxy_pass http://ui/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /v1/ {
return 404;
}
location /v2/ {
proxy_pass http://registry/v2/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /service/ {
proxy_pass http://ui/service/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
}
}
pkey: |
{{4 https_pkey}}
cert: |
{{4 https_cert}}

44
make/kubernetes/templates/registry.cm.yaml Normal file
View File

@ -0,0 +1,44 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: harbor-registry-config
data:
config: |
version: 0.1
log:
level: debug
fields:
service: registry
storage:
filesystem:
rootdirectory: /storage
cache:
layerinfo: inmemory
maintenance:
uploadpurging:
enabled: false
delete:
enabled: true
http:
addr: :5000
secret: placeholder
debug:
addr: localhost:5001
auth:
token:
issuer: registry-token-issuer
realm: {{ui_url}}/service/token
rootcertbundle: /etc/docker/registry/root.crt
service: token-service
notifications:
endpoints:
- name: harbor
disabled: false
url: http://ui/service/notifications
timeout: 3000ms
threshold: 5
backoff: 1s
cert: |
{{4 auth_cert}}

52
make/kubernetes/templates/ui.cm.yaml Normal file
View File

@ -0,0 +1,52 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: harbor-ui-config
data:
MYSQL_HOST: mysql
MYSQL_PORT: "3306"
MYSQL_USR: root
MYSQL_PWD: "{{db_password}}"
REGISTRY_URL: http://registry:5000
CONFIG_PATH: /etc/ui/app.conf
HARBOR_REG_URL: "{{hostname}}"
HARBOR_ADMIN_PASSWORD: "{{harbor_admin_password}}"
HARBOR_URL: http://ui
AUTH_MODE: "{{auth_mode}}"
LDAP_URL: "{{ldap_url}}"
LDAP_SEARCH_DN: "{{ldap_searchdn}}"
LDAP_SEARCH_PWD: "{{ldap_search_pwd}}"
LDAP_BASE_DN: "{{ldap_basedn}}"
LDAP_FILTER: "{{ldap_filter}}"
LDAP_UID: "{{ldap_uid}}"
LDAP_SCOPE: "{{ldap_scope}}"
LOG_LEVEL: debug
UI_SECRET: "{{ui_secret}}"
SECRET_KEY: "{{secret_key}}"
GODEBUG: netdns=cgo
EXT_ENDPOINT: "{{ui_url}}"
TOKEN_URL: http://ui
SELF_REGISTRATION: "{{self_registration}}"
USE_COMPRESSED_JS: "{{use_compressed_js}}"
VERIFY_REMOTE_CERT: "{{verify_remote_cert}}"
TOKEN_EXPIRATION: "{{token_expiration}}"
EXT_REG_URL: "{{hostname}}"
config: |
appname = registry
runmode = dev
[lang]
types = en-US|zh-CN
names = en-US|zh-CN
[dev]
httpport = 80
[mail]
host = {{email_server}}
port = {{email_server_port}}
username = {{email_username}}
password = {{email_password}}
from = {{email_from}}
ssl = {{email_ssl}}
pkey: |
{{4 auth_pkey}}

49
make/kubernetes/ui-rc.yaml
View File

@ -1,49 +0,0 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: ui
labels:
name: ui
spec:
replicas: 1
selector:
name: ui
template:
metadata:
labels:
name: ui
spec:
containers:
- name: ui
image: caicloud/harbor_deploy_ui:latest
imagePullPolicy: Always
env:
- name: MYSQL_HOST
value: mysql
- name: MYSQL_PORT
value: "3306"
- name: MYSQL_USR
value: root
- name: MYSQL_PWD
value: root123
- name: REGISTRY_URL
value: http://registry:5000
- name: CONFIG_PATH
value: /etc/ui/app.conf
- name: HARBOR_REG_URL
value: localhost
- name: HARBOR_ADMIN_PASSWORD
value: Harbor12345
- name: HARBOR_URL
value: http://localhost
- name: AUTH_MODE
value: db_auth
- name: LDAP_URL
value: ldaps://ldap.mydomain.com
- name: LDAP_BASE_DN
value: uid=%s,ou=people,dc=mydomain,dc=com
- name: LOG_LEVEL
value: debug
ports:
- containerPort: 80

174
make/kubernetes/ui/ui.rc.yaml Normal file
View File

@ -0,0 +1,174 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: ui-rc
labels:
name: ui-rc
spec:
replicas: 1
selector:
name: ui-apps
template:
metadata:
labels:
name: ui-apps
spec:
containers:
- name: ui-app
image: harbor/ui
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_HOST
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_HOST
- name: MYSQL_PORT
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_PORT
- name: MYSQL_USR
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_USR
- name: MYSQL_PWD
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_PWD
- name: REGISTRY_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: REGISTRY_URL
- name: CONFIG_PATH
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: CONFIG_PATH
- name: HARBOR_REG_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: HARBOR_REG_URL
- name: HARBOR_ADMIN_PASSWORD
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: HARBOR_ADMIN_PASSWORD
- name: HARBOR_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: HARBOR_URL
- name: AUTH_MODE
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: AUTH_MODE
- name: LDAP_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_URL
- name: LDAP_SEARCH_DN
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_SEARCH_DN
- name: LDAP_SEARCH_PWD
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_SEARCH_PWD
- name: LDAP_BASE_DN
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_BASE_DN
- name: LDAP_FILTER
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_FILTER
- name: LDAP_UID
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_UID
- name: LDAP_SCOPE
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_SCOPE
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LOG_LEVEL
- name: UI_SECRET
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: UI_SECRET
- name: SECRET_KEY
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: SECRET_KEY
- name: GODEBUG
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: GODEBUG
- name: EXT_ENDPOINT
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: EXT_ENDPOINT
- name: TOKEN_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: TOKEN_URL
- name: SELF_REGISTRATION
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: SELF_REGISTRATION
- name: USE_COMPRESSED_JS
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: USE_COMPRESSED_JS
- name: VERIFY_REMOTE_CERT
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: VERIFY_REMOTE_CERT
- name: TOKEN_EXPIRATION
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: TOKEN_EXPIRATION
- name: EXT_REG_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: EXT_REG_URL
ports:
- containerPort: 80
volumeMounts:
- name: config
mountPath: /etc/ui
volumes:
- name: config
configMap:
name: harbor-ui-config
items:
- key: config
path: app.conf
- key: pkey
path: private_key.pem

4
make/kubernetes/ui-svc.yaml → make/kubernetes/ui/ui.svc.yaml
View File

@ -2,10 +2,8 @@ apiVersion: v1
kind: Service
metadata:
name: ui
labels:
name: ui
spec:
ports:
- port: 80
selector:
name: ui
name: ui-apps