Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-12-27 02:58:05 +01:00
Code Issues Projects Releases Wiki Activity

Refine k8s deployment scripts and document

Browse Source
This commit is contained in:
Tan Jiang 2017-09-25 17:25:16 +08:00
parent 01eb9ff7af
commit 0615f7ba9b
14 changed files with 67 additions and 301 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

57
docs/kubernetes_deployment.md
View File

@ -1,10 +1,10 @@
## Integration with Kubernetes
This Document decribes how to deploy Harbor on Kubernetes.
This Document decribes how to deploy Harbor on Kubernetes. It has been verified on **Kubernetes v1.6.5** and **Harbor v1.2.0**
### Prerequisite
* You need to download docker images of Harbor.
* Download the offline installer of Harbor from the [release](https://github.com/vmware/harbor/releases) page.
* Download the offline installer of Harbor v1.2.0 from the [release](https://github.com/vmware/harbor/releases) page.
* Uncompress the offline installer and get the images tgz file harbor.*.tgz.
* Load the images into docker:
```
@ -18,6 +18,7 @@ The script is written in python, so you need a version of python in your deploym
Also the script need `openssl` to generate private key and certification, make sure you have a workable `openssl`.
There are some args of the python script:
- -f: Default Value is `../harbor.cfg`. You can specify other config file of Harbor.
- -k: Path to https private key. This arg can overwrite the value of `ssl_cert_key` in `harbor.cfg`.
- -c: Path to https certification. This arg can overwrite the value of `ssl_cert` in `harbor.cfg`.
@ -25,17 +26,31 @@ There are some args of the python script:
#### Basic Configuration
These Basic Configuration must be set. Otherwise you can't deploy Harbor on Kubernetes.
- `make/harbor.cfg`: Basic config of Harbor. Please refer to `harbor.cfg`.
```
#Hostname is the endpoint for accessing Harbor,
#To accept access from outside of Kubernetes cluster, it should be set to a worker node.
hostname = 10.192.168.5
```
- `make/kubernetes/**/*.svc.yaml`: Specify the service of pods. In particular, the externalIP should be set in `make/kubernetes/nginx/nginx.svc.yaml`:
```
...
metadata:
name: nginx
spec:
ports:
- name: http
port: 80
selector:
name: nginx-apps
externalIPs:
- 10.192.168.5
```
- `make/kubernetes/**/*.rc.yaml`: Specify configs of containers.
You need to specify the path to your images in all `*.rc.yaml`. example:
```
containers:
- name: nginx-app
# it's very importent that you need modify the path of image.
image: harbor/nginx
```
- `make/kubernetes/pv/*.pvc.yaml`: Persistent Volume Claim.
You can set capacity of storage in these files. example:
@ -64,7 +79,7 @@ These Basic Configuration must be set. Otherwise you can't deploy Harbor on Kube
path: /data/registry
```
For more infomation about store ways, Please check [Kubernetes Document](http://kubernetes.io/docs/user-guide/persistent-volumes/)
For more infomation about storage solution, Please check [Kubernetes Document](http://kubernetes.io/docs/user-guide/persistent-volumes/)
Then you can generate ConfigMap files by :
@ -73,20 +88,24 @@ python make/kubernetes/prepare
```
These files will be generated:
- make/kubernetes/jobservice/jobservice.cm.yaml
- make/kubernetes/mysql/mysql.cm.yaml
- make/kubernetes/nginx/nginx.cm.yaml
- make/kubernetes/registry/registry.cm.yaml
- make/kubernetes/ui/ui.cm.yaml
- make/kubernetes/adminserver/adminserver.cm.yaml
#### Advanced Configuration
If Basic Configuration was not covering your requirements, you can read this section for more details.
`./prepare` has a specify format of placeholder:
- `{{key}}`: It means we should replace the placeholder with the value in `config.cfg` which name is `key`.
- `{{num key}}`: It's used for multiple lines text. It will add `num` spaces to the leading of every line in text.
You can find all configs of Harbor in `make/kubernetes/templates/`. There are specifications of these files:
- `jobservice.cm.yaml`: ENV and web config of jobservice
- `mysql.cm.yaml`: Root passowrd of MySQL
- `nginx.cm.yaml`: Https certification and nginx config. If you are fimiliar with nginx, you can modify it.
@ -100,15 +119,15 @@ You can find all configs of Harbor in `make/kubernetes/templates/`. There are sp
```
If you want use another storage backend, please see [Docker Doc](https://docs.docker.com/datacenter/dtr/2.1/guides/configure/configure-storage/)
- `ui.cm.yaml`: Token service private key, ENV and web config of ui
`ui` and `jobservice` are powered by beego. If you are fimiliar with beego, you can modify configs in `jobservice.cm.yaml` and `ui.cm.yaml`.
- `ui.cm.yaml`: Token service private key, ENV and web config of ui.
- `adminserver.cm.yaml`: Initial values of configuration attributes of Harbor.
`ui`, `jobservice` and `adminserver` are powered by beego. If you are fimiliar with beego, you can modify configs in `ui.cm.yaml`, `jobservice.cm.yaml` and `adminserver.cm.yaml`.
### Running
When you finished your configuring and generated ConfigMap files, you can run Harbor on kubernetes with these commands:
```
# create pv & pvc
kubectl apply -f make/kubernetes/pv/log.pv.yaml
@ -124,6 +143,7 @@ kubectl apply -f make/kubernetes/mysql/mysql.cm.yaml
kubectl apply -f make/kubernetes/nginx/nginx.cm.yaml
kubectl apply -f make/kubernetes/registry/registry.cm.yaml
kubectl apply -f make/kubernetes/ui/ui.cm.yaml
kubectl apply -f make/kubernetes/adminserver/adminserver.cm.yaml
# create service
kubectl apply -f make/kubernetes/jobservice/jobservice.svc.yaml
@ -131,6 +151,7 @@ kubectl apply -f make/kubernetes/mysql/mysql.svc.yaml
kubectl apply -f make/kubernetes/nginx/nginx.svc.yaml
kubectl apply -f make/kubernetes/registry/registry.svc.yaml
kubectl apply -f make/kubernetes/ui/ui.svc.yaml
kubectl apply -f make/kubernetes/adminserver/adminserver.svc.yaml
# create k8s rc
kubectl apply -f make/kubernetes/registry/registry.rc.yaml
@ -138,5 +159,7 @@ kubectl apply -f make/kubernetes/mysql/mysql.rc.yaml
kubectl apply -f make/kubernetes/jobservice/jobservice.rc.yaml
kubectl apply -f make/kubernetes/ui/ui.rc.yaml
kubectl apply -f make/kubernetes/nginx/nginx.rc.yaml
kubectl apply -f make/kubernetes/adminserver/adminserver.rc.yaml
```
After the pods are running, you can access Harbor's UI via the configured endpoint `10.192.168.5` or issue docker commands such as `docker login 10.192.168.5` to interact with the registry.

5
make/kubernetes/adminserver/adminserver.rc.yaml
View File

@ -15,7 +15,7 @@ spec:
spec:
containers:
- name: adminserver-app
image: 192.168.56.201:5000/vmware/harbor-adminserver:dev
image: vmware/harbor-adminserver:v1.2.0
imagePullPolicy: IfNotPresent
env:
- name: LOG_LEVEL
@ -218,6 +218,9 @@ spec:
configMapKeyRef:
name: harbor-adminserver-config
key: RESET
#Workaround the volume API issue.
- name: IMAGE_STORE_PATH
value: "/"
ports:
- containerPort: 80
volumeMounts:

52
make/kubernetes/jobservice/jobservice.rc.yaml
View File

@ -15,29 +15,9 @@ spec:
spec:
containers:
- name: jobservice-app
image: harbor/jobservice
image: vmware/harbor-jobservice:v1.2.0
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_HOST
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_HOST
- name: MYSQL_PORT
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_PORT
- name: MYSQL_USR
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_USR
- name: MYSQL_PWD
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MYSQL_PWD
- name: UI_SECRET
valueFrom:
configMapKeyRef:
@ -53,46 +33,16 @@ spec:
configMapKeyRef:
name: harbor-jobservice-config
key: CONFIG_PATH
- name: REGISTRY_URL
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: REGISTRY_URL
- name: VERIFY_REMOTE_CERT
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: VERIFY_REMOTE_CERT
- name: MAX_JOB_WORKERS
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: MAX_JOB_WORKERS
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: LOG_LEVEL
- name: LOG_DIR
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: LOG_DIR
- name: GODEBUG
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: GODEBUG
- name: EXT_ENDPOINT
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: EXT_ENDPOINT
- name: TOKEN_URL
valueFrom:
configMapKeyRef:
name: harbor-jobservice-config
key: TOKEN_URL
ports:
- containerPort: 80
volumeMounts:

2
make/kubernetes/mysql/mysql.rc.yaml
View File

@ -15,7 +15,7 @@ spec:
spec:
containers:
- name: mysql-app
image: harbor/mysql
image: vmware/harbor-db:v1.2.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306

4
make/kubernetes/nginx/nginx.rc.yaml
View File

@ -15,7 +15,7 @@ spec:
spec:
containers:
- name: nginx-app
image: harbor/nginx
image: vmware/nginx-photon:1.11.13
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
@ -33,4 +33,4 @@ spec:
- key: pkey
path: https.key
- key: cert
path: https.crt
path: https.crt

5
make/kubernetes/nginx/nginx.svc.yaml
View File

@ -6,7 +6,8 @@ spec:
ports:
- name: http
port: 80
- name: https
port: 443
selector:
name: nginx-apps
# Set the external IP to an IP of the cluster node, so that the service can be accessed from outside the kubernetes cluster.
# externalIPs:
# - 10.192.168.5

6
make/kubernetes/prepare Normal file → Executable file
View File

@ -158,10 +158,8 @@ with open(os.devnull, 'w') as devnull:
openssl = subprocess.call(['which','openssl'], stdout=devnull, stderr=devnull)
if openssl == 0:
pkey = subprocess.check_output(['openssl','genrsa','4096'], stderr=devnull)
subj = '/C={0}/ST={1}/L={2}/O={3}/OU={4}/CN={5}/emailAddress={6}'.format(get_config('crt_country'),
get_config('crt_state'), get_config('crt_location'), get_config('crt_organization'),
get_config('crt_organizationalunit'), get_config('crt_commonname'), get_config('crt_email'))
openssl = subprocess.Popen(['openssl', 'req', '-new', '-x509', '-key', '/dev/stdin', '-days', '3650', '-subj', subj],
empty_subj = "/C=/ST=/L=/O=/CN=/"
openssl = subprocess.Popen(['openssl', 'req', '-new', '-x509', '-key', '/dev/stdin', '-days', '3650', '-subj', empty_subj],
stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=devnull)
cert = openssl.communicate(input=pkey)[0]
set_config('auth_pkey', pkey.decode())

2
make/kubernetes/registry/registry.rc.yaml
View File

@ -15,7 +15,7 @@ spec:
spec:
containers:
- name: registry-app
image: harbor/registry
image: vmware/registry:2.6.2-photon
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5000

8
make/kubernetes/templates/adminserver.cm.yaml
View File

@ -7,10 +7,10 @@ data:
AUTH_MODE: db_auth
SELF_REGISTRATION: "on"
LDAP_URL: ldaps://ldap.mydomain.com
LDAP_SEARCH_DN:
LDAP_SEARCH_PWD:
LDAP_SEARCH_DN: ""
LDAP_SEARCH_PWD: ""
LDAP_BASE_DN: "ou=people,dc=mydomain,dc=com"
LDAP_FILTER:
LDAP_FILTER: ""
LDAP_UID: uid
LDAP_SCOPE: "3"
LDAP_TIMEOUT: "5"
@ -28,7 +28,7 @@ data:
EMAIL_PWD: abc
EMAIL_SSL: "false"
EMAIL_FROM: "admin <sample_admin@mydomain.com>"
EMAIL_IDENTITY:
EMAIL_IDENTITY: ""
HARBOR_ADMIN_PASSWORD: "{{harbor_admin_password}}"
PROJECT_CREATION_RESTRICTION: everyone
VERIFY_REMOTE_CERT: "on"

12
make/kubernetes/templates/jobservice.cm.yaml
View File

@ -3,21 +3,13 @@ kind: ConfigMap
metadata:
name: harbor-jobservice-config
data:
MYSQL_HOST: mysql
MYSQL_PORT: "3306"
MYSQL_USR: root
MYSQL_PWD: "{{db_password}}"
UI_SECRET: "{{ui_secret}}"
JOBSERVICE_SECRET: "{{jobservice_secret}}"
CONFIG_PATH: /etc/jobservice/app.conf
REGISTRY_URL: http://registry:5000
VERIFY_REMOTE_CERT: "{{verify_remote_cert}}"
SECRET_KEY: "{{secret_key}}"
MAX_JOB_WORKERS: "{{max_job_workers}}"
LOG_LEVEL: debug
LOG_DIR: /var/log/jobs
GODEBUG: netdns=cgo
EXT_ENDPOINT: "{{ui_url}}"
TOKEN_URL: http://ui
CONFIG_PATH: /etc/jobservice/app.conf
config: |
appname = jobservice
runmode = dev

66
make/kubernetes/templates/nginx.cm.yaml
View File

@ -26,72 +26,6 @@ data:
upstream ui {
server ui:80;
}
server {
listen 443 ssl;
server_name {{hostname}};
# SSL
ssl_certificate /etc/nginx/https.crt;
ssl_certificate_key /etc/nginx/https.key;
# Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location / {
proxy_pass http://ui/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /v1/ {
return 404;
}
location /v2/ {
proxy_pass http://registry/v2/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /service/ {
proxy_pass http://ui/service/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
}
server {
listen 80;
server_name {{hostname}};

4
make/kubernetes/templates/registry.cm.yaml
View File

@ -26,10 +26,10 @@ data:
addr: localhost:5001
auth:
token:
issuer: registry-token-issuer
issuer: harbor-token-issuer
realm: {{ui_url}}/service/token
rootcertbundle: /etc/docker/registry/root.crt
service: token-service
service: harbor-registry
notifications:
endpoints:
- name: harbor

26
make/kubernetes/templates/ui.cm.yaml
View File

@ -3,34 +3,12 @@ kind: ConfigMap
metadata:
name: harbor-ui-config
data:
MYSQL_HOST: mysql
MYSQL_PORT: "3306"
MYSQL_USR: root
MYSQL_PWD: "{{db_password}}"
REGISTRY_URL: http://registry:5000
CONFIG_PATH: /etc/ui/app.conf
HARBOR_REG_URL: "{{hostname}}"
HARBOR_ADMIN_PASSWORD: "{{harbor_admin_password}}"
HARBOR_URL: http://ui
AUTH_MODE: "{{auth_mode}}"
LDAP_URL: "{{ldap_url}}"
LDAP_SEARCH_DN: "{{ldap_searchdn}}"
LDAP_SEARCH_PWD: "{{ldap_search_pwd}}"
LDAP_BASE_DN: "{{ldap_basedn}}"
LDAP_FILTER: "{{ldap_filter}}"
LDAP_UID: "{{ldap_uid}}"
LDAP_SCOPE: "{{ldap_scope}}"
LOG_LEVEL: debug
UI_SECRET: "{{ui_secret}}"
JOBSERVICE_SECRET: "{{jobservice_secre}}"
JOBSERVICE_SECRET: "{{jobservice_secret}}"
GODEBUG: netdns=cgo
EXT_ENDPOINT: "{{ui_url}}"
TOKEN_URL: http://ui
SELF_REGISTRATION: "{{self_registration}}"
USE_COMPRESSED_JS: "{{use_compressed_js}}"
VERIFY_REMOTE_CERT: "{{verify_remote_cert}}"
TOKEN_EXPIRATION: "{{token_expiration}}"
EXT_REG_URL: "{{hostname}}"
SECRET_KEY: "{{secret_key}}"
config: |
appname = registry
runmode = dev

119
make/kubernetes/ui/ui.rc.yaml
View File

@ -15,94 +15,14 @@ spec:
spec:
containers:
- name: ui-app
image: harbor/ui
image: vmware/harbor-ui:v1.2.0
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_HOST
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_HOST
- name: MYSQL_PORT
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_PORT
- name: MYSQL_USR
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_USR
- name: MYSQL_PWD
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: MYSQL_PWD
- name: REGISTRY_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: REGISTRY_URL
- name: CONFIG_PATH
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: CONFIG_PATH
- name: HARBOR_REG_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: HARBOR_REG_URL
- name: HARBOR_ADMIN_PASSWORD
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: HARBOR_ADMIN_PASSWORD
- name: HARBOR_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: HARBOR_URL
- name: AUTH_MODE
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: AUTH_MODE
- name: LDAP_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_URL
- name: LDAP_SEARCH_DN
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_SEARCH_DN
- name: LDAP_SEARCH_PWD
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_SEARCH_PWD
- name: LDAP_BASE_DN
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_BASE_DN
- name: LDAP_FILTER
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_FILTER
- name: LDAP_UID
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_UID
- name: LDAP_SCOPE
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: LDAP_SCOPE
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
@ -123,41 +43,6 @@ spec:
configMapKeyRef:
name: harbor-ui-config
key: GODEBUG
- name: EXT_ENDPOINT
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: EXT_ENDPOINT
- name: TOKEN_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: TOKEN_URL
- name: SELF_REGISTRATION
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: SELF_REGISTRATION
- name: USE_COMPRESSED_JS
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: USE_COMPRESSED_JS
- name: VERIFY_REMOTE_CERT
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: VERIFY_REMOTE_CERT
- name: TOKEN_EXPIRATION
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: TOKEN_EXPIRATION
- name: EXT_REG_URL
valueFrom:
configMapKeyRef:
name: harbor-ui-config
key: EXT_REG_URL
ports:
- containerPort: 80
volumeMounts:
@ -172,3 +57,5 @@ spec:
path: app.conf
- key: pkey
path: private_key.pem
- key: SECRET_KEY
path: key