harbor/docs/kubernetes_deployment.md

Integration with Kubernetes

This Document decribes how to deploy Harbor on Kubernetes. It has been verified on Kubernetes v1.6.5 and Harbor v1.2.0

Prerequisite

• You should have domain knowledge about Kubernetes (Replication Controller, Service, Persistent Volume, Persistent Volume Claim, Config Map).
• Optional: Load the docker images onto woker nodes. If you skip this step, worker node will pull images from Docker Hub when starting the pods.
  • Download the offline installer of Harbor v1.2.0 from the release page.
  • Uncompress the offline installer and get the images tgz file harbor.*.tgz, transfer it to each of the worker nodes.
  • Load the images into docker:

    docker load -i harbor.*.tgz 
    

Configuration

We provide a python script make/kubernetes/k8s-prepare to generate Kubernetes ConfigMap files. The script is written in python, so you need a version of python in your deployment environment. Also the script need openssl to generate private key and certification, make sure you have a workable openssl.

There are some args of the python script:

• -f: Default Value is ../harbor.cfg. You can specify other config file of Harbor.
• -k: Path to https private key. This arg can overwrite the value of ssl_cert_key in harbor.cfg.
• -c: Path to https certification. This arg can overwrite the value of ssl_cert in harbor.cfg.

Basic Configuration

These Basic Configuration must be set. Otherwise you can't deploy Harbor on Kubernetes.

• make/harbor.cfg: Basic config of Harbor. Please refer to harbor.cfg.

  #Hostname is the endpoint for accessing Harbor,
  #To accept access from outside of Kubernetes cluster, it should be set to a worker node.
  hostname = 10.192.168.5
  

• make/kubernetes/**/*.svc.yaml: Specify the service of pods. In particular, the externalIP should be set in make/kubernetes/nginx/nginx.svc.yaml:

  ...
  metadata:
      name: nginx
  spec:
      ports:
      - name: http
        port: 80
      selector:
        name: nginx-apps
      externalIPs:
        - 10.192.168.5
  

• make/kubernetes/**/*.rc.yaml: Specify configs of containers.

• make/kubernetes/pv/*.pvc.yaml: Persistent Volume Claim.
  You can set capacity of storage in these files. example:

  resources:
    requests:
      # you can set another value to adapt to your needs
      storage: 100Gi
  

• make/kubernetes/pv/*.pv.yaml: Persistent Volume. Be bound with *.pvc.yaml.
  PVs and PVCs are one to one correspondence. If you changed capacity of PVC, you need to set capacity of PV together. example:

  capacity:
    # same value with PVC
    storage: 100Gi
  

  In PV, you should set another way to store data rather than hostPath:

  # it's default value, you should use others like nfs.
  hostPath:
    path: /data/registry
  

  For more infomation about storage solution, Please check Kubernetes Document

Then you can generate ConfigMap files by :

python make/kubernetes/k8s-prepare

These files will be generated:

• make/kubernetes/jobservice/jobservice.cm.yaml
• make/kubernetes/mysql/mysql.cm.yaml
• make/kubernetes/nginx/nginx.cm.yaml
• make/kubernetes/registry/registry.cm.yaml
• make/kubernetes/ui/ui.cm.yaml
• make/kubernetes/adminserver/adminserver.cm.yaml

Advanced Configuration

If Basic Configuration was not covering your requirements, you can read this section for more details.

./k8s-prepare has a specify format of placeholder:

• {{key}}: It means we should replace the placeholder with the value in config.cfg which name is key.
• {{num key}}: It's used for multiple lines text. It will add num spaces to the leading of every line in text.

You can find all configs of Harbor in make/kubernetes/templates/. There are specifications of these files:

• jobservice.cm.yaml: ENV and web config of jobservice

• mysql.cm.yaml: Root passowrd of MySQL

• nginx.cm.yaml: Https certification and nginx config. If you are fimiliar with nginx, you can modify it.

• registry.cm.yaml: Token service certification and registry config Registry use filesystem to store data of images. You can find it like:

  storage:
      filesystem:
        rootdirectory: /storage
  

  If you want use another storage backend, please see Docker Doc

• ui.cm.yaml: Token service private key, ENV and web config of ui.

• adminserver.cm.yaml: Initial values of configuration attributes of Harbor.

ui, jobservice and adminserver are powered by beego. If you are fimiliar with beego, you can modify configs in ui.cm.yaml, jobservice.cm.yaml and adminserver.cm.yaml.

Running

When you finished your configuring and generated ConfigMap files, you can run Harbor on kubernetes with these commands:

# create pv & pvc
kubectl apply -f make/kubernetes/pv/log.pv.yaml
kubectl apply -f make/kubernetes/pv/registry.pv.yaml
kubectl apply -f make/kubernetes/pv/storage.pv.yaml
kubectl apply -f make/kubernetes/pv/log.pvc.yaml
kubectl apply -f make/kubernetes/pv/registry.pvc.yaml
kubectl apply -f make/kubernetes/pv/storage.pvc.yaml

# create config map
kubectl apply -f make/kubernetes/jobservice/jobservice.cm.yaml
kubectl apply -f make/kubernetes/mysql/mysql.cm.yaml
kubectl apply -f make/kubernetes/nginx/nginx.cm.yaml
kubectl apply -f make/kubernetes/registry/registry.cm.yaml
kubectl apply -f make/kubernetes/ui/ui.cm.yaml
kubectl apply -f make/kubernetes/adminserver/adminserver.cm.yaml

# create service
kubectl apply -f make/kubernetes/jobservice/jobservice.svc.yaml
kubectl apply -f make/kubernetes/mysql/mysql.svc.yaml
kubectl apply -f make/kubernetes/nginx/nginx.svc.yaml
kubectl apply -f make/kubernetes/registry/registry.svc.yaml
kubectl apply -f make/kubernetes/ui/ui.svc.yaml
kubectl apply -f make/kubernetes/adminserver/adminserver.svc.yaml

# create k8s rc
kubectl apply -f make/kubernetes/registry/registry.rc.yaml
kubectl apply -f make/kubernetes/mysql/mysql.rc.yaml
kubectl apply -f make/kubernetes/jobservice/jobservice.rc.yaml
kubectl apply -f make/kubernetes/ui/ui.rc.yaml
kubectl apply -f make/kubernetes/nginx/nginx.rc.yaml
kubectl apply -f make/kubernetes/adminserver/adminserver.rc.yaml

After the pods are running, you can access Harbor's UI via the configured endpoint 10.192.168.5 or issue docker commands such as docker login 10.192.168.5 to interact with the registry.

Limitation

1. Current deployment is http only, to enable https you need to either add another layer of proxy or modify the nginx.cm.yaml to enable https and include a correct certificate
2. Current deployment does not include Clair and Notary, which are supported in docker-compose deployment. They will be supported in near future, stay tuned.