[Chart] Add support for deploying Notary

Assuming Harbor FQDN is specified as harbor.my.domain by the user, the Notary server FQDN will be set to notary-harbor.my.domain. So the user must point both Harbor FQDN and Notary FQDN to the K8s ingress controller IP in the DNS server. Fix issue https://github.com/vmware/harbor/issues/4394
2024-12-22 16:48:30 +01:00 · 2018-04-18 18:08:17 +08:00 · 2018-04-18 18:08:17 +08:00 · ce48dbfec2
commit ce48dbfec2
parent 9d13842a29
12 changed files with 365 additions and 19 deletions
--- a/contrib/helm/harbor/README.md
+++ b/contrib/helm/harbor/README.md
@ -176,24 +176,32 @@ The following tables lists the configurable parameters of the Harbor chart and t
 | `registry.resources` | [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) to allocate for container   | undefined |
 | `registry.volumes` | used to create PVCs if persistence is enabled (see instructions in values.yaml) | see values.yaml |
 | **Clair** |
-| `clair.enabled` | Enable clair? | `true` |
+| `clair.enabled` | Enable Clair? | `true` |
 | `clair.image.repository` | Repository for clair image | `vmware/clair-photon` |
 | `clair.image.tag` | Tag for clair image | `v2.0.1-v1.4.0`
 | `clair.resources` | [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) to allocate for container   | undefined
 | `postgresql` | Overrides for postgresql chart [values.yaml](https://github.com/kubernetes/charts/blob/f2938a46e3ae8e2512ede1142465004094c3c333/stable/postgresql/values.yaml) | see values.yaml
-| | | |
+| **Notary** |
+| `notary.enabled` | Enable Notary? | `true` |
+| `notary.server.image.repository` | Repository for notary server image | `vmware/notary-server-photon` |
+| `notary.server.image.tag` | Tag for notary server image | `v0.5.1-v1.4.0`
+| `notary.signer.image.repository` | Repository for notary signer image | `vmware/notary-signer-photon` |
+| `notary.signer.image.tag` | Tag for notary signer image | `v0.5.1-v1.4.0`
+| `notary.db.image.repository` | Repository for notary database image | `vmware/mariadb-photon` |
+| `notary.db.image.tag` | Tag for notary database image | `v1.4.0`
+| `notary.db.password` | The password of users for notary database | Specify your own password |


 Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example:

 ```bash
-helm install --name my-release --set mysql.pass=baconeggs .
+helm install . --name my-release --set externalDomain=harbor.<IP>.xip.io
 ```

 Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

 ```bash
-helm install --name my-release -f /path/to/values.yaml .
+helm install . --name my-release -f /path/to/values.yaml
 ```

 > **Tip**: You can use the default [values.yaml](values.yaml)
--- a/contrib/helm/harbor/templates/NOTES.txt
+++ b/contrib/helm/harbor/templates/NOTES.txt
@ -1,5 +1,8 @@

-Add the Harbor CA certificate to Docker by executing the following command:
+Please wait for several minutes for Harbor deployment to complete.
+Then follow the steps below to use Harbor.
+
+1. Add the Harbor CA certificate to Docker by executing the following command:

  sudo mkdir -p /etc/docker/certs.d/{{ .Values.externalDomain }}
  kubectl get secret \
@ -7,14 +10,17 @@ Add the Harbor CA certificate to Docker by executing the following command:
    -o jsonpath="{.data.ca\.crt}" | base64 --decode | \
    sudo tee /etc/docker/certs.d/{{ .Values.externalDomain }}/ca.crt

-Get Harbor admin password by executing the following command:
+2. Get Harbor admin password by executing the following command:

  kubectl get secret --namespace {{ .Release.Namespace }} {{ template "harbor.fullname" . }}-adminserver -o jsonpath="{.data.HARBOR_ADMIN_PASSWORD}" | base64 --decode; echo

-Add Harbor FQDN {{ .Values.externalDomain }} to K8s Ingress Controller IP resolution on DNS Server or in file /etc/hosts.
+3. Add DNS resolution entry for Harbor FQDN {{ .Values.externalDomain }} to K8s Ingress Controller IP on DNS Server or in file /etc/hosts.
+{{- if .Values.notary.enabled }}
+   Add DNS resolution entry for Notary FQDN {{ template "harbor.notaryFQDN" . }} to K8s Ingress Controller IP on DNS Server or in file /etc/hosts.
+{{- end }}

-Access Harbor UI via https://{{ .Values.externalDomain }}
+4. Access Harbor UI via https://{{ .Values.externalDomain }}

-Login Harbor with Docker CLI:
+5. Login Harbor with Docker CLI:

  docker login {{ .Values.externalDomain }}
--- a/contrib/helm/harbor/templates/_helpers.tpl
+++ b/contrib/helm/harbor/templates/_helpers.tpl
@ -29,3 +29,29 @@ app: "{{ template "harbor.name" . }}"
 release: {{ .Release.Name }}
 app: "{{ template "harbor.name" . }}"
 {{- end -}}
+
+{{/*
+Use *.domain.com as the Common Name in the certificate,
+so it can match Harbor service FQDN and Notary service FQDN.
+*/}}
+{{- define "harbor.certCommonName" -}}
+{{- $list := splitList "." .Values.externalDomain -}}
+{{- $list := prepend (rest $list) "*" -}}
+{{- $cn := join "." $list -}}
+{{- printf "%s" $cn -}}
+{{- end -}}
+
+{{/* The external FQDN of Notary server. */}}
+{{- define "harbor.notaryFQDN" -}}
+{{- printf "notary-%s" .Values.externalDomain -}}
+{{- end -}}
+
+{{/*
+The internal service name of Notary server.
+notary-server hostname is not configurable in Harbor 1.4.0.
+Once Harbor 1.5.x is released, use this instead:
+  {{- printf "%s-notary-server" (include "harbor.fullname") -}}
+*/}}
+{{- define "harbor.notaryServiceName" -}}
+{{- printf "%s" "notary-server" -}}
+{{- end -}}
--- a/contrib/helm/harbor/templates/adminserver/adminserver-cm.yaml
+++ b/contrib/helm/harbor/templates/adminserver/adminserver-cm.yaml
@ -27,6 +27,7 @@ data:
  REGISTRY_URL: "http://{{ template "harbor.fullname" . }}-registry:5000"
  TOKEN_SERVICE_URL: "http://{{ template "harbor.fullname" . }}-ui/service/token"
  WITH_NOTARY: "{{ .Values.notary.enabled }}"
+  NOTARY_URL: "http://{{ template "harbor.notaryServiceName" . }}:4443"
  LOG_LEVEL: "info"
  IMAGE_STORE_PATH: "/" # This is a temporary hack.
  AUTH_MODE: "{{ .Values.adminserver.authenticationMode }}"
--- a/contrib/helm/harbor/templates/ingress/ingress.yaml
+++ b/contrib/helm/harbor/templates/ingress/ingress.yaml
@ -11,6 +11,7 @@ spec:
  tls:
  - hosts:
    - "{{ .Values.externalDomain }}"
+    - "{{ template "harbor.notaryFQDN" . }}"
    secretName: "{{ template "harbor.fullname" . }}-ingress"
 {{ end }}
  rules:
@ -25,3 +26,10 @@ spec:
        backend:
          serviceName: {{ template "harbor.fullname" . }}-registry
          servicePort: 5000
+  - host: "{{ template "harbor.notaryFQDN" . }}"
+    http:
+      paths:
+      - path: /
+        backend:
+          serviceName: {{ template "harbor.notaryServiceName" . }}
+          servicePort: 4443
--- a/contrib/helm/harbor/templates/ingress/secret.yaml
+++ b/contrib/helm/harbor/templates/ingress/secret.yaml
@ -1,7 +1,7 @@
 {{ if not .Values.insecureRegistry }}
 {{ if .Values.generateCertificates }}
-{{ $ca := genCA "harbor-ca" 365 }}
-{{ $cert := genSignedCert .Values.externalDomain nil nil 365 $ca }}
+{{ $ca := genCA "harbor-ca" 3650 }}
+{{ $cert := genSignedCert (include "harbor.certCommonName" .) nil nil 3650 $ca }}
 apiVersion: v1
 kind: Secret
 metadata:
@ -14,4 +14,4 @@ data:
  tls.key: {{ .Values.tlsKey | default $cert.Key | b64enc | quote }}
  ca.crt: {{ .Values.caCrt | default $ca.Cert | b64enc | quote }}
 {{ end }}
-{{ end }}
+{{ end }}
--- a/contrib/helm/harbor/templates/notary/notary-cm.yaml
+++ b/contrib/helm/harbor/templates/notary/notary-cm.yaml
@ -0,0 +1,82 @@
+{{ if .Values.notary.enabled }}
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ template "harbor.fullname" . }}-notary-db
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+    component: notary-db
+data:
+  initial-notaryserver.sql: |
+    CREATE DATABASE IF NOT EXISTS `notaryserver`;
+    CREATE USER "server"@"%" IDENTIFIED BY "{{ .Values.notary.db.password }}";
+    GRANT ALL PRIVILEGES ON `notaryserver`.* TO "server"@"%"
+  initial-notarysigner.sql: |
+    CREATE DATABASE IF NOT EXISTS `notarysigner`;
+    CREATE USER "signer"@"%" IDENTIFIED BY "{{ .Values.notary.db.password }}";
+    GRANT ALL PRIVILEGES ON `notarysigner`.* TO "signer"@"%";
+
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ template "harbor.fullname" . }}-notary
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+    component: notary
+data:
+  {{ $ca := genCA "harbor-notary-ca" 3650 }}
+  {{ $cert := genSignedCert (printf "%s-notary-signer" (include "harbor.fullname" .)) nil nil 3650 $ca }}
+  notary-signer-ca.crt: |
+{{ .Values.notary.signer.caCrt  | default $ca.Cert   | indent 4 }}
+  notary-signer.crt: |
+{{ .Values.notary.signer.tlsCrt | default $cert.Cert | indent 4 }}
+  notary-signer.key: |
+{{ .Values.notary.signer.tlsKey | default $cert.Key  | indent 4 }}
+  server-config.json: |
+    {
+      "server": {
+        "http_addr": ":4443"
+      },
+      "trust_service": {
+        "type": "remote",
+        "hostname": "{{ template "harbor.fullname" . }}-notary-signer",
+        "port": "7899",
+        "tls_ca_file": "/config/notary-signer-ca.crt",
+        "key_algorithm": "ecdsa"
+      },
+      "logging": {
+        "level": "debug"
+      },
+      "storage": {
+        "backend": "mysql",
+        "db_url": "server:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notaryserver?parseTime=True"
+      },
+      "auth": {
+          "type": "token",
+          "options": {
+              "realm": "https://{{ .Values.externalDomain }}/service/token",
+              "service": "harbor-notary",
+              "issuer": "harbor-token-issuer",
+              "rootcertbundle": "/config/root.crt"
+          }
+      }
+    }
+  signer-config.json: |
+    {
+      "server": {
+        "grpc_addr": ":7899",
+        "tls_cert_file": "./notary-signer.crt",
+        "tls_key_file": "./notary-signer.key"
+      },
+      "logging": {
+        "level": "debug"
+      },
+      "storage": {
+        "backend": "mysql",
+        "db_url": "signer:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notarysigner?parseTime=True",
+        "default_alias": "defaultalias"
+      }
+    }
+{{ end }}
--- a/contrib/helm/harbor/templates/notary/notary-db.yaml
+++ b/contrib/helm/harbor/templates/notary/notary-db.yaml
@ -0,0 +1,62 @@
+{{ if .Values.notary.enabled }}
+apiVersion: apps/v1beta2
+kind: StatefulSet
+metadata:
+  name: {{ template "harbor.fullname" . }}-notary-db
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+    component: notary-db
+spec:
+  replicas: 1
+  serviceName: "{{ template "harbor.fullname" . }}-notary-db"
+  selector:
+    matchLabels:
+{{ include "harbor.matchLabels" . | indent 6 }}
+      component: notary-db
+  template:
+    metadata:
+      labels:
+{{ include "harbor.labels" . | indent 8 }}
+        component: notary-db
+    spec:
+      containers:
+      - name: notary-db
+        image: {{ .Values.notary.db.image.repository }}:{{ .Values.notary.db.image.tag }}
+        imagePullPolicy: {{ .Values.notary.db.image.pullPolicy }}
+        args: ["--innodb_file_per_table"]
+        env:
+          - name: TERM
+            value: "dumb"
+          - name: MYSQL_ALLOW_EMPTY_PASSWORD
+            value: "true"
+        resources:
+{{ toYaml .Values.notary.db.resources | indent 10 }}
+        volumeMounts:
+        - name: notary-db-config
+          mountPath: /docker-entrypoint-initdb.d
+        - name: notary-db
+          mountPath: /var/lib/mysql
+      volumes:
+      - name: notary-db-config
+        configMap:
+          name: "{{ template "harbor.fullname" . }}-notary-db"
+      {{- if not .Values.persistence.enabled }}
+      - name: notary-db
+        emptyDir: {}
+      {{- end -}}
+  {{- if .Values.persistence.enabled }}
+  volumeClaimTemplates:
+  - metadata:
+      name: notary-db
+    spec:
+      accessModes: [{{ .Values.notary.db.volumes.data.accessMode | quote }}]
+      {{- if (eq "-" .Values.notary.db.volumes.data.storageClass) }}
+      storageClassName: ""
+      {{- else }}
+      storageClassName: "{{ .Values.notary.db.volumes.data.storageClass }}"
+      {{- end }}
+      resources:
+        requests:
+          storage: {{ .Values.adminserver.volumes.data.size | quote }}
+  {{- end -}}
+{{ end }}
--- a/contrib/helm/harbor/templates/notary/notary-server.yaml
+++ b/contrib/helm/harbor/templates/notary/notary-server.yaml
@ -0,0 +1,43 @@
+{{ if .Values.notary.enabled }}
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: {{ template "harbor.fullname" . }}-notary-server
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+    component: notary-server
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+{{ include "harbor.matchLabels" . | indent 6 }}
+      component: notary-server
+  template:
+    metadata:
+      labels:
+{{ include "harbor.labels" . | indent 8 }}
+        component: notary-server
+    spec:
+      containers:
+      - name: notary-server
+        image: {{ .Values.notary.server.image.repository }}:{{ .Values.notary.server.image.tag }}
+        imagePullPolicy: {{ .Values.notary.server.image.pullPolicy }}
+        resources:
+{{ toYaml .Values.notary.server.resources | indent 10 }}
+        env:
+        - name: DB_URL
+          value: "mysql://server:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notaryserver?parseTime=True"
+        volumeMounts:
+        - name: notary-config
+          mountPath: /config
+        - name: root-certificate
+          mountPath: /config/root.crt
+          subPath: root.crt
+      volumes:
+      - name: notary-config
+        configMap:
+          name: "{{ template "harbor.fullname" . }}-notary"
+      - name: root-certificate
+        secret:
+          secretName: "{{ template "harbor.fullname" . }}-registry"
+{{ end }}
--- a/contrib/helm/harbor/templates/notary/notary-signer.yaml
+++ b/contrib/helm/harbor/templates/notary/notary-signer.yaml
@ -0,0 +1,39 @@
+{{ if .Values.notary.enabled }}
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: {{ template "harbor.fullname" . }}-notary-signer
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+    component: notary-signer
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+{{ include "harbor.matchLabels" . | indent 6 }}
+      component: notary-signer
+  template:
+    metadata:
+      labels:
+{{ include "harbor.labels" . | indent 8 }}
+        component: notary-signer
+    spec:
+      containers:
+      - name: notary-signer
+        image: {{ .Values.notary.signer.image.repository }}:{{ .Values.notary.signer.image.tag }}
+        imagePullPolicy: {{ .Values.notary.signer.image.pullPolicy }}
+        resources:
+{{ toYaml .Values.notary.signer.resources | indent 10 }}
+        env:
+        - name: DB_URL
+          value: "mysql://signer:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notarysigner?parseTime=True"
+        - name: NOTARY_SIGNER_DEFAULTALIAS
+          value: {{ .Values.notary.signer.env.NOTARY_SIGNER_DEFAULTALIAS }}
+        volumeMounts:
+        - name: notary-config
+          mountPath: /config
+      volumes:
+      - name: notary-config
+        configMap:
+          name: "{{ template "harbor.fullname" . }}-notary"
+{{ end }}
--- a/contrib/helm/harbor/templates/notary/notary-svc.yaml
+++ b/contrib/helm/harbor/templates/notary/notary-svc.yaml
@ -0,0 +1,43 @@
+{{ if .Values.notary.enabled }}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ template "harbor.fullname" . }}-notary-db
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+spec:
+  ports:
+  - port: 3306
+  selector:
+{{ include "harbor.matchLabels" . | indent 4 }}
+    component: notary-db
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ template "harbor.notaryServiceName" . }}
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+spec:
+  ports:
+  - port: 4443
+  selector:
+{{ include "harbor.matchLabels" . | indent 4 }}
+    component: notary-server
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ template "harbor.fullname" . }}-notary-signer
+  labels:
+{{ include "harbor.labels" . | indent 4 }}
+spec:
+  ports:
+  - port: 7899
+  selector:
+{{ include "harbor.matchLabels" . | indent 4 }}
+    component: notary-signer
+{{ end }}
--- a/contrib/helm/harbor/values.yaml
+++ b/contrib/helm/harbor/values.yaml
@ -50,8 +50,8 @@ secretKey: not-a-secure-key
 ingress:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
-    ingress.kubernetes.io/body-size: "0"
    ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"

 adminserver:
  image:
@ -285,12 +285,6 @@ clair:
  #    memory: 256Mi
  #    cpu: 100m

-## Notary support is not yet fully implemented in the Helm Charts
-## Enabling it will just break things.
-#
-notary:
-  enabled: false
-
 ## Settings for postgresql dependency.
 ## see https://github.com/kubernetes/charts/tree/master/stable/postgresql
 ## for further configurables.
@ -300,3 +294,37 @@ postgresql:
  postgresDatabase: clair
  persistence:
    enabled: false
+
+notary:
+  enabled: true
+  server:
+    image:
+      repository: vmware/notary-server-photon
+      tag: v0.5.1-v1.4.0
+      pullPolicy: IfNotPresent
+  signer:
+    image:
+      repository: vmware/notary-signer-photon
+      tag: v0.5.1-v1.4.0
+      pullPolicy: IfNotPresent
+    env:
+      NOTARY_SIGNER_DEFAULTALIAS: defaultalias
+    # The TLS certificate for Notary Signer. Will auto generate them if unspecified here.
+    caCrt:
+    tlsCrt:
+    tlsKey:
+  db:
+    image:
+      repository: vmware/mariadb-photon
+      tag: *harbor_image_tag
+      pullPolicy: IfNotPresent
+    password: not-a-secure-password
+    volumes:
+      data:
+        # storageClass: "-"
+        accessMode: ReadWriteOnce
+        size: 1Gi
+    # resources:
+    #  requests:
+    #    memory: 256Mi
+    #    cpu: 100m