[Chart] Add support for deploying Notary

Assuming Harbor FQDN is specified as harbor.my.domain by the user,
the Notary server FQDN will be set to notary-harbor.my.domain.
So the user must point both Harbor FQDN and Notary FQDN to the K8s
ingress controller IP in the DNS server.

Fix issue https://github.com/vmware/harbor/issues/4394
This commit is contained in:
Jesse Hu 2018-04-18 18:08:17 +08:00
parent 9d13842a29
commit ce48dbfec2
12 changed files with 365 additions and 19 deletions

View File

@ -176,24 +176,32 @@ The following tables lists the configurable parameters of the Harbor chart and t
| `registry.resources` | [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) to allocate for container | undefined | | `registry.resources` | [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) to allocate for container | undefined |
| `registry.volumes` | used to create PVCs if persistence is enabled (see instructions in values.yaml) | see values.yaml | | `registry.volumes` | used to create PVCs if persistence is enabled (see instructions in values.yaml) | see values.yaml |
| **Clair** | | **Clair** |
| `clair.enabled` | Enable clair? | `true` | | `clair.enabled` | Enable Clair? | `true` |
| `clair.image.repository` | Repository for clair image | `vmware/clair-photon` | | `clair.image.repository` | Repository for clair image | `vmware/clair-photon` |
| `clair.image.tag` | Tag for clair image | `v2.0.1-v1.4.0` | `clair.image.tag` | Tag for clair image | `v2.0.1-v1.4.0`
| `clair.resources` | [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) to allocate for container | undefined | `clair.resources` | [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) to allocate for container | undefined
| `postgresql` | Overrides for postgresql chart [values.yaml](https://github.com/kubernetes/charts/blob/f2938a46e3ae8e2512ede1142465004094c3c333/stable/postgresql/values.yaml) | see values.yaml | `postgresql` | Overrides for postgresql chart [values.yaml](https://github.com/kubernetes/charts/blob/f2938a46e3ae8e2512ede1142465004094c3c333/stable/postgresql/values.yaml) | see values.yaml
| | | | | **Notary** |
| `notary.enabled` | Enable Notary? | `true` |
| `notary.server.image.repository` | Repository for notary server image | `vmware/notary-server-photon` |
| `notary.server.image.tag` | Tag for notary server image | `v0.5.1-v1.4.0`
| `notary.signer.image.repository` | Repository for notary signer image | `vmware/notary-signer-photon` |
| `notary.signer.image.tag` | Tag for notary signer image | `v0.5.1-v1.4.0`
| `notary.db.image.repository` | Repository for notary database image | `vmware/mariadb-photon` |
| `notary.db.image.tag` | Tag for notary database image | `v1.4.0`
| `notary.db.password` | The password of users for notary database | Specify your own password |
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example: Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example:
```bash ```bash
helm install --name my-release --set mysql.pass=baconeggs . helm install . --name my-release --set externalDomain=harbor.<IP>.xip.io
``` ```
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example, Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
```bash ```bash
helm install --name my-release -f /path/to/values.yaml . helm install . --name my-release -f /path/to/values.yaml
``` ```
> **Tip**: You can use the default [values.yaml](values.yaml) > **Tip**: You can use the default [values.yaml](values.yaml)

View File

@ -1,5 +1,8 @@
Add the Harbor CA certificate to Docker by executing the following command: Please wait for several minutes for Harbor deployment to complete.
Then follow the steps below to use Harbor.
1. Add the Harbor CA certificate to Docker by executing the following command:
sudo mkdir -p /etc/docker/certs.d/{{ .Values.externalDomain }} sudo mkdir -p /etc/docker/certs.d/{{ .Values.externalDomain }}
kubectl get secret \ kubectl get secret \
@ -7,14 +10,17 @@ Add the Harbor CA certificate to Docker by executing the following command:
-o jsonpath="{.data.ca\.crt}" | base64 --decode | \ -o jsonpath="{.data.ca\.crt}" | base64 --decode | \
sudo tee /etc/docker/certs.d/{{ .Values.externalDomain }}/ca.crt sudo tee /etc/docker/certs.d/{{ .Values.externalDomain }}/ca.crt
Get Harbor admin password by executing the following command: 2. Get Harbor admin password by executing the following command:
kubectl get secret --namespace {{ .Release.Namespace }} {{ template "harbor.fullname" . }}-adminserver -o jsonpath="{.data.HARBOR_ADMIN_PASSWORD}" | base64 --decode; echo kubectl get secret --namespace {{ .Release.Namespace }} {{ template "harbor.fullname" . }}-adminserver -o jsonpath="{.data.HARBOR_ADMIN_PASSWORD}" | base64 --decode; echo
Add Harbor FQDN {{ .Values.externalDomain }} to K8s Ingress Controller IP resolution on DNS Server or in file /etc/hosts. 3. Add DNS resolution entry for Harbor FQDN {{ .Values.externalDomain }} to K8s Ingress Controller IP on DNS Server or in file /etc/hosts.
{{- if .Values.notary.enabled }}
Add DNS resolution entry for Notary FQDN {{ template "harbor.notaryFQDN" . }} to K8s Ingress Controller IP on DNS Server or in file /etc/hosts.
{{- end }}
Access Harbor UI via https://{{ .Values.externalDomain }} 4. Access Harbor UI via https://{{ .Values.externalDomain }}
Login Harbor with Docker CLI: 5. Login Harbor with Docker CLI:
docker login {{ .Values.externalDomain }} docker login {{ .Values.externalDomain }}

View File

@ -29,3 +29,29 @@ app: "{{ template "harbor.name" . }}"
release: {{ .Release.Name }} release: {{ .Release.Name }}
app: "{{ template "harbor.name" . }}" app: "{{ template "harbor.name" . }}"
{{- end -}} {{- end -}}
{{/*
Use *.domain.com as the Common Name in the certificate,
so it can match Harbor service FQDN and Notary service FQDN.
*/}}
{{- define "harbor.certCommonName" -}}
{{- $list := splitList "." .Values.externalDomain -}}
{{- $list := prepend (rest $list) "*" -}}
{{- $cn := join "." $list -}}
{{- printf "%s" $cn -}}
{{- end -}}
{{/* The external FQDN of Notary server. */}}
{{- define "harbor.notaryFQDN" -}}
{{- printf "notary-%s" .Values.externalDomain -}}
{{- end -}}
{{/*
The internal service name of Notary server.
notary-server hostname is not configurable in Harbor 1.4.0.
Once Harbor 1.5.x is released, use this instead:
{{- printf "%s-notary-server" (include "harbor.fullname") -}}
*/}}
{{- define "harbor.notaryServiceName" -}}
{{- printf "%s" "notary-server" -}}
{{- end -}}

View File

@ -27,6 +27,7 @@ data:
REGISTRY_URL: "http://{{ template "harbor.fullname" . }}-registry:5000" REGISTRY_URL: "http://{{ template "harbor.fullname" . }}-registry:5000"
TOKEN_SERVICE_URL: "http://{{ template "harbor.fullname" . }}-ui/service/token" TOKEN_SERVICE_URL: "http://{{ template "harbor.fullname" . }}-ui/service/token"
WITH_NOTARY: "{{ .Values.notary.enabled }}" WITH_NOTARY: "{{ .Values.notary.enabled }}"
NOTARY_URL: "http://{{ template "harbor.notaryServiceName" . }}:4443"
LOG_LEVEL: "info" LOG_LEVEL: "info"
IMAGE_STORE_PATH: "/" # This is a temporary hack. IMAGE_STORE_PATH: "/" # This is a temporary hack.
AUTH_MODE: "{{ .Values.adminserver.authenticationMode }}" AUTH_MODE: "{{ .Values.adminserver.authenticationMode }}"

View File

@ -11,6 +11,7 @@ spec:
tls: tls:
- hosts: - hosts:
- "{{ .Values.externalDomain }}" - "{{ .Values.externalDomain }}"
- "{{ template "harbor.notaryFQDN" . }}"
secretName: "{{ template "harbor.fullname" . }}-ingress" secretName: "{{ template "harbor.fullname" . }}-ingress"
{{ end }} {{ end }}
rules: rules:
@ -25,3 +26,10 @@ spec:
backend: backend:
serviceName: {{ template "harbor.fullname" . }}-registry serviceName: {{ template "harbor.fullname" . }}-registry
servicePort: 5000 servicePort: 5000
- host: "{{ template "harbor.notaryFQDN" . }}"
http:
paths:
- path: /
backend:
serviceName: {{ template "harbor.notaryServiceName" . }}
servicePort: 4443

View File

@ -1,7 +1,7 @@
{{ if not .Values.insecureRegistry }} {{ if not .Values.insecureRegistry }}
{{ if .Values.generateCertificates }} {{ if .Values.generateCertificates }}
{{ $ca := genCA "harbor-ca" 365 }} {{ $ca := genCA "harbor-ca" 3650 }}
{{ $cert := genSignedCert .Values.externalDomain nil nil 365 $ca }} {{ $cert := genSignedCert (include "harbor.certCommonName" .) nil nil 3650 $ca }}
apiVersion: v1 apiVersion: v1
kind: Secret kind: Secret
metadata: metadata:
@ -14,4 +14,4 @@ data:
tls.key: {{ .Values.tlsKey | default $cert.Key | b64enc | quote }} tls.key: {{ .Values.tlsKey | default $cert.Key | b64enc | quote }}
ca.crt: {{ .Values.caCrt | default $ca.Cert | b64enc | quote }} ca.crt: {{ .Values.caCrt | default $ca.Cert | b64enc | quote }}
{{ end }} {{ end }}
{{ end }} {{ end }}

View File

@ -0,0 +1,82 @@
{{ if .Values.notary.enabled }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "harbor.fullname" . }}-notary-db
labels:
{{ include "harbor.labels" . | indent 4 }}
component: notary-db
data:
initial-notaryserver.sql: |
CREATE DATABASE IF NOT EXISTS `notaryserver`;
CREATE USER "server"@"%" IDENTIFIED BY "{{ .Values.notary.db.password }}";
GRANT ALL PRIVILEGES ON `notaryserver`.* TO "server"@"%"
initial-notarysigner.sql: |
CREATE DATABASE IF NOT EXISTS `notarysigner`;
CREATE USER "signer"@"%" IDENTIFIED BY "{{ .Values.notary.db.password }}";
GRANT ALL PRIVILEGES ON `notarysigner`.* TO "signer"@"%";
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "harbor.fullname" . }}-notary
labels:
{{ include "harbor.labels" . | indent 4 }}
component: notary
data:
{{ $ca := genCA "harbor-notary-ca" 3650 }}
{{ $cert := genSignedCert (printf "%s-notary-signer" (include "harbor.fullname" .)) nil nil 3650 $ca }}
notary-signer-ca.crt: |
{{ .Values.notary.signer.caCrt | default $ca.Cert | indent 4 }}
notary-signer.crt: |
{{ .Values.notary.signer.tlsCrt | default $cert.Cert | indent 4 }}
notary-signer.key: |
{{ .Values.notary.signer.tlsKey | default $cert.Key | indent 4 }}
server-config.json: |
{
"server": {
"http_addr": ":4443"
},
"trust_service": {
"type": "remote",
"hostname": "{{ template "harbor.fullname" . }}-notary-signer",
"port": "7899",
"tls_ca_file": "/config/notary-signer-ca.crt",
"key_algorithm": "ecdsa"
},
"logging": {
"level": "debug"
},
"storage": {
"backend": "mysql",
"db_url": "server:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notaryserver?parseTime=True"
},
"auth": {
"type": "token",
"options": {
"realm": "https://{{ .Values.externalDomain }}/service/token",
"service": "harbor-notary",
"issuer": "harbor-token-issuer",
"rootcertbundle": "/config/root.crt"
}
}
}
signer-config.json: |
{
"server": {
"grpc_addr": ":7899",
"tls_cert_file": "./notary-signer.crt",
"tls_key_file": "./notary-signer.key"
},
"logging": {
"level": "debug"
},
"storage": {
"backend": "mysql",
"db_url": "signer:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notarysigner?parseTime=True",
"default_alias": "defaultalias"
}
}
{{ end }}

View File

@ -0,0 +1,62 @@
{{ if .Values.notary.enabled }}
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: {{ template "harbor.fullname" . }}-notary-db
labels:
{{ include "harbor.labels" . | indent 4 }}
component: notary-db
spec:
replicas: 1
serviceName: "{{ template "harbor.fullname" . }}-notary-db"
selector:
matchLabels:
{{ include "harbor.matchLabels" . | indent 6 }}
component: notary-db
template:
metadata:
labels:
{{ include "harbor.labels" . | indent 8 }}
component: notary-db
spec:
containers:
- name: notary-db
image: {{ .Values.notary.db.image.repository }}:{{ .Values.notary.db.image.tag }}
imagePullPolicy: {{ .Values.notary.db.image.pullPolicy }}
args: ["--innodb_file_per_table"]
env:
- name: TERM
value: "dumb"
- name: MYSQL_ALLOW_EMPTY_PASSWORD
value: "true"
resources:
{{ toYaml .Values.notary.db.resources | indent 10 }}
volumeMounts:
- name: notary-db-config
mountPath: /docker-entrypoint-initdb.d
- name: notary-db
mountPath: /var/lib/mysql
volumes:
- name: notary-db-config
configMap:
name: "{{ template "harbor.fullname" . }}-notary-db"
{{- if not .Values.persistence.enabled }}
- name: notary-db
emptyDir: {}
{{- end -}}
{{- if .Values.persistence.enabled }}
volumeClaimTemplates:
- metadata:
name: notary-db
spec:
accessModes: [{{ .Values.notary.db.volumes.data.accessMode | quote }}]
{{- if (eq "-" .Values.notary.db.volumes.data.storageClass) }}
storageClassName: ""
{{- else }}
storageClassName: "{{ .Values.notary.db.volumes.data.storageClass }}"
{{- end }}
resources:
requests:
storage: {{ .Values.adminserver.volumes.data.size | quote }}
{{- end -}}
{{ end }}

View File

@ -0,0 +1,43 @@
{{ if .Values.notary.enabled }}
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "harbor.fullname" . }}-notary-server
labels:
{{ include "harbor.labels" . | indent 4 }}
component: notary-server
spec:
replicas: 1
selector:
matchLabels:
{{ include "harbor.matchLabels" . | indent 6 }}
component: notary-server
template:
metadata:
labels:
{{ include "harbor.labels" . | indent 8 }}
component: notary-server
spec:
containers:
- name: notary-server
image: {{ .Values.notary.server.image.repository }}:{{ .Values.notary.server.image.tag }}
imagePullPolicy: {{ .Values.notary.server.image.pullPolicy }}
resources:
{{ toYaml .Values.notary.server.resources | indent 10 }}
env:
- name: DB_URL
value: "mysql://server:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notaryserver?parseTime=True"
volumeMounts:
- name: notary-config
mountPath: /config
- name: root-certificate
mountPath: /config/root.crt
subPath: root.crt
volumes:
- name: notary-config
configMap:
name: "{{ template "harbor.fullname" . }}-notary"
- name: root-certificate
secret:
secretName: "{{ template "harbor.fullname" . }}-registry"
{{ end }}

View File

@ -0,0 +1,39 @@
{{ if .Values.notary.enabled }}
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "harbor.fullname" . }}-notary-signer
labels:
{{ include "harbor.labels" . | indent 4 }}
component: notary-signer
spec:
replicas: 1
selector:
matchLabels:
{{ include "harbor.matchLabels" . | indent 6 }}
component: notary-signer
template:
metadata:
labels:
{{ include "harbor.labels" . | indent 8 }}
component: notary-signer
spec:
containers:
- name: notary-signer
image: {{ .Values.notary.signer.image.repository }}:{{ .Values.notary.signer.image.tag }}
imagePullPolicy: {{ .Values.notary.signer.image.pullPolicy }}
resources:
{{ toYaml .Values.notary.signer.resources | indent 10 }}
env:
- name: DB_URL
value: "mysql://signer:{{ .Values.notary.db.password }}@tcp({{ template "harbor.fullname" . }}-notary-db:3306)/notarysigner?parseTime=True"
- name: NOTARY_SIGNER_DEFAULTALIAS
value: {{ .Values.notary.signer.env.NOTARY_SIGNER_DEFAULTALIAS }}
volumeMounts:
- name: notary-config
mountPath: /config
volumes:
- name: notary-config
configMap:
name: "{{ template "harbor.fullname" . }}-notary"
{{ end }}

View File

@ -0,0 +1,43 @@
{{ if .Values.notary.enabled }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ template "harbor.fullname" . }}-notary-db
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
ports:
- port: 3306
selector:
{{ include "harbor.matchLabels" . | indent 4 }}
component: notary-db
---
apiVersion: v1
kind: Service
metadata:
name: {{ template "harbor.notaryServiceName" . }}
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
ports:
- port: 4443
selector:
{{ include "harbor.matchLabels" . | indent 4 }}
component: notary-server
---
apiVersion: v1
kind: Service
metadata:
name: {{ template "harbor.fullname" . }}-notary-signer
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
ports:
- port: 7899
selector:
{{ include "harbor.matchLabels" . | indent 4 }}
component: notary-signer
{{ end }}

View File

@ -50,8 +50,8 @@ secretKey: not-a-secure-key
ingress: ingress:
annotations: annotations:
ingress.kubernetes.io/ssl-redirect: "true" ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/body-size: "0"
ingress.kubernetes.io/proxy-body-size: "0" ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
adminserver: adminserver:
image: image:
@ -285,12 +285,6 @@ clair:
# memory: 256Mi # memory: 256Mi
# cpu: 100m # cpu: 100m
## Notary support is not yet fully implemented in the Helm Charts
## Enabling it will just break things.
#
notary:
enabled: false
## Settings for postgresql dependency. ## Settings for postgresql dependency.
## see https://github.com/kubernetes/charts/tree/master/stable/postgresql ## see https://github.com/kubernetes/charts/tree/master/stable/postgresql
## for further configurables. ## for further configurables.
@ -300,3 +294,37 @@ postgresql:
postgresDatabase: clair postgresDatabase: clair
persistence: persistence:
enabled: false enabled: false
notary:
enabled: true
server:
image:
repository: vmware/notary-server-photon
tag: v0.5.1-v1.4.0
pullPolicy: IfNotPresent
signer:
image:
repository: vmware/notary-signer-photon
tag: v0.5.1-v1.4.0
pullPolicy: IfNotPresent
env:
NOTARY_SIGNER_DEFAULTALIAS: defaultalias
# The TLS certificate for Notary Signer. Will auto generate them if unspecified here.
caCrt:
tlsCrt:
tlsKey:
db:
image:
repository: vmware/mariadb-photon
tag: *harbor_image_tag
pullPolicy: IfNotPresent
password: not-a-secure-password
volumes:
data:
# storageClass: "-"
accessMode: ReadWriteOnce
size: 1Gi
# resources:
# requests:
# memory: 256Mi
# cpu: 100m