From a8dc8d784f880e3931469b048ad50af23e5d9545 Mon Sep 17 00:00:00 2001 From: yixingj Date: Fri, 26 Jan 2018 14:11:39 +0800 Subject: [PATCH] Update HA document. Format the docs --- docs/high_availability_installation_guide.md | 257 ++++++++++++------- 1 file changed, 171 insertions(+), 86 deletions(-) diff --git a/docs/high_availability_installation_guide.md b/docs/high_availability_installation_guide.md index 12703d1cd..4fa115f9d 100644 --- a/docs/high_availability_installation_guide.md +++ b/docs/high_availability_installation_guide.md @@ -1,30 +1,23 @@ # Harbor High Availability Guide -## Abstract - -This guide describes how to install and configure Harbor for high availability. It supplements the Installation Guides and assumes that you are familiar with the material in those guides. - -**Important!** - -This guide was last updated as of the Harbor 1.4 release. It will not apply to releases before 1.4. We advise that you read this at your own discretion when planning on your Harbor high availability implementation. - -This guide is intended as advice only. It doesn't support Notary(Content trust functionality) for now. - ## Contents **Notices** Notices take these forms: -**Note!**: A comment with additional information that explains a part of the text. +**Note**: A comment with additional information that explains a part of the text. -**Important!**: Something you must be aware of before proceeding. +**Important**: Something you must be aware of before proceeding. -**Tip!**: An extra but helpful piece of practical advice. +**Tip**: An extra but helpful piece of practical advice. -**Caution!**: Helpful information that prevents the user from making mistakes. +## Abstract -**Warning!**: Critical information about the risk of data loss or security issues. +This guide describes how to install and configure Harbor for high availability. It supplements the [Installation Guides](installation_guide.md) and assumes that you are familiar with the material in those guides. +**Important** This guide was last updated as of the Harbor 1.4.0 release. It will not apply to releases before 1.4.0. We advise that you read this at your own discretion when planning on your Harbor high availability implementation. + +This guide is intended as advice only. ## Harbor High Availability Introduction @@ -49,17 +42,18 @@ Stateful services are more difficult to manage. Providing additional instances a - Clair database(PostgresSQL) - Notary database(MariaDB) - Redis -Making those services highly available can depend the high available implementation for each application. + +Making those services highly available can depend the high availability implementation for each application. ## High Availability Architecture Again, this architecture is for advice only. ![HA](img/ha/Architecture.png) -As seen in the figure, components involved in the architecture are: +As you can seen in the above figure, components involved in the architecture are: -**VIP**: Virtual IP. The Harbor user will access Harbor through this virtual IP address. This VIP will only active on one load balancer node at the same time. It will automatically switch to the other node if the active loadbalancer node is down. +**VIP**: [Virtual IP](https://en.wikipedia.org/wiki/Virtual_IP_address) The Harbor user will access Harbor through this virtual IP address. This VIP will only active on one load balancer node at the same time. It will automatically switch to the other node if the active loadbalancer node is down. -**LoadBalancer 01 and 02**: They together compose as a group which avoid single point failure of load balancer nodes. Keepalived is installed on both load balancer nodes. The two Keepalived instance will form a VRRP group to provide the VIP and ensure the VIP only shows on one node at the same time. The LVS component in Keepalived is responsible for balance the requests between different Harbor servers according to the routing algorithm. +**LoadBalancer 01 and 02**: They together compose as a group which avoid single point failure of load balancer nodes. [Keepalived](www.keepalived.org) is installed on both load balancer nodes. The two Keepalived instances will form a VRRP group to provide the VIP and ensure the VIP only shows on one node at the same time. The LVS component in Keepalived is responsible for balance the requests between different Harbor servers according to the routing algorithm. **Harbor server 1..n**: These are the running Harbor instances. They are in active-active mode. User can setup multiple nodes according to their workload. @@ -71,63 +65,74 @@ As seen in the figure, components involved in the architecture are: **Redis**: The purpose of having Redis is to store UI session data and store the registry metadata cache. When one Harbor instance fails or the load balancer routes a user request to another Harbor instance, any Harbor instance can query the Redis to retrieve session information to make sure the end-user has a continued session. User should follow the best practice of Redis to make it HA protected. -From the above high availability architecture, we can see that we don't setup LB per stateless services. Instead we group those stateless service as a group. The communicate between each services are protected by host based docker network with isolation. **Please Note!** As the component communicate with each other through rest API. You can always define the group granularity according to your use scenarios. +From the above high availability architecture, we can see that we don't setup LB per stateless services. Instead we group those stateless service as a group. The communicate between each services are protected by host based docker network with isolation. **Note** As the component communicate with each other through rest API. You can always define the group granularity according to your use scenarios. ### Limitation -Currently Harbor doesn’t support Notary in HA scenario. Which means Content trust functionality is not supported in this HA setup. +Currently Harbor doesn’t support Notary in HA scenario. Which means Content trust functionality is not supported in this HA setup. ## Setup -Follow the setup instruction in this section we build a Harbor high availability deployment as the follow figure shows. You can setup more Harbor nodes if needed. +Follow the setup instruction in this section we can build a Harbor high availability deployment as the follow figure shows. You can setup more Harbor nodes if needed. ![setup](img/ha/LabInstallation.png) ### Prerequisites - 1> MariaDB cluster(Harbor-DB,192.168.1.215, currently Harbor use MariaDB 10.2.10) - - 2> Shared Storage (Swift Server. 192.168.1.216. **Important!** The shared storage is replaceable you can choose other shared storage, just need to make sure the storage you used is supported by registry https://docs.docker.com/registry/storage-drivers) + - 2> Shared Storage (Swift Server. 192.168.1.216) - 3> Redis cluster(192.168.1.217) - - 4> PostgreSQL (Clair DB 192.168.1.50. Note! The PostgreSQL is optional, it only needed when you use the vulnerability scanning function, currently harbor use PostgreSQL 9.6.5) - -**Important!** Item 1,2,3,4 are considered external components to Harbor. Before configuring Harbor HA, we assume these components are present and all of them are HA protected. Otherwise, any of these components can be a single point of failure. - -**Tips!** - -If you are only setting up the HA for POC purpose. You can use docker to run MariaDB, Redis and PostgreSQL in one OS with follow command. -``` -#> docker run --name redis-server -p 6379:6379 -d redis -#> docker run -d --restart=always -e MYSQL_ROOT_PASSWORD=root123 -v /data/database:/var/lib/mysql:z -p 3306:3306 --name mariadb vmware/mariadb-photon:10.2.10 -#> docker run -d -e POSTGRES_PASSWORD="password" -p 5432:5432 postgres:9.6 -``` + - 4> PostgreSQL (Clair DB 192.168.1.50) - 5> 2 VMs for Load balancer cluster. - 6> n VMs for Harbor stateless services (n >=2), in this example we will set up 2 Harbor nodes. - 7> n+1 static IPs (1 for VIP and the other n IPs will be used by Harbor stateless servers) -### Load balancer setup +**Important** Item 1,2,3,4 are statefull components to Harbor. Before configuring Harbor HA, we assume these components are present and all of them are HA protected. Otherwise, any of these components can be a single point of failure. -Here we will use Ubuntu16.04 LTS as the Load balancer cluster server. You can choose other Linux OS if needed. +The shared storage is replaceable you can choose other shared storage, just need to make sure the storage you used is supported by registry https://docs.docker.com/registry/storage-drivers +The PostgreSQL is optional, it only needed when you use the vulnerability scanning function, currently harbor use PostgreSQL 9.6.5 + +**Tips** + +If you are only setting up the HA for POC purpose. You can use docker to run MariaDB, Redis and PostgreSQL in one OS with follow commands. +``` +#> docker run --name redis-server -p 6379:6379 -d redis +#> docker run -d --restart=always -e MYSQL_ROOT_PASSWORD=root123 -v /data/database:/var/lib/mysql:z -p 3306:3306 --name mariadb vmware/mariadb-photon:10.2.10 +#> docker run -d -e POSTGRES_PASSWORD="password" -p 5432:5432 postgres:9.6 +``` + +### Load Harbor DB schema + +Import Harbor database schema to your external MariaDB + +#### 1> Login to a machine that has MariaDB client installed +#### 2> Save the [Harbor DB Schema](https://github.com/vmware/harbor/blob/release-1.4.0/make/common/db/registry.sql) to ```registry.sql``` +#### 3> Load the schema +``` +#> mysql -u your_db_username -p -h your_db_ip < registry.sql +``` + +### Load balancer setup +As all the Harbor nodes are in active status. A loadbancer will be needed to efficiently distributing incoming requests between the Harbor nodes. You can choose either hardware loadbalancer or software loadbalancer at your convenient. + +Here we will use Ubuntu16.04 + Keepalived to build up a software loadbalancer. #### On Loadbalancer01: - - 1> install the Keepalived application. +##### 1> Install Keepalived and curl applications. +Curl will be used in the keepalived check script. ``` #> apt-get install keepalived curl ``` - - 2> Config Keepalived +##### 2> Config Keepalived -Save the configuration file sample -https://github.com/vmware/harbor/blob/master/make/ha/sample/active_active/keepalived_active_active.conf +Save the [Keepalived configuration file](https://github.com/vmware/harbor/blob/release-1.4.0/make/ha/sample/active_active/keepalived_active_active.conf) to ```/etc/keepalived/keepalived.conf``` -to -```/etc/keepalived/keepalived.conf``` - - -**Important!**: +**Important** You need to change the **** to the real IP address.(There are two place.) @@ -135,11 +140,14 @@ Change **harbor_node1_IP** (Two places) and **harbor_node2_IP** (Two places)to t If you have more than two nodes then please add more real_server definitions in the keepalived.conf - - 3> Save the server [health check](https://github.com/vmware/harbor/blob/master/make/ha/sample/active_active/check.sh) script to ```/usr/local/bin/check.sh``` +##### 3> Configure health check +Save the server [health check](https://github.com/vmware/harbor/blob/release-1.4.0/make/ha/sample/active_active/check.sh) script to ```/usr/local/bin/check.sh``` - run ```chmod +x /usr/local/bin/check.sh``` to give the file execute privilege. +Run the follow command to add the execute privilege. - - 4> Enable ip forward +```#> chmod +x /usr/local/bin/check.sh``` + +##### 4> Enable ip forward ``` add the follow two lines to /etc/sysctl.conf @@ -147,13 +155,13 @@ add the follow two lines to /etc/sysctl.conf net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1 -run the follow command to apply the change. +Run the follow command to apply the change. #>sysctl -p ``` - - 5> Restart the Keepalived service to make the change take affect. +##### 5> Restart the Keepalived service. ``` #>systemctl restart keepalived @@ -164,36 +172,97 @@ run the follow command to apply the change. Follow the same steps 1 to 5 as Loadbalancer01 list, only change the ```priority``` to 20 in the /etc/keepalived/keepalived.conf in step 2. The higher number will get the VIP address. #### Harbor node 1 setup - - 1> Download harbor offline package from GitHub to you home directory - - 2> Extract the harbor-offline-installer-v.x.x.x.tgz You will get a folder ```harbor``` in the current directory - - 3> cd to ```harbor``` directory - - 4> Edit the ```harbor.cfg``` file - - 5> Edit the hostname = reg.mydomain.com , change reg.mydomain.com to your FQDN or VIP(eg. 192.168.111.220) - - 6> Edit the ```db_host``` to point to the external db host IP/FQDN (eg. 192.168.1.215) - - 7> Edit the ```db_password``` to the password of the external DB - - 8> Edit the ```db_user``` to the user of the external DB - - 9> Edit the ```db_port``` to the port of the external DB - - 10> Edit the ```redis_url``` to the redis server or cluster address (eg. 192.168.1.217:6379) - - 11> Edit the ```clair_db_host```, ```clair_db_port```, ```clair_db_username```, ```clair_db_password```, ```clair_db``` - - 12> Edit the ```registry_storage_provider_name``` and its related settings ```registry_storage_provider_config``` +##### 1> Download harbor offline package from [GitHub](https://github.com/vmware/harbor/releases) to your home directory +##### 2> Extract the harbor-offline-installer-v.x.x.x.tgz You will get a folder ```harbor``` in the current directory +##### 3> cd to ```harbor``` directory +##### 4> Configure hostname +In ```harbor.cfg``` + +``` +hostname = reg.mydomain.com + +``` +Change reg.mydomain.com to your FQDN or VIP(eg. 192.168.1.220) + +##### 5> Provide the Harbor db connection info + +In ```Harbor.cfg``` +Change the value to your Harbor DB's + +``` +#The password for the root user of mysql db, change this before any production use. +db_password = root123 + +... + +##################################################### +#the address of the mysql database. +db_host = 192.168.1.215 + +#The port of mysql database host +db_port = 3306 + +#The user name of mysql database +db_user = root + +``` + +##### 6> Provide the Redis server/cluster address +In harbor.cfg + +``` +#The redis server address +redis_url = 192.168.1.217:6379 +``` + +If you have multiple Redis servers then separate them with ";" + +##### 7> Provide the Clair DB connection information +In harbor.cfg + +``` +clair_db_password = password + +... + +#Clair DB host address +clair_db_host = 192.168.1.50 + +#Clair DB connect port +clair_db_port = 5432 + +#Clair DB username +clair_db_username = postgres + +#Clair default database +clair_db = postgres + +``` +##### 8> Provide the storage config information +In harbor.cfg + +``` +### Docker Registry setting ### +#registry_storage_provider can be: filesystem, s3, gcs, azure, etc. +registry_storage_provider_name = filesystem +#registry_storage_provider_config is a comma separated "key: value" pairs, e.g. "key1: value, key2: value2". +#Refer to https://docs.docker.com/registry/configuration/#storage for all available configuration. +registry_storage_provider_config = +``` + You can find the configure examples for all kinds of storages at https://docs.docker.com/registry/configuration/#storage . For example if you use swift as the storage backend then you need to set the follow: ``` registry_storage_provider_name = swift -registry_storage_provider_config = username: yourusername,password: yourpass,authurl: http://192.168.1.217/identity/v3,tenant: admin,domain: default,region: RegionOne,container: docker_images +registry_storage_provider_config = username:yourusername,password:yourpass,authurl:http://192.168.1.217/identity/v3,tenant:admin,domain:default,region:RegionOne,container:docker_images ``` -**Important!** +**Important** If you set ```filesystem``` for the ```registry_storage_provider_name``` you must make sure the registry directory ```/data/registry``` mount to a shared storage like NFS,Ceph, etc. You need to create the /data/registry directory first and change it's owner to 10000:10000 as registry will run as userID 10000 and groupID 10000. - - 13> Import Harbor database schema to your external MariaDB -``` - Copy the ~/harbor/ha/registry.sql file to a machine that has MariaDB client installed - #> mysql -u your_db_username -p -h your_db_ip < registry.sql -``` - - 14> (Optional) If you enable https, you need to prepare the certificate and key and copy them to ```/data/cert/``` directory(you need to create that folder if it not exist). +##### 9> (Optional) If you enable https, you need to prepare the certificate and key and copy them to ```/data/cert/``` directory(you need to create that folder if it not exist). ``` #>mkdir -p /data/cert @@ -205,11 +274,11 @@ If you set ```filesystem``` for the ```registry_storage_provider_name``` you mus If you want keep your own filename for the certificate, you need to modify the ssl_cert and ssl_cert_key properties in harbor.cfg. If you use a certificate that signed by a private CA then you need to put your CA file to the /data/ca_download/ca.crt - - 15> Start the Harbor on the first node by run +##### 10> Start the Harbor on the first node ``` #>./install.sh --ha ``` -**Note!** +**Note** If you want to use vulnerability scanning functionality. Then use the follow command instead @@ -217,9 +286,9 @@ If you want to use vulnerability scanning functionality. Then use the follow com #>./install.sh --ha --with-clair ``` - - 16> Change iptables +##### 11> Change iptables -**Important!** +**Important** You need to change 192.168.1.220 to your VIP address before issue the follow command, if you just use http for Harbor, then you don't need run the second command. @@ -229,31 +298,44 @@ You need to change 192.168.1.220 to your VIP address before issue the follow com ``` - - 17> Zip the harbor directory +##### 12> Zip the harbor directory ``` #> tar -cvf harbor_ha.tar ~/harbor ``` -- 18> Copy the harbor_ha.tar to harbor_node2 +##### 13> Copy the harbor_ha.tar to harbor_node2 #### Harbor node 2…n setup - - 1> mv the harbor_ha.tar to your home directory on harbor_node2 - - 2> Untar the harbor_ha.tar, you will get a “harbor” folder in your home directory. - - 3> Create certificate folder +##### 1>Place the tar file to home directory - ``` +Move the harbor_ha.tar file to your home directory on harbor_node2 + +##### 2> Untar the file +``` +#> tar -xvf harbor_ha.tar +``` +You will get a “harbor” folder in your home directory. + +##### 3> (Optional)Create certificate folder +This step is only need when enable https for Harbor. + +These folder will be used to store the certificate file. + +``` #> mkdir -p /data/cert #> mkdir -p /data/ca_download - ``` +``` + +##### 4> Start Harbor - - 3> Start Harbor ``` #> cd harbor #> ./install.sh --ha ``` -**Note!** + +**Note** If you enable vulnerability scanning then use @@ -261,9 +343,9 @@ If you enable vulnerability scanning then use #> ./install.sh --ha --with-clair ``` - - 4> Change iptables +##### 5> Change iptables -**Important!** +**Important** You need to change 192.168.1.220 to your VIP address before issue the follow command, if you just use http for Harbor, then you don't need to run the second command. @@ -286,3 +368,6 @@ Workaround: - For all Harbor servers, mount the /data/job_logs directory to a folder on NFS server. - Make sure the folder on the NFS server has read/write permission for UID:GroupID 10000:10000 - Restart jobservices container by ```docker harbor-jobservice restart``` on all Harbor servers. + +2> https://github.com/vmware/harbor/issues/4012 +In Harbor 1.4 we support stop a running Jobs. But in HA scenarios, you may not be able to stop the Jobs. As currently the Job status is stored in memory instead of persistent storages. Request may not be able to schedule to the node which execute the job. We will plan to refactor the jobservices model to sovle this limitation in next release.