Merge branch 'master' into dev

AUTHORS

@@ -8,6 +8,7 @@ Andre Cruz <andre at brpx.com>
 Benniu Ji <benniuji at gmail.com>
 Bin Liu <liubin0329 at gmail.com>
 Bobby Zhang <junzhang at vmware.com>
+Brian Christner <brian.christner at gmail.com>
 Chaofeng Wu <chaofengw at vmware.com>
 Daniel Jiang <jiangd at vmware.com>
 Deshi Xiao <xiaods at gmail.com>

CONTRIBUTING.md

@@ -0,0 +1,54 @@
+# Contributing to Harbor
+
+This guide provides information on filing issues and guidelines for open source contributors.
+
+## Reporting issues
+
+It is a great way to contribute to Harbor by reporting an issue. Well-written and complete bug reports are always welcome! Please open an issue on Github and follow the template to fill in required information.
+
+Before opening any issue, please look up the existing [issues](https://github.com/vmware/harbor/issues) to avoid submitting a duplication.
+If you find a match, you can "subscribe" it to get notified on updates. If you have additional helpful information about the issue, please leave a comment.
+
+When reporting issues, always include:
+
+* Version of docker engine and docker-compose
+* Configuration files of Harbor
+* Log files in /var/log/harbor/ 
+
+Because the issues are open to the public, when submitting the log and configuration files, be sure to remove any sensitive information, e.g. user name, password, IP address, and company name. You can
+replace those parts with "REDACTED" or other strings like "****".
+
+Be sure to include the steps to reproduce the problem if applicable. It can help us understand and fix your issue faster.
+
+
+## Contribution guidelines
+
+### Pull requests
+
+Pull requests (PR) are always welcome, even they are small fixes like typos or a few lines of code changes. If there will be significant effort, please first document as an issue and get the discussion going before starting to work on it.
+
+Please submit a PR to contain changes bit by bit. A PR consisting of a lot features and code changes may be hard to review. It is recommended to submit PRs in a incremental fasion.
+
+### Design new features
+
+You can propose new designs for existing Harbor features. You can also design
+entirely new features. Please do open an issue on Github for discussion first. This is necessary to ensure the overall architecture is consistent and to avoid duplicated work in the roadmap.
+
+### Conventions
+
+Fork Harbor's repository and make changes on your own fork in a new branch. The branch should be named  XXX-description where XXX is the number of the issue. Please run the full test suite on your branch before creating a PR.
+
+Always run [golint](https://github.com/golang/lint) on source code before
+committing your changes.
+
+Pull requests should be rebased on top of master without multiple branches mixed into the PR. If your pull requests do not merge cleanly, use `rebase master` in your branch rather than `merge master`.
+
+Update the documentation if you are creating or changing features. Good documentation is as important as the code itself.
+
+Before you submitting any pull request, always squash your commits into logical units of change. A logical unit of change is defined as a set of codes and documents that should be treated as a whole. When possible, compact your commits into one. The commands to use are `git rebase -i` and/or `git push -f`. 
+
+Description of a pull request should refer to all the issues that it addresses. Remember to put a reference to issues (such as `Closes #XXX` and `Fixes #XXX`) in commits so that the issues can be closed when the PR is merged.
+
+### Signing CLA
+If it is the first time you are making a PR, be sure to sign the contributor license agreement (CLA) online. A bot will automatically update the PR for the CLA process.
+

README.md

@@ -25,7 +25,7 @@ Project Harbor is an enterprise-class registry server that stores and distribute
 
 **On vSphere:** vCenter 5.x+ for deployment of Harbor's virtual appliance. 
 
-Download binaries of **[Harbor release ](https://github.com/vmware/harbor/releases)** and follow **[Installation & Confiugration Guide](docs/installation_guide.md)** to install Harbor.
+Download binaries of **[Harbor release ](https://github.com/vmware/harbor/releases)** and follow **[Installation & Configuration Guide](docs/installation_guide.md)** to install Harbor.
 
 Refer to **[User Guide](docs/user_guide.md)** for more details on how to use Harbor.
 
@@ -35,7 +35,7 @@ Refer to **[User Guide](docs/user_guide.md)** for more details on how to use Har
 **WeChat Group:** Add WeChat id *connect1688* to join WeChat discussion group. 
 
 ### Contribution
-We welcome contributions from the community. If you wish to contribute code and you have not signed our contributor license agreement (CLA), our bot will update the issue when you open a pull request. For any questions about the CLA process, please refer to our [FAQ](https://cla.vmware.com/faq). Contact us for any qustions: harbor@ vmware.com .
+We welcome contributions from the community. If you wish to contribute code and you have not signed our contributor license agreement (CLA), our bot will update the issue when you open a pull request. For any questions about the CLA process, please refer to our [FAQ](https://cla.vmware.com/faq). Contact us for any questions: harbor @vmware.com .
 
 ### License
 Harbor is available under the [Apache 2 license](LICENSE).

docs/README.md

@@ -23,7 +23,7 @@ Guide to deploy Harbor on Kubenetes. (maintained by community)
 
 ### Developer documents
 
-[Arthicture Overview of Harbor](https://github.com/vmware/harbor/wiki/Architecture-Overview-of-Harbor)  
+[Arthitecture Overview of Harbor](https://github.com/vmware/harbor/wiki/Architecture-Overview-of-Harbor)  
 Developers read this first.
 
 [Harbor API Specs by Swagger](configure_swagger.md)  

docs/configure_https.md

@@ -1,6 +1,6 @@
 #Configuring Harbor with HTTPS Access 
 
-Because Harbor does not ship with any certificates, it uses HTTP by default to serve registry requests. This makes it relatively simple to configure. However, it is highly recommended that security be enabled for any production environment. Harbor has an Nginx instance as a reverse proxy for all services, you can configure Nginx to enable https.
+Because Harbor does not ship with any certificates, it uses HTTP by default to serve registry requests.  However, it is highly recommended that security be enabled for any production environment. Harbor has an Nginx instance as a reverse proxy for all services, you can use the prepare script to configure Nginx to enable https.
 
 ##Getting a certificate
 
@@ -9,6 +9,7 @@ Assuming that your registry's **hostname** is **reg.yourdomain.com**, and that i
 In a test or development environment, you may choose to use a self-signed certificate instead of the one from a CA. The below commands generate your own certificate:
 
 1) Create your own CA certificate:
+
 ```
   openssl req \
     -newkey rsa:4096 -nodes -sha256 -keyout ca.key \
@@ -18,6 +19,7 @@ In a test or development environment, you may choose to use a self-signed certif
 
 If you use FQDN like **reg.yourdomain.com** to connect your registry host, then you must use **reg.yourdomain.com** as CN (Common Name). 
 Otherwise, if you use IP address to connect your registry host, CN can be anything like your name and so on:
+
 ```
   openssl req \
     -newkey rsa:4096 -nodes -sha256 -keyout yourdomain.com.key \
@@ -26,85 +28,62 @@ Otherwise, if you use IP address to connect your registry host, CN can be anythi
 3) Generate the certificate of your registry host:
 
 On Ubuntu, the config file of openssl locates at **/etc/ssl/openssl.cnf**. Refer to openssl document for more information. The default CA directory of openssl is called demoCA. Let's create necessary directories and files:
+
 ```
   mkdir demoCA
   cd demoCA
   touch index.txt
   echo '01' > serial
   cd ..
- ```
+```
 If you're using FQDN like **reg.yourdomain.com** to connect your registry host, then run this command to generate the certificate of your registry host:
+
 ```
   openssl ca -in yourdomain.com.csr -out yourdomain.com.crt -cert ca.crt -keyfile ca.key -outdir .
 ```
-If you're using **IP** to connect your registry host, you may instead run the command below:
+If you're using **IP**, say **192.168.1.101** to connect your registry host, you may instead run the command below:
-```
 
-  echo subjectAltName = IP:your registry host IP > extfile.cnf
+```
+  echo subjectAltName = IP:192.168.1.101 > extfile.cnf
 
   openssl ca -in yourdomain.com.csr -out yourdomain.com.crt -cert ca.crt -keyfile ca.key -extfile extfile.cnf -outdir .
 ```
-##Configuration of Nginx
+##Configuration and Installation
-After obtaining the **yourdomain.com.crt** and **yourdomain.com.key** files, change the directory to make/config/nginx in Harbor project.
+After obtaining the **yourdomain.com.crt** and **yourdomain.com.key** files, 
+you can put them into directory such as ```/root/cert/```:
+
 ```
-  cd make/config/nginx
+  cp yourdomain.com.crt /root/cert/
-```
+  cp yourdomain.com.key /root/cert/ 
-Create a new directory cert/, if it does not exist. Then copy **yourdomain.com.crt** and **yourdomain.com.key** to cert/, e.g. :
-```
-  cp yourdomain.com.crt cert/
-  cp yourdomain.com.key cert/ 
 ```
 
-Rename the existing configuration file of Nginx:
-```
-  mv nginx.conf nginx.conf.bak
-```
-Copy the template **nginx.https.conf** as the new configuration file:
-```
-  cp nginx.https.conf nginx.conf
-```
-Edit the file nginx.conf and replace two occurrences of **harbordomain.com** to your own host name, such as reg.yourdomain.com . If you use a customized port rather than the default port 443, replace the port "443" in the line "rewrite ^/(.*) https://$server_name:443/$1 permanent;" as well. Please refer to the [installation guide](https://github.com/vmware/harbor/blob/master/docs/installation_guide.md) for other required steps of port customization.  
-```
-  server {
-    listen 443 ssl;
-    server_name harbordomain.com;
 
-    ...
+Next, edit the file make/harbor.cfg , update the hostname and the protocol, and update the attributes ```ssl_cert``` and ```ssl_cert_key```:
 
-  server {
-    listen 80;
-    server_name harbordomain.com;
-    rewrite ^/(.*) https://$server_name:443/$1 permanent;
-```
-Then look for the SSL section to make sure the files of your certificates match the names in the config file. Do not change the path of the files.
-```
-    ...
-    
-    # SSL
-    ssl_certificate /etc/nginx/cert/yourdomain.com.crt;
-    ssl_certificate_key /etc/nginx/cert/yourdomain.com.key;
-```
-Save your changes in nginx.conf.
-
-##Installation of Harbor
-Next, edit the file make/harbor.cfg , update the hostname and the protocol:
 ```
   #set hostname
   hostname = reg.yourdomain.com
   #set ui_url_protocol
   ui_url_protocol = https
+  ......
+  #The path of cert and key files for nginx, they are applied only the protocol is set to https 
+  ssl_cert = /root/cert/yourdomain.com.crt
+  ssl_cert_key = /root/cert/yourdomain.com.key
 ```
 
 Generate configuration files for Harbor:
+
 ```
-./prepare
+  ./prepare
 ```
+
 If Harbor is already running, stop and remove the existing instance. Your image data remain in the file system
+
 ```
-  docker-compose stop
+  docker-compose down  
-  docker-compose rm
 ```
 Finally, restart Harbor:
+
 ```
   docker-compose up -d
 ```
@@ -126,17 +105,20 @@ If you've mapped nginx 443 port to another, you need to add the port to login, l
 
 ##Troubleshooting
 1. You may get an intermediate certificate from a certificate issuer. In this case, you should merge the intermediate certificate with your own certificate to create a certificate bundle. You can achieve this by the below command:  
+
     ```
     cat intermediate-certificate.pem >> yourdomain.com.crt 
     ```
 2. On some systems where docker daemon runs, you may need to trust the certificate at OS level.  
    On Ubuntu, this can be done by below commands:  
+   
     ```sh
     cp youdomain.com.crt /usr/local/share/ca-certificates/reg.yourdomain.com.crt
     update-ca-certificates
     ```  
     
    On Red Hat (CentOS etc), the commands are:  
+   
     ```sh
     cp yourdomain.com.crt /etc/pki/ca-trust/source/anchors/reg.yourdomain.com.crt
     update-ca-trust

docs/developer_guide_i18n.md

@@ -45,7 +45,7 @@
 
 5. Add the new language to the `app.conf` file.
     
-    In the file `make/config/ui/app.conf`, append a new item to the configuration section.
+    In the file `make/common/templates/ui/app.conf`, append a new item to the configuration section.
     ```
 		 [lang]
 		 types = en-US|zh-CN|<language>-<locale>

docs/installation_guide.md

@@ -3,7 +3,7 @@ Harbor can be installed by one of three approaches:
 
 - **Online installer:** The installer downloads Harbor's images from Docker hub. For this reason, the installer is very small in size.
 
-- **Offline installer:** Use this installer when the host does not have Internet connection. The installer contains pre-built images so its size is larger.
+- **Offline installer:** Use this installer when the host does not have an Internet connection. The installer contains pre-built images so its size is larger.
 
 - **Virtual Appliance:** If you are installing Harbor as the registry component of vSphere Integrated Containers (VIC), or using Harbor as a standalone registry on vSphere platform, download the OVA version of Harbor. 
 
@@ -59,25 +59,25 @@ The parameters are described below - note that at the very least, you will need
 	* email_from = admin <sample_admin@mydomain.com>  
 	* email_ssl = false
 
-* **harbor_admin_password**: The adminstrator's initial password. This password only takes effect for the first time Harbor launches. After that, this setting is ignored and the adminstrator's password should be set in the UI. _Note that the default username/password are **admin/Harbor12345** ._   
+* **harbor_admin_password**: The administrator's initial password. This password only takes effect for the first time Harbor launches. After that, this setting is ignored and the administrator's password should be set in the UI. _Note that the default username/password are **admin/Harbor12345** ._   
-* **auth_mode**: The type of authentication that is used. By default it is **db_auth**, i.e. the credentials are stored in a database. For LDAP authentication, set this to **ldap_auth**.  
+* **auth_mode**: The type of authentication that is used. By default, it is **db_auth**, i.e. the credentials are stored in a database. For LDAP authentication, set this to **ldap_auth**.  
 * **ldap_url**: The LDAP endpoint URL (e.g. `ldaps://ldap.mydomain.com`).  _Only used when **auth_mode** is set to *ldap_auth* ._    
 * **ldap_searchdn**: The DN of a user who has the permission to search an LDAP/AD server (e.g. `uid=admin,ou=people,dc=mydomain,dc=com`).
 * **ldap_search_pwd**: The password of the user specified by *ldap_searchdn*.
 * **ldap_basedn**: The base DN to look up a user, e.g. `ou=people,dc=mydomain,dc=com`.  _Only used when **auth_mode** is set to *ldap_auth* ._ 
 * **ldap_filter**:The search filter for looking up a user, e.g. `(objectClass=person)`.
-* **ldap_uid**: The attribute used to match a user during a ldap search, it could be uid, cn, email or other attributes.
+* **ldap_uid**: The attribute used to match a user during a LDAP search, it could be uid, cn, email or other attributes.
 * **ldap_scope**: The scope to search for a user, 1-LDAP_SCOPE_BASE, 2-LDAP_SCOPE_ONELEVEL, 3-LDAP_SCOPE_SUBTREE. Default is 3. 
-* **db_password**: The root password for the mySQL database used for **db_auth**. _Change this password for any production use!_ 
+* **db_password**: The root password for the MySQL database used for **db_auth**. _Change this password for any production use!_ 
 * **self_registration**: (**on** or **off**. Default is **on**) Enable / Disable the ability for a user to register themselves. When disabled, new users can only be created by the Admin user, only an admin user can create new users in Harbor.  _NOTE: When **auth_mode** is set to **ldap_auth**, self-registration feature is **always** disabled, and this flag is ignored._  
 * **use_compressed_js**: (**on** or **off**. Default is **on**) For production use, turn this flag to **on**. In development mode, set it to **off** so that js files can be modified separately.
 * **max_job_workers**: (default value is **3**) The maximum number of replication workers in job service. For each image replication job, a worker synchronizes all tags of a repository to the remote destination. Increasing this number allows more concurrent replication jobs in the system. However, since each worker consumes a certain amount of network/CPU/IO resources, please carefully pick the value of this attribute based on the hardware resource of the host. 
 * **secret_key**: The key to encrypt or decrypt the password of a remote registry in a replication policy, its length has to be 16 characters. Change this key before any production use. *NOTE: After changing this key, previously encrypted password of a policy can not be decrypted.*
 
-* **token_expiration**: The expiration time (in minute) of a token created by token service, default is 30 minutes.
+* **token_expiration**: The expiration time (in minutes) of a token created by token service, default is 30 minutes.
 
 * **verify_remote_cert**: (**on** or **off**.  Default is **on**) This flag determines whether or not to verify SSL/TLS certificate when Harbor communicates with a remote registry instance. Setting this attribute to **off** bypasses the SSL/TLS verification, which is often used when the remote instance has a self-signed or untrusted certificate.
-* **customize_crt**: (**on** or **off**.  Default is **on**) When this attribute is **on**, the prepare script creates private key and root certificate for the generation/verification of the regitry's token.  
+* **customize_crt**: (**on** or **off**.  Default is **on**) When this attribute is **on**, the prepare script creates private key and root certificate for the generation/verification of the registry's token.  
 * The following attributes:**crt_country**, **crt_state**, **crt_location**, **crt_organization**, **crt_organizationalunit**, **crt_commonname**, **crt_email** are used as parameters for generating the keys. Set this attribute to **off** when the key and root certificate are supplied by external sources. Refer to [Customize Key and Certificate of Harbor Token Service](customize_token_service.md) for more info.
 
 #### Configuring storage backend (optional)

docs/user_guide.md

@@ -1,6 +1,6 @@
 #User Guide
 ##Overview
-This guide takes you through the fundamentals of using Harbor. You'll learn how to use Harbor to:  
+This guide walks you through the fundamentals of using Harbor. You'll learn how to use Harbor to:  
 
 * Manage your projects.
 * Manage members of a project.
@@ -11,7 +11,7 @@ This guide takes you through the fundamentals of using Harbor. You'll learn how
  + Manage destinations.
  + Manage replication policies.
 * Pull and push images using Docker client.
-* Delete repositories.
+* Delete repositories and images.
 
 
 ##Role Based Access Control
@@ -152,24 +152,30 @@ $ docker push 10.117.169.182/demo/ubuntu:14.04
 
 ##Deleting repositories
 
-Repositories deletion runs in two steps.  
+Repository deletion runs in two steps.  
-First, delete repositories in Harbor's UI. This is soft deletion. You can delete the entire repository or just a tag of it.  
+
+First, delete a repository in Harbor's UI. This is soft deletion. You can delete the entire repository or just a tag of it. After the soft deletion, 
+the repository is no longer managed in Harbor, however, the files of the repository still remains in Harbor's storage.  
 
 ![browse project](img/new_delete_repository.png)
 
-**Note: If both tag A and tag B reference the same image, after deleting tag A, B will also disappear.**  
+**CAUTION: If both tag A and tag B refer to the same image, after deleting tag A, B will also get deleted.**  
 
-Second, delete the real data using registry's garbage colliection(GC).  
+Next, delete the actual files of the repository using the registry's garbage collection(GC). Make sure that no one is pushing images or Harbor is not running at all before you perform a GC. If someone were pushing an image while GC is running, there is a risk that the image's layers will be mistakenly deleted which results in a corrupted image. So before running GC, a preferred approach is to stop Harbor first.  
-Make sure that no one is pushing images or Harbor is not running at all before you do GC. If someone were to push an image while GC is running, there is the risk that the image's layers will be mistakenly deleted, leading to a corrupted image. So before running GC, a preferred approach is to stop Harbor first.  
 
-Run the command on the host which harbor is deployed on. 
+Run the below commands on the host which Harbor is deployed on to preview what files/images will be affect: 
 
 ```sh
 $ docker-compose stop
-$ docker run -it --name gc --rm --volumes-from deploy_registry_1 registry:2.4.0 garbage-collect [--dry-run] /etc/registry/config.yml
+$ docker run -it --name gc --rm --volumes-from deploy_registry_1 registry:2.5.0 garbage-collect --dry-run /etc/registry/config.yml
+```  
+**NOTE:** The above option "--dry-run" will print the progress without removing any data.  
+
+Verify the result of the above test, then use the below commands to perform garbage collection and restart Harbor. 
+
+```sh
+$ docker run -it --name gc --rm --volumes-from deploy_registry_1 registry:2.5.0 garbage-collect  /etc/registry/config.yml
 $ docker-compose start
 ```  
 
-Option "--dry-run" will print the progress without removing any data.  
+For more information about GC, please see [GC](https://github.com/docker/docker.github.io/blob/master/registry/garbage-collection.md).  
-
-About the details of GC, please see [GC](https://github.com/docker/docker.github.io/blob/master/registry/garbage-collection.md).  

tests/testcases/Group2-image-management/2-05-DB-user-delete-image.md

@@ -24,7 +24,7 @@ In below test, user A is non-admin user. User A and project X, Y should be repla
 3. On a Docker client, log in as User A and run `docker push` to push an image to the project X, e.g. projectX/myimage:v1.
 4. Push a second image with different tag to project X, e.g. projectX/myimage:v2 .
 5. Push an image with different name to project X, e.g. projectX/newimage:v1 .
-6. Run `docker pull` to verify images can be pushed successfully.
+6. Run `docker pull` to verify images can be pulled successfully.
 7. In UI, delete the three images one by one.
 8. On a Docker client, log in as User A and run `docker pull` to pull the three deleted images of project X.
 9. In UI, delete project X. 

tests/testcases/Group2-image-management/2-06-DB-user-delete-projects.md

@@ -25,7 +25,7 @@ User guide
 2. Create a project X so that the user has the project admin role.
 3. On a Docker client, log in as User A and run `docker push` to push an image to project X, e.g. projectX/myimage:v1.
 4. Push an image with different name to project X, e.g. projectX/newimage:v1 .
-5. Run `docker pull` to verify images can be pushed successfully.
+5. Run `docker pull` to verify images can be pulled successfully.
 6. In UI, delete project X directly. (should fail with errors)
 7. While keeping the current user A logged on, in a different browser, log in as admin user. 
 8. Under "Admin Options", create a replication policy of project X to another Harbor instance. (Do not need to activate this policy.)

tests/testcases/Group2-image-management/2-23-admin-delete-images.md

@@ -23,7 +23,7 @@ In below test, user A is non-admin user. User A and project X should be replaced
 2. Create a project X so that the user has the project admin role.
 3. On a Docker client, log in as User A and run `docker push` to push an image to the project X, e.g. projectX/myimage:v1.
 4. Push an image with different name to project X, e.g. projectX/newimage:v1 .
-5. Run `docker pull` to verify images can be pushed successfully.
+5. Run `docker pull` to verify images can be pulled successfully.
 6. In UI, log out user A's session. 
 7. Log in as admin user.
 8. Under project X, delete the two images one by one.

tests/testcases/Group2-image-management/2-24-admin-delete-projects.md

@@ -24,7 +24,7 @@ User guide
 1. Log in to UI as user A (non-admin).
 2. Create a project X so that the user has the project admin role.
 3. On a Docker client, log in as User A and run `docker push` to push an image to project X, e.g. projectX/myimage:v1.
-4. Run `docker pull` to verify images can be pushed successfully.
+4. Run `docker pull` to verify images can be pulled successfully.
 5. In UI, log out user A.
 6. Log in as admin user.
 7. Delete project X. (should fail with errors)
@@ -35,7 +35,7 @@ User guide
 # Expected Outcome:
 * Step 7, deleting project X should fail because there are images under it.
 * Step 8-9, deleting images of project X and then deleting project X should succeed.
-* Step 10, there should be logs for delete and create of project X, notice the project name of the deleted operation is displayed differently.
+* Step 10, there should be logs for deletion and creation of project X.
 
 # Possible Problems:
 None

tests/testcases/Group3-RBAC/3-11-DB-admin-user-manage-project-members.md

@@ -51,17 +51,15 @@ User guide
 25. In admin's UI, change user C's role to project admin of project X.
 26. In user C's UI, verify his/her role is project admin of project X. 
 
-27. In admin's UI, remove user C's from project X.
+27. In admin's UI, remove user C from project X.
 28. Set project X's publicity to on.
 29. On a Docker client host, log in as user C. 
 30. Use `docker pull` to pull the image from project X. 
 31. Use `docker push` to push an image to project X. (should fail)
-32. In admin's UI, remove user C's from project X.
+32. Set project X's publicity to off.
-33. Set project X's publicity to off.
+33. On a Docker client host, log in as user C. 
-34. On a Docker client host, log in as user C. 
+34. Use `docker pull` to pull the image from project X. (should fail) 
-35. Use `docker pull` to pull the image from project X. (should fail) 
+35. Use `docker push` to push an image to project X. (should fail)
-36. Use `docker push` to push an image to project X. (should fail)
-
 
 # Expected Outcome:
 
@@ -72,7 +70,7 @@ User guide
 * Step 26 as described.
 * Step 30 should succeed.
 * Step 31 should fail.
-* Step 35-36 should fail.
+* Step 34-35 should fail.
 
 # Possible Problems:
 None