Merge branch 'master' into new-ui-with-sync-image

AUTHORS

@@ -5,6 +5,7 @@ Alexey Erkak <eryigin at mail.ru>
 Allen Heavey <xheavey at gmail.com>
 Amanda Zhang <amzhang at vmware.com>
 Benniu Ji <benniuji at gmail.com>
+Bin Liu <liubin0329 at gmail.com>
 Bobby Zhang <junzhang at vmware.com>
 Chaofeng Wu <chaofengw at vmware.com>
 Daniel Jiang <jiangd at vmware.com>

README.md

@@ -13,7 +13,7 @@ Project Harbor is an enterprise-class registry server, which extends the open so
 * **Graphical user portal**: User can easily browse, search Docker repositories, manage projects/namespaces.
 * **AD/LDAP support**: Harbor integrates with existing enterprise AD/LDAP for user authentication and management.
 * **Auditing**: All the operations to the repositories are tracked.
-* **Internationalization**: Already localized for English, Chinese, German and Russian. More languages can be added.
+* **Internationalization**: Already localized for English, Chinese, German, Japanese and Russian. More languages can be added.
 * **RESTful API**: RESTful APIs for most administrative operations, easing intergration with external management platforms.
 
 ### Getting Started
@@ -67,7 +67,7 @@ Harbor is available under the [Apache 2 license](LICENSE).
 &nbsp; &nbsp; <a href="https://www.caicloud.io" border="0"><img alt="CaiCloud" src="docs/img/caicloudLogoWeb.png"></a>
 
 ### Users
-<a href="https://www.madailicai.com/" border="0" target="_blank"><img alt="MaDaiLiCai" src="docs/img/UserMaDai.jpg"></a>
+<a href="https://www.madailicai.com/" border="0" target="_blank"><img alt="MaDaiLiCai" src="docs/img/UserMaDai.jpg"></a> <a href="https://www.dianrong.com/" border="0" target="_blank"><img alt="Dianrong" src="docs/img/dianrong.png"></a>
 
 ### Supporting Technologies
 <img alt="beego" src="docs/img/beegoLogo.png"> Harbor is powered by <a href="http://beego.me/">Beego</a>, an open source framework to build and develop applications in the Go way.

ROADMAP.md

@@ -0,0 +1,47 @@
+## Harbor Roadmap
+
+### About this document
+
+This document provides description of items that are gathered from the community and planned in Harbor's roadmap. This should serve as a reference point for Harbor users and contributors to understand where the project is heading, and help determine if a contribution could be conflicting with a longer term plan.
+
+### How to help?
+
+Discussion on the roadmap can take place in threads under [Issues](https://github.com/vmware/harbor/issues). Please open and comment on an issue if you want to provide suggestions and feedback to an item in the roadmap. Please review the roadmap to avoid potential duplicated effort.
+
+### How to add an item to the roadmap?
+Please open an issue to track any initiative on the roadmap of Harbor. We will work with and rely on our community to focus our efforts to improve Harbor.
+
+
+---
+
+
+### 1. Image replication between Harbor instances 
+Enable images to be replicated between two or more Harbor instances. This is useful to have multiple registry servers servicing a large cluster of nodes, or have distributed registry instances with identical images.
+
+### 2. Image deletion and garbage collection 
+a) Images can be deleted from UI. The files of deleted images are not removed immediately. 
+
+b) The files of deleted images are recycled by an administrator during system maintenance(Garbage collection). The registry service must be shut down during the process of garbage collection.
+
+
+### 3. Authentication (OAuth2) 
+In addition to LDAP/AD and local users, OAuth 2.0 can be used to authenticate a user.
+
+### 4. High Availability 
+Support multi-node deployment of Harbor for high availability, scalability and load-balancing purposes.
+
+### 5. Statistics and description for repositories
+User can add a description to a repository. The access count of a repo can be aggregated and displayed.
+
+
+### 6. Audit all operations in the system
+Currently only image related operations are logged. Other operations in Harbor, such as user creation/deletion, role changes, password reset, should be tracked as well.
+
+
+### 7. Migration tool to move from an existing registry to Harbor 
+A tool to migrate images from a vanilla registry server to Harbor, without the need to export/import a large amount of data.
+
+
+### 8. Support API versioning 
+Provide versioning of Harbor's API.
+

migration/README.md

@@ -1,54 +1,56 @@
-# migration
-Migration is a module for migrating database schema between different version of project [harbor](https://github.com/vmware/harbor)
+# Migration guide
+Migration is a module for migrating database schema between different version of project [Harbor](https://github.com/vmware/harbor)
+
+This module is for those machine running Harbor's old version, such as 0.1.0. If your Harbor' version is up to date, please ignore this module.
 
 **WARNING!!** You must backup your data before migrating
 
-###installation
-- step 1: modify migration.cfg
+###Installation
+- step 1: change `db_username`, `db_password`, `db_port`, `db_name` in migration.cfg
 - step 2: build image from dockerfile
     ```
     cd harbor-migration
     
-    docker build -t your-image-name .
+    docker build -t migrate-tool .
     ```
 
-###migration operation
-- show instruction of harbor-migration
-
-    ```docker run your-image-name help```
-    
-- test mysql connection in harbor-migration
-
-    ```docker run -v /data/database:/var/lib/mysql your-image-name test```
-
-- create backup file in `/path/to/backup`
-
-    ```
-    docker run -ti -v /data/database:/var/lib/mysql -v /path/to/backup:/harbor-migration/backup your-image-name backup
-    ```
-
-- restore from backup file in `/path/to/backup`
-
-    ```
-    docker run -ti -v /data/database:/var/lib/mysql -v /path/to/backup:/harbor-migration/backup your-image-name restore
-    ```
-
-- perform database schema upgrade
-
-    ```docker run -ti -v /data/database:/var/lib/mysql your-image-name up head```
-
-you can use `-v /etc/localtime:/etc/localtime` to sync container timezone with host timezone.
-
-you may change `/data/database` to the mysql volumes path you set in docker-compose.yml.
-###migration step
-- step 1: stop and remove harbor service
+###Migrate Step
+- step 1: stop and remove Harbor service
 
     ``` 
     docker-compose down
     ```
-- step 2: perform migration operation
-- step 3: rebuild newest harbor images and restart service
+- step 2: create backup file in `/path/to/backup`
+
+    ```
+    docker run -ti --rm -v /data/database:/var/lib/mysql -v /path/to/backup:/harbor-migration/backup migrate-tool backup
+    ```
+
+- step 3: perform database schema upgrade
+
+    ```docker run -ti --rm -v /data/database:/var/lib/mysql migrate-tool up head```
+
+
+
+- step 4: rebuild newest Harbor images and restart service
 
     ```
     docker-compose build && docker-compose up -d
     ```
+
+You may change `/data/database` to the mysql volumes path you set in docker-compose.yml.
+
+###Migration operation reference
+- You can use `help` to show instruction of Harbor migration
+
+    ```docker run migrate-tool help```
+    
+- You can use `test` to test mysql connection in Harbor migration
+
+    ```docker run --rm -v /data/database:/var/lib/mysql migrate-tool test```
+
+- You can restore from backup file in `/path/to/backup`
+
+    ```
+    docker run -ti --rm -v /data/database:/var/lib/mysql -v /path/to/backup:/harbor-migration/backup migrate-tool restore
+    ```

migration/db_meta.py

@@ -4,6 +4,7 @@
 import sqlalchemy as sa
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import sessionmaker, relationship
+from sqlalchemy.dialects import mysql
 
 Base = declarative_base()
 
@@ -20,8 +21,8 @@ class User(Base):
     reset_uuid = sa.Column(sa.String(40))
     salt = sa.Column(sa.String(40))
     sysadmin_flag = sa.Column(sa.Integer)
-    creation_time = sa.Column(sa.DateTime)
-    update_time = sa.Column(sa.DateTime)
+    creation_time = sa.Column(mysql.TIMESTAMP)
+    update_time = sa.Column(mysql.TIMESTAMP)
 
 class Properties(Base):
     __tablename__ = 'properties'
@@ -35,8 +36,8 @@ class ProjectMember(Base):
     project_id = sa.Column(sa.Integer(), primary_key = True)
     user_id = sa.Column(sa.Integer(), primary_key = True)
     role = sa.Column(sa.Integer(), nullable = False)
-    creation_time = sa.Column(sa.DateTime(), nullable = True)
-    update_time = sa.Column(sa.DateTime(), nullable = True)
+    creation_time = sa.Column(mysql.TIMESTAMP, nullable = True)
+    update_time = sa.Column(mysql.TIMESTAMP, nullable = True)
     sa.ForeignKeyConstraint(['project_id'], [u'project.project_id'], ),
     sa.ForeignKeyConstraint(['role'], [u'role.role_id'], ),
     sa.ForeignKeyConstraint(['user_id'], [u'user.user_id'], ),
@@ -79,8 +80,8 @@ class Project(Base):
     project_id = sa.Column(sa.Integer, primary_key=True)
     owner_id = sa.Column(sa.ForeignKey(u'user.user_id'), nullable=False, index=True)
     name = sa.Column(sa.String(30), nullable=False, unique=True)
-    creation_time = sa.Column(sa.DateTime)
-    update_time = sa.Column(sa.DateTime)
+    creation_time = sa.Column(mysql.TIMESTAMP)
+    update_time = sa.Column(mysql.TIMESTAMP)
     deleted = sa.Column(sa.Integer, nullable=False, server_default=sa.text("'0'"))
     public = sa.Column(sa.Integer, nullable=False, server_default=sa.text("'0'"))
     owner = relationship(u'User')

migration/migration_harbor/versions/0_1_1.py

@@ -27,9 +27,10 @@ branch_labels = None
 depends_on = None
 
 from alembic import op
-from datetime import datetime
 from db_meta import *
 
+from sqlalchemy.dialects import mysql
+
 Session = sessionmaker()
 
 def upgrade():
@@ -44,12 +45,9 @@ def upgrade():
     session.add(Properties(k='schema_version', v='0.1.1'))
 
     #add column to table user
-    op.add_column('user', sa.Column('creation_time', sa.DateTime(), nullable=True))
+    op.add_column('user', sa.Column('creation_time', mysql.TIMESTAMP, nullable=True))
     op.add_column('user', sa.Column('sysadmin_flag', sa.Integer(), nullable=True))
-    op.add_column('user', sa.Column('update_time', sa.DateTime(), nullable=True))
-
-    #fill update_time data into table user
-    session.query(User).update({User.update_time: datetime.now()})
+    op.add_column('user', sa.Column('update_time', mysql.TIMESTAMP, nullable=True))
 
     #init all sysadmin_flag = 0
     session.query(User).update({User.sysadmin_flag: 0})
@@ -62,7 +60,7 @@ def upgrade():
     for result in join_result:
         session.add(ProjectMember(project_id=result.project_role.project_id, \
             user_id=result.user_id, role=result.project_role.role_id, \
-            creation_time=datetime.now(), update_time=datetime.now()))
+            creation_time=None, update_time=None))
 
     #update sysadmin_flag
     sys_admin_result = session.query(UserProjectRole).\
@@ -89,10 +87,8 @@ def upgrade():
     session.query(Access).update({Access.access_id: Access.access_id - 1})
 
     #add column to table project        
-    op.add_column('project', sa.Column('update_time', sa.DateTime(), nullable=True))
+    op.add_column('project', sa.Column('update_time', mysql.TIMESTAMP, nullable=True))
 
-    #fill update_time data into table project
-    session.query(Project).update({Project.update_time: datetime.now()})
     session.commit()
 
 def downgrade():