Commit Graph

80 Commits

Author SHA1 Message Date
Daniel Jiang
3a1ffb3c49 Update migrator to support only from >=v1.6.0
This commit removes code to support upgrade from <v1.6.0
It also removes packages for supporting mysql/mariadb from Dockerfile

It does not handle the optimization in the script such as run.sh after the update.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-05-06 22:46:58 -07:00
Qian Deng
c1e676ad99 Add migration script from 1.7.0 to 1.8.0
Add jinja2 to migrator
Add template to migrator
Add config upgrading script

Signed-off-by: Qian Deng <dengq@vmware.com>
2019-04-17 15:14:54 +08:00
Qian Deng
93af296eeb Enhance: refactor the mount dirs and workflow of generate cert
mount a temp dir input for all input files and configs
generated secrets file stored in data volumns keys dir
certs file stored in data volumns nginx dir

Signed-off-by: Qian Deng <dengq@vmware.com>
2019-03-15 15:51:11 +08:00
Qian Deng
b0f158c4c8 Add migratior script
Add migrator template and script

Signed-off-by: Qian Deng <dengq@vmware.com>
2019-03-08 16:46:14 +08:00
Daniel Jiang
091ed91ad8 Install psycopg2 via photon's repo (#7018)
The psycopg2 installed by pip has very old `.so` files.
This commit will mitigate such issue by using `tdnf` to install this package.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-02-26 10:58:05 +08:00
wang yan
cd50d85c47 Upgrade migrator base os to photon 2.0
Signed-off-by: wang yan <wangyan@vmware.com>
2019-02-14 11:35:44 +08:00
Yan
5412e581de
Add a flag judging on building migrator (#6905)
This commit is to add a flag judging when to build image of migator, which is not necessary.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-02-12 20:33:42 +08:00
wang yan
7936d14abc fix per comments, add if not exists to create pqsql sequence
Signed-off-by: wang yan <wangyan@vmware.com>
2018-12-08 16:11:30 +08:00
wang yan
0517ecca7a Fix unable to use Notary after updating to v1.6.0
This commit is to fix the issue on notary migrations from mysql to pgsql.
1, alter sequence for the primary key of changeseed, this is missed in v1.6.0 migrator.
2, alter table owners from postgres to notarysigner and notaryserver.

Issue:
https://github.com/goharbor/harbor/issues/6465

Workaround:
https://github.com/goharbor/harbor/issues/6465#issuecomment-445162616

Impacted upgrade path:
1, Upgrade from version older then v1.6.0 with migrator:v1.6.0, and migrates the notarty DB.

No impacted upgrade path:
1, Upgrade from version older than v1.6.0 with migrator:v1.6.0, but without migrates the notarty DB.

Notes:
After merge this fix, we need to provide an new migrator with an new tag, like v1.6.1, and
deprecated the v1.6.0. For those who was impacted by migrator v1.6.0, will open an new PR to build
the workaround into the migrator and expose an specical command for hot-fix.

Signed-off-by: wang yan <wangyan@vmware.com>
2018-12-07 19:33:44 +08:00
Daniel Jiang
2a9d46e052 Update no_proxy from 'portal' to 'core' in cfg migrator (#6464)
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2018-12-06 11:49:40 +08:00
Qian Deng
7647e688fd Feat: Upgrade harbor.cfg from 1.6.0 to 1.7.0
Add script to upgrade harbor.cfg

Signed-off-by: Qian Deng <dengq@vmware.com>
2018-12-04 10:36:11 +08:00
Brett Johnson
f3640acf67 corrected tdnf build install command.
build failing due to tdnf attempting to install
x86_64/gcc-5.3.0-5.ph1.x86_64.rpm

Signed-off-by: Brett Johnson <brett@sdbrett.com>
2018-11-29 17:55:23 +11:00
Brett Johnson
2d134c2f21 corrected tdnf build install command.
build failing due to tdnf attempting to install
x86_64/gcc-5.3.0-5.ph1.x86_64.rpm

Signed-off-by: Brett Johnson <brett@sdbrett.com>
2018-11-29 16:30:27 +11:00
clouderati
587459df15 Replacing copyright notices with "Copyright Project Harbor Authors".
Signed-off-by: clouderati <35942204+clouderati@users.noreply.github.com>
2018-09-19 16:59:36 +00:00
wang yan
b3918fe748 Add the encoding to the data converter script
This commit is to fix bug reported by community to upgrade v1.6.0, for the details
just refer to #5788. The fix is to add the encoding(utf-8) for the converter when to convert
the mysql data to pqsql data. Already passed on test with Chinese/Japanese.

Signed-off-by: wang yan <wangyan@vmware.com>

update
2018-09-04 12:02:48 +08:00
wang yan
f8b964d8cf Extend configuration length to 1024 in DB
Signed-off-by: wang yan <wangyan@vmware.com>
2018-08-31 13:42:24 +08:00
Daniel Jiang
bda3878ab8 Update registry image to enable inject root cert
In some user's environment, there's local object storage hosted with
self-signed certificate.
Because registry process runs in a photon container, it has to trust
the certificate in the photon level such that the registry can access
the storage service.

This commit updates the registry image to append custom cert to the root
bundle when the container is started.  And make the customer cert
configurable in `harbor.cfg`

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2018-08-16 18:36:21 +08:00
Daniel Jiang
3dc95ee7e2 Update CFG migrator to match the new redis setting (#5567)
There's a limitation in the new harbor.cfg file the redis url will be
hard coded to the internal redis instance.  We expect user to manually
edit the .cfg file if he's using a customized redis, this would be a
very rare case.  We'll mention it in the release note of 1.6
2018-08-09 15:30:35 +08:00
wangyan
063e44c486 Remove the tdnf error eater in docker files 2018-08-07 22:58:01 -07:00
wangyan
e89f72e081 Add default value for creation time and update time trigger
These chanes are introduced by PR 5430, please refer to the PR for details
2018-07-30 23:48:46 -07:00
wangyan
22411cf6b2 Fix pgsql creation column bug
Root cause: Use default 'now'::timestamp will not generate timestamp for each transaction,
PG will convert now to a timestamp as soon as the constant is parsed. To fix it, update it
to defult CURRENT_TIMESTAMP, thie setting is the same as default now(), which returns the
start time of current transaction because ther are fuction calls, hey will give the desired
behavior of defaulting to the time of row insertion.

Reference: https://www.postgresql.org/docs/9.6/static/functions-datetime.html#FUNCTIONS-DATETIME-CURRENT
PG version: 9.6.9
2018-07-30 04:58:44 -07:00
Daniel Jiang
46de1432f2 Enable cfg migrator to 1.6.0
In 1.6, there will be only one DB process in the default deployment.
The migrator will try to handle the setting by "guessing" whether Harbor
was pointed to external DB.

Verified 1.5->1.6 and 1.4->1.6 migration.
2018-07-27 17:11:45 +08:00
wangyan
5b492ee321 Update migrator for support new table admin_job 2018-07-24 20:29:56 -07:00
Daniel Jiang
6c664ee993 Update photon base images (#5346)
This commit update the base photon image from vmware/photon:1.0 to
photon:1.0, per suggestion by photon team.
2018-07-19 20:45:20 +08:00
Deng, Qian
edbe2fe620 Update migrator to 1 6 0
1. Add new alembic_pg folder for postgres
2. Add migration file for 1.6.0
3. Update version to 1.6.0
4. update migrator dockerfile
2018-07-02 21:23:47 +08:00
wangyan
2379123ff7 fix up clair issue and typo 2018-06-10 23:33:37 -07:00
wangyan
8d4d1582a3 add checking for clair DB migration 2018-06-10 20:15:55 -07:00
wangyan
c7a54e115c Add clair data migration in DB migrator 2018-06-08 01:49:28 -07:00
Wenkai Yin 79628
0c56493fb6 Soft delete label
Modify the deletion of label to soft deletion, in this way the names of deleted labels referenced by replication rules can be shown to users
2018-06-07 17:14:12 +08:00
Yan
6d800cabbd
enable migrator to support 1.5.0 migration from mysql to pgsql (#5029)
This commit is to enable data migrator to support migrates data
from mysql to pgsql, this is a specific step for user to upgrade
harbor across v1.5.0, as we have move harbor DB to pgsql from
1.5.0. It supports both harbor and notary db data migration,
and be split into two steps with dependency.

It also fix issue #4847, add build DB migrator in make process.
2018-06-01 14:58:43 +08:00
Wenkai Yin
5f58c01e4a Fix bug #4840 (#4851)
Fix the bug: upgrade failed in VIC if replication policies exist
2018-05-02 09:37:35 -07:00
wangyan
2b3d70357e Overwrite some attributes in harbor.cfg 1.5
The values of "redis_url" and "max_job_workers" should be set to new
values due to new implementation.
2018-05-01 09:28:29 +08:00
Wenkai Yin
73babbf1ab Modify unique constraint of table harbor_label
Add unique constraint to column name, scope and project_id  of table harbor_label to make creating same name labels under different projects valid
2017-12-19 22:15:56 +08:00
Yan
bc473f8521
Enable output parameter in harbor migrator (#4561) 2018-04-02 03:04:00 -07:00
Daniel Jiang
3f46df8087
Merge pull request #4560 from reasonerjt/cfg-upgrade-fix
Refine the migrator
2018-04-02 14:22:37 +08:00
Tan Jiang
8b9a67fe1e Update harbor.cfg template to contain latest change 2018-04-02 13:01:55 +08:00
Tan Jiang
1e260b59fe Refine the python file to make it run on python3 2018-04-02 12:34:12 +08:00
Daniel Jiang
53cea404fd
Merge pull request #4523 from ninjadq/migrator_1_5
Upgrade migrator to 1.5.0
2018-04-02 09:25:11 +08:00
Yan
5bbbb06807
add all in one migrator scripts (#4511) 2018-04-01 08:53:21 -07:00
Deng, Qian
93c96da18b Upgrade migrator to 1.5.0 2018-03-30 22:45:32 +08:00
Tan Jiang
b1923bf219 Refine the python file to make it run on python3 2018-03-30 14:41:12 +08:00
Daniel Jiang
24dfad4afe Return zero when input version is same as target (#4535) 2018-03-29 23:17:07 -07:00
Tan Jiang
2debdb7229 Return zero when input version is same as target 2018-03-30 11:34:51 +08:00
Tan Jiang
b6df6cf169 Add indexes to job tables and bump up schema version. 2018-03-28 16:15:54 +08:00
Tan Jiang
6f1c46624f Update SQL script to add uuid to job tables.
We have to add the uuid/id mapping as new job service will only store uuid.

Further work is in feature branch for now, commit this change to
accelerate migration work.
2018-03-23 17:45:50 +08:00
Tan Jiang
421bd182bc Update DB changelog to reflect schema change for user group 2018-03-21 16:45:21 +08:00
Tan Jiang
c8265a8d53 Provide migration scripts for harbor.cfg
Default target version is 1.5.0
This is mainly for VIC-appliance upgrade, and should be considered
experimental for oss due to limited test.
Tested with 1.2 and 1.3 harbor.cfg from VIC appliance.
2018-03-16 14:38:50 +08:00
Wenkai Yin
36b9c4e458 Implement adding/removing labels to/from repositories and images API 2018-03-12 19:30:05 +08:00
Wenkai Yin
379f113452 Implement label management API 2018-03-09 12:17:27 +08:00
Deng, Qian
b3e65ed71e Update migration tool for v1.4
1. Update database meta file
2. Add migration file for 1.4
2018-01-16 15:38:51 +08:00