Commit Graph

144 Commits

Author SHA1 Message Date
wang yan
9822e5bb15 fix db migration issue
fixes #12827
 After user migrates Harbor from v2.0.2, user got 404 when to pull specific images, and no work after push the same images again.

 Fix:
 1, If the issue is caused by missing repository data, this fix can revert the missing repository data and all things should be fine.
 2, If the issue is caused by missing blob data, this fix can revert the missing repository data and still left the media type of artifact
    as 'UNKNOWN', which leads the meta data and build history of the image cannot be shown in UI. User can delete and push the image again to
    resolve it.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-26 12:02:20 +08:00
Wang Yan
ad47d2f444
fix upgrade issue (#12857)
fixes #12849

1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.

2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-24 16:08:15 +08:00
Daniel Jiang
c0602b5fb3
Merge pull request #12832 from ywk253100/200820_data
Add id column to data_migration table
2020-08-21 19:30:05 +08:00
Ted Guan
645dea36a6
Fix for duplicate webhook policy name (#12729)
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-08-20 18:02:13 +08:00
Wenkai Yin
975ef193dd Add id column to data_migration table
Add id column to data_migration table and add logic to make sure there is only one data version record

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-20 17:43:15 +08:00
Wenkai Yin
0fd230c2d6 Refresh the status of execution for every status changing of task
Refresh the status of execution for every status changing of task to support filtering executions by status directly

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-17 17:38:55 +08:00
Wenkai Yin
b1ddb5e2cc Implement the icon API to get the icon of artifact
Implement the icon API to get the icon of artifact

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-15 08:40:38 +08:00
Wenkai Yin
cca1dcca51 Use a separated database table to store the data version
Use a separated database table to store the data version.
Fixes #12747

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-14 11:38:13 +08:00
Yiyang Huang
b98dc97fbd feat: enhanced default processor
Signed-off-by: Yiyang Huang <huangyiyang.huangyy@bytedance.com>
2020-08-11 01:31:02 +08:00
chlins
b765cfe0ce fix(replication): refactor quay adapter to fix authorization and support quay.io and enterprise quay (#10317)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-08-08 13:17:01 +08:00
Wenkai Yin
d6288a43e8 Do some refine for the scheduler
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-05 17:43:18 +08:00
Steven Zou
ee35e1ecc6
Merge pull request #12507 from chlins/fix/preheat-update-instance
fix(preheat): fix preheat handler PingInstance and UpdateInstance
2020-07-20 17:45:24 +08:00
Steven Zou
46d7434d0b
Merge pull request #12473 from ywk253100/200706_scheduler
Refactor the scheduler with the task manager mechanism
2020-07-20 15:53:14 +08:00
chlins
78927af032 fix(preheat): fix preheat handler PingInstance and UpdateInstance
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-20 15:42:24 +08:00
Wenkai Yin
4dc4b6728c Refactor the scheduler with the task manager mechanism
Refactor the scheduler with the task manager mechanism, this will reduce the duplicate code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-20 14:03:15 +08:00
Daniel Jiang
e96165412d
Merge pull request #12432 from ywk253100/200709_allowlist
Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist"
2020-07-13 16:42:43 +08:00
chlins
38d14dff30 fix(preheat): validate instance/policy name, set unique filed and policy
manager adds parsePolicy

Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-10 10:29:47 +08:00
Wenkai Yin
cd6c1b8c31 Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist"
Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist" in project metadata

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-09 11:04:44 +08:00
Daniel Jiang
1637e6a588 Rename master role to maintainer
This commit rename the var name, text appearance, and swagger of "master" role
to "maintainer" role.
It only covers backend code.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-07-08 09:20:07 +08:00
fanjiankong
a0c2d0ac9e feat(preheat):add preheat api, controller and manager
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-03 11:25:42 +08:00
chlins
15e4361d6e feat: add p2p preheat policy dao and manager(#12286)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-06-30 15:56:50 +08:00
AllForNothing
fff6f7529a Replace all whitelist with allowlist
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-06-24 16:17:17 +08:00
wang yan
c10467eb36 continue refactor
Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
Wang Yan
de504993ad update blob controller & manager
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
Wenkai Yin
127988b70c Define the task manager interface and data model
Define the task manager interface and data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-15 18:42:09 +08:00
Wenkai Yin
a79bb127b3 Update creating project API to support proxy cache project
Update creating project API to support proxy cache project

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-10 17:14:12 +08:00
Daniel Jiang
f91d7080d1
Merge pull request #11753 from tedgxt/2.0-webhook-event-types-fix
Webhook data fix when updgrding to 2.0
2020-04-28 19:36:44 +08:00
guanxiatao
f96cfab100 Table notification_policy fix when updgrding to 2.0
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-04-27 20:13:46 +08:00
Wang Yan
add8dedc90
Fix the database upgrade issue (#11766)
It's a workaround for issue https://github.com/goharbor/harbor/issues/11754

The phenomenon is the repository data is gone, but artifacts belong to the repository are still there.
To resolve it, just set the repository_id to a negative, and cannot duplicate.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-27 17:28:36 +08:00
Wang Yan
790064df2e
fix notification policy ugrade issue (#11627)
Fixes #11624

All of the existing policies created v1.10 has no name, it fails the upgrade process.
When to set the unique constraint for policy name, the empty can be seen as duplicated key.

ERROR:  could not create unique index "notification_policy_name_key"
DETAIL:  Key (name)=() is duplicated.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-16 14:53:58 +08:00
He Weiwei
f5487479dd feat(scanner): make Clair and Trivy as reserved name for scanners
Closes #11524

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-04-15 02:26:02 +00:00
wang yan
a96d2f3746 remove the chart handling in quota
1, remove the chartmuseum controller
2, doesn't handle chartrepo url in v2 middleware

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-07 15:26:34 +08:00
wang yan
44825e819e deprecate quota count on artifact
Fixes #11241

1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-06 16:56:11 +08:00
Wenkai Yin(尹文开)
9ebcf95758
Merge pull request #11122 from ywk253100/200318_replication_task
Increase the length the columns (src_resource, dst_resource)of replication_task
2020-03-19 12:16:27 +08:00
Wang Yan
b4e941e961
drop table access log in migration (#11118)
Use the audit log instead, the access log table should be dropped after migration

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-18 19:04:38 +08:00
Wenkai Yin
ac9658bc1e Increase the length the columns (src_resource, dst_resource)of replication_task
Fixes #10786 by increaseing the length of src_resource and dst_resource to 256

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-18 17:05:32 +08:00
He Weiwei
7d20154db5
fix: remove old artifact model (#11112)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-03-18 14:20:06 +08:00
Wenkai Yin
a4a1913598 Repair the count usage during the upgrading
As the count quota is against artifact rather than tag in 2.0, the count usage should be recalculated

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-13 13:59:48 +08:00
Ted Guan
4ac31c6d46
Add API for query supported event types and notify types; Return policy name in last trigger info; Remove project_id unique constraint in table notification_policy (#11029)
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-03-11 18:06:58 +08:00
Wenkai Yin(尹文开)
8452100148
Merge pull request #10942 from ywk253100/200305_reference
Persistent the URLs and annotations of artifact references in database
2020-03-11 16:20:18 +08:00
wang yan
288c7790d0 add sql for migrating access log
1, loop each access log, change to resource/resource_type, and insert into audit log
2, loop each first push operation, change it to create repository and insert into audit log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-06 12:06:12 +08:00
wang yan
df237a5b17 add audit logs API
1, add API entry for get audit logs
2. add audit log manager to hanlder CRUD

Use the new format of audit log to cover differernt resource, artifact/tag/repostory/project

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-05 11:40:51 +08:00
Wenkai Yin
76c04b0219 Persistent the URLs and annotations of artifact references in database
Persistent the URLs and annotations of artifact references in database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-05 10:54:45 +08:00
Wenkai Yin
4c9b59c904 Migrate artifact data in 2.0
Abstract extra attributes and annotations for artifacts stored in database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-28 18:09:02 +08:00
Wenkai Yin
02c2647e1e Use the repository name of artifact model
As we store the repository name in the artifact table, we can use it direclty in the code to reduce the database query

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-26 13:37:09 +08:00
Wang Yan
948d45604c Revise the GC job flow,
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-02-24 18:29:55 +08:00
Wenkai Yin
9312b788dc Upgrade the artifact table
Split the table artifact into artifact and tags, and populate related data

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-21 20:37:31 +08:00
Wang Yan
f160505686 add artifact trash manager
1, move the deleted artifact into trash
2, disable GC to delete the untagged manifest

Signed-off-by: wang yan <wangyan@vmware.com>
2020-02-19 14:52:58 +08:00
Wenkai Yin
93731eeb2e Support add/remove label to/from artifact
This commit add supporting for adding/removing label to/from artifacts and populates labels when listing artifacts

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-13 10:46:23 +08:00
Wenkai Yin
7dc28bcff9 Add foreign key to avoid the concurrent issue
Add foreign key to avoid the concurrent issue when operating the artifacts, tags and references

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-01-21 16:03:51 +08:00