Commit Graph

212 Commits

Author SHA1 Message Date
Chlins Zhang
2148ee9fbe
migration: add sql to migrate the old webhook job to new way (#18342)
Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-16 16:27:43 +08:00
Yang Jiao
1e38565aae
Migrate robot account and notification policy related to Chartmuseum (#18316)
Migrate robot permission and notification policy related to Chartmuseum

Update robot permission and notification policy as Harbor v2.8.0 deprecates chartmuseum.
Harbor deprecates chartmuseum as of v2.8.0
Epic: goharbor#17958
Discussion: goharbor#15057

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2023-03-10 12:19:11 +08:00
Wang Yan
295260b7a3
replace subject id with digest (#18278)
Since it has to support push subject and accessories in either order, it has to replace digest with id

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-08 17:03:12 +08:00
Yang Jiao
1f3f732bd6
Update the replication rule related to the Chartmuseum (#18274)
Update the registry and replication rule related to the Chartmuseum

Update replication_policy and registry as Harbor v2.8.0 deprecates chartmuseum.
Harbor deprecates chartmuseum as of v2.8.0
Epic: #17958
Discussion: #15057

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2023-03-02 17:28:27 +08:00
Wang Yan
74dfc905ec
remove the redundant data from table artifact_blob (#17937)
use sql in the migration process to delete all the useless data of table artifact_blob

Signed-off-by: Wang Yan <wangyan@vmware.com>

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-12-13 15:32:57 +08:00
stonezdj(Daojun Zhang)
57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
Chlins Zhang
e81067badb
fix: cherry pick 262 sql for migrations (#17735)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-03 10:05:46 +08:00
Chlins Zhang
c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
Chenyu Zhang
70a95a9696
fix: reduce the high db cpu usage for tag retention (#17296)
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: #14708

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-04 10:08:02 +08:00
Chenyu Zhang
41d10d571c
fix: repair execution status when it inconsistent (#17128)
Add migrations sql to repair the execution status when it does not
consistent with task status.

Closes: #17114

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 17:02:03 +08:00
stonezdj(Daojun Zhang)
8115484f32
Create index for performance (#17022)
Add index to avoid full table scan
Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-04 14:38:05 +08:00
Chenyu Zhang
0cf036e73a
migrate: add db index on artifact repository name (#17053)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-22 14:34:40 +08:00
Chenyu Zhang
5de1a62002
migrate: add db index on artifact repository name (#17035)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-20 16:01:04 +08:00
prahaladdarkin
27ec871185
System Artifact Manager database schema creation, model definitons, and tests (#16678)
Closes:
https://github.com/goharbor/harbor/issues/16540
https://github.com/goharbor/harbor/issues/16541
https://github.com/goharbor/harbor/issues/16542

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-05-09 17:32:57 +08:00
chlins
f7d10474d1 migrations: correct project metadata public value
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-03-28 15:17:20 +08:00
stonezdj
9d480a84d2 Create index on task with job_id
Missing index with job_id, when query task with job_id, it cause a full table scan, caused performance issue
  Fixes #15271

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-03-02 20:51:36 +08:00
Shengwen Yu
aacebcc8c4 fix: vulnerability_record & report_vulnerability_record table "integer out of range" error
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-02-09 10:19:46 +08:00
He Weiwei
8554b7d7c5
refactor: replace lib/pq with jackc/pgx (#16267)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-27 11:09:37 +08:00
He Weiwei
89fe7b0819
Move removing old version scan reports of trivy to 2.4.2 (#16261)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-24 14:05:24 +08:00
He Weiwei
097efb201b
fix: Clean old version scan reports of trivy (#16235)
From harbor 2.4, the trivy does not support scan report v1.0, we need to remove the old version scan reports from the DB.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-17 13:59:28 +08:00
Wang Yan
742e7ded00
add accessory dao service (#16045)
Signed-off-by: wang yan <wangyan@vmware.com>
2021-12-03 14:34:02 +08:00
He Weiwei
b390112f5a
fix: convert severity from negligible to none before saving to db (#15791)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-10-14 16:02:38 +08:00
stonezdj
06715af303 Allow empty email attribute for ldap/oidc user
Define user.Email as sql.NullString to avoid unique constraint when email is empty in LDAP/OIDC
  Separate the common/models/User with the pkg/user/dao/User
  Fixes #10400
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-09-30 14:35:18 +08:00
He Weiwei
ecc1a04c92
refactor: implement a lock free quota (#15399)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-09-17 11:26:37 +08:00
Ziming Zhang
98cef43ead limit replication bandwidth
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-09-14 11:23:28 +08:00
stonezdj
c163bc8317 Delete users under auth_mode other than db_auth
The following information should cleanup before delete user:
  Delete project member of this user.
  Delete oidc_user when auth_mode is oidc_auth.
  Fixes #8424
  It also removes the deleted user from project member and the deleted condition in the project member query for consistency
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-07-12 11:31:40 +08:00
ChenYu Zhang
a038ba672f
perf: add 2.2.3 schema up sql to master (#15027)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2021-06-01 14:12:25 +08:00
Wenkai Yin
c585f92034 Fix bug of migration sql for replication policy
Fix bug of migration sql for replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-26 18:57:23 +08:00
Wang Yan
73bd373a75
create index for audit (#14930)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-20 19:50:13 +08:00
Wenkai Yin
dc059a9a8f Improve the performance of artifact related APIs
Improve the performance of artifact related APIs by adding indexes and refactoring sql logic

Closes #13890 #14813 #14814

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-20 11:25:43 +08:00
Wenkai Yin
c04f3a2aac Fix duplicate execution record issue
When the core service cannot response the checkin request in time, duplicated execution records may be created, this commit introduces the revision column to make sure there is only one record for one schedule trigger

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-12 15:40:52 +08:00
He Weiwei
476732df89 fix: change art_size to bigint in migrations
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-05-11 04:01:02 +00:00
Wenkai Yin
710c80078b Refactor the replication policy destination namespace logic
Support specifying what part of the repository will be replaced by the provided namespace

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-04-28 10:16:20 +08:00
Wenkai Yin
8b1817be0f Fix the consume too much CPU issue
1. Update execution status during the upgrade
2. Refine the execution sweeper

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-17 13:34:24 +08:00
Wenkai Yin
43df3bf8a4 Add upgrade sql file introduced in 2.1.4
1. Add upgrade sql file introduced in 2.1.4
2. Minor improvement for task/execution to cover corner cases

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-15 16:48:51 +08:00
Ziming Zhang
ec83f49a1a fix(retention) migrate sql error
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-02-05 09:09:26 +00:00
Wang Yan
24ec772978
fix gc migration issue (#14174)
For the upgrade path v1.10 - v2.1.* - v2.2.0, if user doesn't reset the GC schdule that was created in 1.10 in the v2.1,
the job parameters will keep empty in the database. The fix gives a default value for the schedule record.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-02-05 12:25:24 +08:00
Wenkai Yin(尹文开)
50a1e85095
Make sure the revision of execution isn't null during the upgrade (#14085)
Make sure the revision of execution isn't null during the upgrade
Fixes #14075

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-27 10:10:36 +08:00
Wenkai Yin
7c072e17a6 Fix the legacy scheduled job issue for GC/scan all
Fix the legacy scheduled job issue for GC/scan all

Fixes #13968

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-15 22:02:36 +08:00
He Weiwei
4a326aa8b0 chore: delete records of scan_report
The report in previous scan_report records not work well the
vulnerabilities stored in the schema table, so delete the scan_report
records.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-01-08 03:39:11 +00:00
Ziming Zhang
8faa76a1b6 feat(retention) refactor task manager
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-01-05 12:08:03 +08:00
prahaladdarkin
a890b28e1e
Store vulnerability data from scanner into a relational format (#13616)
feat: Store vulnerability report from scanner into a relational format

Convert vulnerability report JSON obtained  from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2020-12-25 08:47:46 +08:00
He Weiwei
3831e82b20
refactor: remove code of admin job (#13819)
Remove code of admin job as it's not needed by scan all/gc now.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-22 11:48:16 +08:00
Wang Yan
9bc6f3cee4
fix robot account update issue (#13741)
* fix robot account update issue

enable the update method to support both v1 & v2 robot update

Signed-off-by: Wang Yan <wangyan@vmware.com>

* resolve review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 20:01:26 +08:00
Wang Yan
6bc1047013
migration admin job data (#13766)
1, migrate gc and scan all schedule to schedule/task/exectuion
2, migrate gc history to task/execution

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 16:35:24 +08:00
Daniel Jiang
b0c8cadad7
Add default CVE allowlist to project library (#13770)
fixes #12700

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-16 14:20:56 +08:00
Wenkai Yin
69808f033e Tiny fixes for task manager
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-12-14 17:26:32 +08:00
He Weiwei
08580f9fec
refactor(scan): refactor scan/scan all job to task manager (#13684)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 13:34:35 +08:00
Wang Yan
39e1a4f2b4 add extra attributes in the schedule table
Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-14 02:28:52 +08:00
wang yan
1bb79d402d update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 16:08:52 +08:00