Commit Graph

224 Commits

Author SHA1 Message Date
Shengwen YU
d0a5032b84
feat: bump up golang-runtime from 1.19.3 to 1.21.4; upgrade MOCKERY_VERSION; upgrade golangci-lint; fix mock issue (#19608)
feat: upgrade golang-runtime from 1.19.3 to 1.21.4; upgrade golangci-lint version to v1.51.2; upgrade MOCKERY_VERSION to v2.22.1; fix mock issue

Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2023-11-24 15:11:45 +08:00
Chlins Zhang
3db3d60c02
[cherry-pick] refactor: migrate the redis command keys to scan (#19148)
refactor: migrate the redis command keys to scan

Refine the cache interface, migrate the Keys to Scan, change the redis
underlying keys command to scan.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-08-16 11:21:34 +08:00
Wang Yan
ca8bbadb58
set tag pull time for proxy cache (#18742)
fixes #18708

to set the pull time of tag for the first time cache the artifact.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-05-29 09:42:02 +08:00
Chlins Zhang
e180723bdc
[cherry-pick] fix: improve the performance of list artifacts (#18632)
fix: improve the performance of list artifacts

1. Change the query for listing tasks of scan which can use the db
   index.
2. Add the gin index for task.extra_attrs.report_uuids

Fixes: #18013

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-05-05 14:15:55 +08:00
Chlins Zhang
67ff82ff99
[cherry-pick] fix: resolve the oidc or ldap group user cannot export cve (#18227)
fix: resolve the oidc or ldap group user cannot export cve

Remove the project filter in the scan data export job as they have been
validated by API handler, fix the oidc or ldap group users cannot export
cve.

Fixes: #18112

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-02-16 10:37:57 +08:00
stonezdj(Daojun Zhang)
64c03e8679
Update the execution status after tasks stopped (#17875)
Fixes #17862

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-12-01 21:17:35 +08:00
Wang Yan
18a3373725
bump beego (#17801)
* bump beego

upgrade beego version from v1.10.12 to v2.0.5

1, beego v2 vserver/web refactor
2, beego v2 context refactor
3, beego v2 session refactor
4, beego v2 cache refactor
5, beego v2 orm refactor

Signed-off-by: MinerYang <yminer@vmware.com>
2022-11-24 18:07:42 +08:00
stonezdj(Daojun Zhang)
57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
yanggang
0f4e2daf4c
Replaced io/ioutil with "os / io" package. (#17792)
Signed-off-by: yanggang <gang.yang@daocloud.io>

update typo

Signed-off-by: yminer <yminer@vmware.com>

Signed-off-by: yminer <yminer@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2022-11-17 10:02:29 +08:00
stonezdj(Daojun Zhang)
39ca918ffe
Add jobservice monitoring api list pool, worker and stop running task (#17658)
Add REST API to list job pool, worker, stop running task

  Add jobservice handler to retrieve configuration
  Add RBAC for jobservice monitoring dashboard
  Add REST API to list pool, worker and stop running task

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-03 10:47:40 +08:00
Chlins Zhang
c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
stonezdj(Daojun Zhang)
3050e856d9
Add check when adding project member (#17701)
Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-10-28 15:06:27 +08:00
Chenyu Zhang
5d7a549620
upgrade: bump up mockery to v2.14.0 (#17415)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-22 14:21:13 +08:00
Chenyu Zhang
19e73174e2
refactor: refact codebase of cve export process and filter (#17254)
1. Refact the scan export FilterProcessor interface
2. Optmize the sql template for export cve report
3. Update the process of cve export job

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-28 16:05:12 +08:00
Loong Dai
09371b48e8
lint: sort imports (#17131)
* lint: add goimports

Signed-off-by: Loong Dai <loong.dai@intel.com>
2022-07-20 11:33:08 +08:00
prahaladdarkin
130452111b
Vulnerability scan data export functionality (#15998)
Vulnerability Scan Data (CVE) Export Functionality
Proposal - goharbor/community#174
Closes - https://github.com/goharbor/harbor/issues/17150
Changes:
* CVE Data export to CSV with filtering support.
* Implement CSV data export job for creating CSVs
* APIs to trigger CSV export job executions

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-11 16:35:04 +08:00
Chenyu Zhang
1a1ce634cc
Fix the process of cache layer (#17010)
fix: fix cache layer issues (#16995,#16997,#16996,#17038)

1. Load config and initialize cache layer in jobservice(for GC)
2. Cache artifact by digest the key should contains repository name
3. Repository cache cleanup error when update
4. Skip save cache when request ctx in transaction

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 16:11:53 +08:00
stonezdj(Daojun Zhang)
dcccb44db8
Add options to the user.Count method (#16285)
Fixes #16269, exclude the admin account by default
  Add excludeDefaultAdmin method -- exclude default admin by option
  Update authModeCanBeModified method -- the user count should be 0 without admin

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-06 13:57:31 +08:00
stonezdj(Daojun Zhang)
aa63569de2
Fix scan log mismatch issue (#17085)
Add checks in label

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 18:12:38 +08:00
stonezdj(Daojun Zhang)
c639257ba7
Support stop purge audit log job (#17033)
fixes #17006

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 11:15:11 +08:00
prahaladdarkin
7024442f25
Project name validation to prevent system artifact reserved project names (#17029)
Include system artifact size in system storage space calculation.

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-21 12:51:29 +08:00
MinerYang
e9378ea00e
clean up unused mock file&& regenerate mocks (#17025)
clean up unused mock file&& re gen_mocks

Signed-off-by: yminer <yminer@vmware.com>
2022-06-17 18:19:23 +08:00
Maksym Trofimenko
9a3cb4a041
GDPR compliant deletion of Users (#16859)
fixes #16697

Signed-off-by: Maksym Trofimenko <maksym@container-registry.com>
2022-06-16 14:28:15 +02:00
stonezdj(Daojun Zhang)
e778ec2edf
Add audit_log forward endpoint (#16914)
add config item to set log forward endpoint
  fallback the audit log to default log when endpoint in error

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-10 10:59:40 +08:00
prahaladdarkin
4d062c33d1
Introduce system artifact manager cleanup job (#16879)
Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-09 14:16:50 +08:00
MinerYang
9b22e2943f
bumpup mockery to v1.12.3 (#16955)
Signed-off-by: yminer <yminer@vmmware.com>

Co-authored-by: yminer <yminer@vmmware.com>
2022-06-07 16:04:33 +08:00
Chenyu Zhang
b356d58253
Merge pull request #16861 from chlins/feat/cache-layer-for-manifest
feat(manifest): introduce cache layer for manifest (#16459)
2022-05-27 14:03:42 +08:00
stonezdj
1839965c7a Add REST API for purge audit log
Add rest API for purge audit log
  Extract scheduler, execution, task from GC controller for common use

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-05-23 16:58:31 +08:00
chlins
7eed9db404 feat(manifest): introduce cache layer for manifest (#16459)
Implement cache layer for resource manifest, it will read manifest
from cache instead of proxying to distribution if enabled.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-05-19 10:25:50 +08:00
stonezdj
3ac7281177 Add job service to purge audit_log
Add jobservice type PurgeAuditLog

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-05-16 21:52:58 +08:00
prahaladdarkin
27ec871185
System Artifact Manager database schema creation, model definitons, and tests (#16678)
Closes:
https://github.com/goharbor/harbor/issues/16540
https://github.com/goharbor/harbor/issues/16541
https://github.com/goharbor/harbor/issues/16542

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-05-09 17:32:57 +08:00
MinerYang
083b44e719
bump up astaxie/beego@v1.12.1 to beego/beego/@v1.12.7 (#16770)
Signed-off-by: yminer <yminer@vmmware.com>

goo mod tidy

Signed-off-by: yminer <yminer@vmmware.com>

Co-authored-by: yminer <yminer@vmmware.com>
2022-04-28 11:58:44 +08:00
chlins
8c223135e7 feat: add cache layer for artifact (#16739)
Implement cache layer for resource artifact and define common
cache manager and workflow. Also add cache related options to
configuration yaml.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-04-25 18:01:45 +08:00
stonezdj(Daojun Zhang)
1449988353
Delete unused files and functions (#16599)
testing/apitest/apilib is not used after harbor 2.4 refactor
 no testcase removed

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-03-29 14:19:36 +08:00
chlins
5be0b93caa refactor: import go-redis to core as replacement of redigo
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-03-15 20:44:44 +08:00
He Weiwei
abd85284d2
feat: merge local and remote tags for repo of proxy cache project in list API (#16394)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-02-24 10:27:45 +08:00
Wang Yan
01c6f6084b
modify artifact copy api to support cosign (#16194)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-01-17 15:52:14 +08:00
Chenyu Zhang
b417e877b5
Merge pull request #16107 from chlins/feat/async-update-artifact-pull_count
feat: async update artifact pull time and repository pull count
2022-01-05 11:14:28 +08:00
Wang Yan
2111703d8d
Cosign artifact api
1,update artifact list & delete api to support accessory
2, add list accesories api

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-01-05 11:13:40 +08:00
chlins
de7978e1b5 feat: async update artifact pull time and repository pull count
Signed-off-by: chlins <chenyuzh@vmware.com>
2021-12-24 11:17:44 +08:00
Wang Yan
742e7ded00
add accessory dao service (#16045)
Signed-off-by: wang yan <wangyan@vmware.com>
2021-12-03 14:34:02 +08:00
Wang Yan
6014646bcb
fixes gc dry run issue (#15804)
fixes #15332, for the dry run mode, gc job should not remove the untagged candidates.
To fix it, use the simulate untagged artifact deletion for dry-run.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-10-19 07:42:54 +08:00
stonezdj
06715af303 Allow empty email attribute for ldap/oidc user
Define user.Email as sql.NullString to avoid unique constraint when email is empty in LDAP/OIDC
  Separate the common/models/User with the pkg/user/dao/User
  Fixes #10400
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-09-30 14:35:18 +08:00
He Weiwei
ecc1a04c92
refactor: implement a lock free quota (#15399)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-09-17 11:26:37 +08:00
Shengwen YU
9cb266a8a0
Merge pull request #15548 from zyyw/ISSUE-14831
Add stop scan a particular artifact & stop scan all
2021-09-13 15:07:49 +08:00
Shengwen Yu
e2e3bcca1c feat: add stop scan & stop scan-all feature
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2021-09-13 11:19:21 +08:00
Wang Yan
b9228096dc
enable robot to support create project (#15461)
1, for admin only, the system level robot should contains the project creation access.
2, for not admin only, the system level robot can create project.
3, for the project that created by system level robot, use the admin ID as the ownerID.

No path for project level robot to create project.

Signed-off-by: wang yan <wangyan@vmware.com>
2021-08-24 09:34:02 +08:00
Wang Yan
e5a614967d
drop gc api of registryctl (#15325)
The API is desiged for read-only gc job, as the read only gc has already deprecated, remove this api accordingly.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-07-23 14:15:02 +08:00
stonezdj
c163bc8317 Delete users under auth_mode other than db_auth
The following information should cleanup before delete user:
  Delete project member of this user.
  Delete oidc_user when auth_mode is oidc_auth.
  Fixes #8424
  It also removes the deleted user from project member and the deleted condition in the project member query for consistency
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-07-12 11:31:40 +08:00
Wang Yan
1abe978e87 refactor job log
Move job service logger to new program model

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-06-29 13:37:45 +08:00