Commit Graph

237 Commits

Author SHA1 Message Date
Wang Yan
3338f27189
support accessory in either order (#19375)
In certain cases, the OCI client may push the subject artifact and accessory in either order.
Therefore, it is necessary to handle situations where the client pushes the accessory ahead of the subject artifact.

Signed-off-by: wang yan <wangyan@vmware.com>
2023-09-23 02:42:58 +08:00
stonezdj(Daojun Zhang)
3de778e9f1
Fix incorrect artifact and scanned artifact count issue (#19106)
* Fix incorrect artifact and scanned artifact count issue

  fixes #19009 #19020 #19013

Signed-off-by: stonezdj <daojunz@vmware.com>

* fix issue

Signed-off-by: stonezdj <daojunz@vmware.com>

---------

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-07 08:40:26 +08:00
stonezdj(Daojun Zhang)
d4aa9b13c4
Add vulnerability search API (#18924)
use q.Query to pass all query conditions

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-19 10:17:14 +08:00
stonezdj(Daojun Zhang)
93e428d0d2
Add security hub summary API (#18872)
include WithCVE, WithArtifact option

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-12 19:18:08 +08:00
stonezdj(Daojun Zhang)
d84b1d07d2
Update table scan_report and extract cvss_v3_score from vendor attribute (#18854)
For better performance when query cve information, add summary information to scan_report
    Extract cve_score from vendor attribute in vulnerability_record
    SQL migrate script for the update

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-06-29 17:30:50 +08:00
Chlins Zhang
02a1c417d4
refactor: migrate the redis command keys to scan (#18825)
Refine the cache interface, migrate the Keys to Scan, change the redis
underlying keys command to scan.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-06-29 11:28:19 +08:00
Chlins Zhang
fbeeaa7537
fix: add checkpoint when enqueue scan tasks for scan all (#18680)
Fix the scanAll cannot be stopped in case of large number of artifacts,
add the checkpoint before submit scan tasks, mark the scanAll stopped
flag in the redis.

Fixes: #18044

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-06-05 15:12:54 +08:00
Wang Yan
06aa87a72a
set tag pull time for proxy cache (#18731)
fixes #18708

to set the pull time of tag for the first time cache the artifact.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-05-26 13:08:37 +08:00
Chlins Zhang
362387f914
fix: improve the performance of list artifacts (#18610)
1. Change the query for listing tasks of scan which can use the db
   index.
2. Add the gin index for task.extra_attrs.report_uuids

Fixes: #18013

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-04-30 09:10:28 +08:00
Wang Yan
deaecf2de5
add subject artifact repo (#18394)
add suject_artifact_repo column in the table artifact_accessory

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-22 21:48:09 +08:00
Wang Yan
2f380495bf
revert subject id in the accessory (#18377)
DO not replact id with digest and just add digest into the accessory table

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-22 10:58:30 +08:00
Shengwen YU
4efee54f07
feat: upgrade mockery version (#18332)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2023-03-11 08:16:11 +08:00
Wang Yan
295260b7a3
replace subject id with digest (#18278)
Since it has to support push subject and accessories in either order, it has to replace digest with id

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-08 17:03:12 +08:00
hyeyoung-leee
2c287cf27a
Modified to check the expiration time of the allowlist when determining the vulnerability of an artifact (#18106)
Signed-off-by: hyeyoung-lee <hyeyoung.lee@nhn.com>
2023-03-08 11:25:40 +08:00
stonezdj(Daojun Zhang)
5c0266e719
Get job log by job_id in worker (#18261)
Get job by job_id in redis

  Get the last 10MB of data if it exceeds 10MB

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-03-06 13:19:10 +08:00
Chlins Zhang
90db04e92d
refactor: refact the webhook API and life process (#18255)
refactor: refact the notification job API and life process

1. Introduce new APIs for webhook jobs management.
2. Refact legacy APIs for backforward compatible.
3. Migrate the webhook jobs process to unified execution/task framework.

Closes: #18210

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-03 10:17:47 +08:00
Chlins Zhang
da1637e1d3
fix: resolve the oidc or ldap group user cannot export cve (#18219)
Remove the project filter in the scan data export job as they have been
validated by API handler, fix the oidc or ldap group users cannot export
cve.

Fixes: #18112

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-02-20 14:22:27 +08:00
Wang Yan
738fde7d3b
remove chartmuseum backend (#18191)
Harbor deprecates chartmuseum as of v2.8.0

Epic: https://github.com/goharbor/harbor/issues/17958

Discussion: https://github.com/goharbor/harbor/discussions/15057

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-02-16 18:11:05 +08:00
stonezdj(Daojun Zhang)
64c03e8679
Update the execution status after tasks stopped (#17875)
Fixes #17862

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-12-01 21:17:35 +08:00
Wang Yan
18a3373725
bump beego (#17801)
* bump beego

upgrade beego version from v1.10.12 to v2.0.5

1, beego v2 vserver/web refactor
2, beego v2 context refactor
3, beego v2 session refactor
4, beego v2 cache refactor
5, beego v2 orm refactor

Signed-off-by: MinerYang <yminer@vmware.com>
2022-11-24 18:07:42 +08:00
stonezdj(Daojun Zhang)
57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
yanggang
0f4e2daf4c
Replaced io/ioutil with "os / io" package. (#17792)
Signed-off-by: yanggang <gang.yang@daocloud.io>

update typo

Signed-off-by: yminer <yminer@vmware.com>

Signed-off-by: yminer <yminer@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2022-11-17 10:02:29 +08:00
stonezdj(Daojun Zhang)
39ca918ffe
Add jobservice monitoring api list pool, worker and stop running task (#17658)
Add REST API to list job pool, worker, stop running task

  Add jobservice handler to retrieve configuration
  Add RBAC for jobservice monitoring dashboard
  Add REST API to list pool, worker and stop running task

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-03 10:47:40 +08:00
Chlins Zhang
c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
stonezdj(Daojun Zhang)
3050e856d9
Add check when adding project member (#17701)
Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-10-28 15:06:27 +08:00
Chenyu Zhang
5d7a549620
upgrade: bump up mockery to v2.14.0 (#17415)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-22 14:21:13 +08:00
Chenyu Zhang
19e73174e2
refactor: refact codebase of cve export process and filter (#17254)
1. Refact the scan export FilterProcessor interface
2. Optmize the sql template for export cve report
3. Update the process of cve export job

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-28 16:05:12 +08:00
Loong Dai
09371b48e8
lint: sort imports (#17131)
* lint: add goimports

Signed-off-by: Loong Dai <loong.dai@intel.com>
2022-07-20 11:33:08 +08:00
prahaladdarkin
130452111b
Vulnerability scan data export functionality (#15998)
Vulnerability Scan Data (CVE) Export Functionality
Proposal - goharbor/community#174
Closes - https://github.com/goharbor/harbor/issues/17150
Changes:
* CVE Data export to CSV with filtering support.
* Implement CSV data export job for creating CSVs
* APIs to trigger CSV export job executions

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-11 16:35:04 +08:00
Chenyu Zhang
1a1ce634cc
Fix the process of cache layer (#17010)
fix: fix cache layer issues (#16995,#16997,#16996,#17038)

1. Load config and initialize cache layer in jobservice(for GC)
2. Cache artifact by digest the key should contains repository name
3. Repository cache cleanup error when update
4. Skip save cache when request ctx in transaction

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 16:11:53 +08:00
stonezdj(Daojun Zhang)
dcccb44db8
Add options to the user.Count method (#16285)
Fixes #16269, exclude the admin account by default
  Add excludeDefaultAdmin method -- exclude default admin by option
  Update authModeCanBeModified method -- the user count should be 0 without admin

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-06 13:57:31 +08:00
stonezdj(Daojun Zhang)
aa63569de2
Fix scan log mismatch issue (#17085)
Add checks in label

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 18:12:38 +08:00
stonezdj(Daojun Zhang)
c639257ba7
Support stop purge audit log job (#17033)
fixes #17006

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 11:15:11 +08:00
prahaladdarkin
7024442f25
Project name validation to prevent system artifact reserved project names (#17029)
Include system artifact size in system storage space calculation.

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-21 12:51:29 +08:00
MinerYang
e9378ea00e
clean up unused mock file&& regenerate mocks (#17025)
clean up unused mock file&& re gen_mocks

Signed-off-by: yminer <yminer@vmware.com>
2022-06-17 18:19:23 +08:00
Maksym Trofimenko
9a3cb4a041
GDPR compliant deletion of Users (#16859)
fixes #16697

Signed-off-by: Maksym Trofimenko <maksym@container-registry.com>
2022-06-16 14:28:15 +02:00
stonezdj(Daojun Zhang)
e778ec2edf
Add audit_log forward endpoint (#16914)
add config item to set log forward endpoint
  fallback the audit log to default log when endpoint in error

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-10 10:59:40 +08:00
prahaladdarkin
4d062c33d1
Introduce system artifact manager cleanup job (#16879)
Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-09 14:16:50 +08:00
MinerYang
9b22e2943f
bumpup mockery to v1.12.3 (#16955)
Signed-off-by: yminer <yminer@vmmware.com>

Co-authored-by: yminer <yminer@vmmware.com>
2022-06-07 16:04:33 +08:00
Chenyu Zhang
b356d58253
Merge pull request #16861 from chlins/feat/cache-layer-for-manifest
feat(manifest): introduce cache layer for manifest (#16459)
2022-05-27 14:03:42 +08:00
stonezdj
1839965c7a Add REST API for purge audit log
Add rest API for purge audit log
  Extract scheduler, execution, task from GC controller for common use

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-05-23 16:58:31 +08:00
chlins
7eed9db404 feat(manifest): introduce cache layer for manifest (#16459)
Implement cache layer for resource manifest, it will read manifest
from cache instead of proxying to distribution if enabled.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-05-19 10:25:50 +08:00
stonezdj
3ac7281177 Add job service to purge audit_log
Add jobservice type PurgeAuditLog

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-05-16 21:52:58 +08:00
prahaladdarkin
27ec871185
System Artifact Manager database schema creation, model definitons, and tests (#16678)
Closes:
https://github.com/goharbor/harbor/issues/16540
https://github.com/goharbor/harbor/issues/16541
https://github.com/goharbor/harbor/issues/16542

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-05-09 17:32:57 +08:00
MinerYang
083b44e719
bump up astaxie/beego@v1.12.1 to beego/beego/@v1.12.7 (#16770)
Signed-off-by: yminer <yminer@vmmware.com>

goo mod tidy

Signed-off-by: yminer <yminer@vmmware.com>

Co-authored-by: yminer <yminer@vmmware.com>
2022-04-28 11:58:44 +08:00
chlins
8c223135e7 feat: add cache layer for artifact (#16739)
Implement cache layer for resource artifact and define common
cache manager and workflow. Also add cache related options to
configuration yaml.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-04-25 18:01:45 +08:00
stonezdj(Daojun Zhang)
1449988353
Delete unused files and functions (#16599)
testing/apitest/apilib is not used after harbor 2.4 refactor
 no testcase removed

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-03-29 14:19:36 +08:00
chlins
5be0b93caa refactor: import go-redis to core as replacement of redigo
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-03-15 20:44:44 +08:00
He Weiwei
abd85284d2
feat: merge local and remote tags for repo of proxy cache project in list API (#16394)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-02-24 10:27:45 +08:00
Wang Yan
01c6f6084b
modify artifact copy api to support cosign (#16194)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-01-17 15:52:14 +08:00