Commit Graph

253 Commits

Author SHA1 Message Date
stonezdj(Daojun Zhang) 30767f6612
Response an error message when there is incomplete sbom generate job (#20526)
Response an error message when there is uncomplete sbom generate job

Signed-off-by: stonezdj <stone.zhang@broadcom.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2024-05-31 17:20:53 +08:00
stonezdj(Daojun Zhang) 6d782ae695
Separate the execution vendor type sbom from image_scan (#20504)
Add vendor type SBOM for execution
  fixes #20495

Signed-off-by: stonezdj <stone.zhang@broadcom.com>
2024-05-29 15:45:28 +08:00
stonezdj(Daojun Zhang) 1f0c8289a5
Add sbom_report table to store sbom related information (#20473)
fixes #20445
  Refactor scan/base_controller.go
  Move MakeReportPlaceholder, GetReportPlaceholder, GetSummary to vul and sbom scanHandler

Signed-off-by: stonezdj <stone.zhang@broadcom.com>
2024-05-24 08:48:55 +00:00
Wang Yan 2977fec006
fix issue 19928 (#20409)
* fix issue 19928

it needs to consider the user who is in any group that has been granted with the project admin role.

Signed-off-by: wang yan <wangyan@vmware.com>
2024-05-15 13:07:30 +08:00
stonezdj(Daojun Zhang) 232f9ba7ea
Skip scan in-toto sbom artifact (#20415)
fixes #20337

Signed-off-by: stonezdj <stone.zhang@broadcom.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2024-05-13 17:12:04 +08:00
stonezdj(Daojun Zhang) fba4c40c65
Delete scan_report when accessory is removed (#20365)
Signed-off-by: stonezdj <stone.zhang@broadcom.com>
2024-04-27 01:56:30 +00:00
stonezdj(Daojun Zhang) b7d4bf0d07
Log and skip adapter ping error when retrieve adapter capability (#20314)
Signed-off-by: stonezdj <stone.zhang@broadcom.com>
2024-04-22 09:43:04 +00:00
stonezdj(Daojun Zhang) 654aa8edcf
Add generate SBOM feature (#20251)
* Add SBOM scan feature

  Add scan handler for sbom
  Delete previous sbom accessory before the job service

Signed-off-by: stonezdj <daojunz@vmware.com>

* fix issue

Signed-off-by: stonezdj <stone.zhang@broadcom.com>

---------

Signed-off-by: stonezdj <daojunz@vmware.com>
Signed-off-by: stonezdj <stone.zhang@broadcom.com>
Co-authored-by: stonezdj <daojunz@vmware.com>
2024-04-16 13:34:19 +00:00
MinerYang e9d2f50669
update mockery to v2.42.2 (#20258)
Signed-off-by: yminer <yminer@vmware.com>
2024-04-11 03:37:59 +00:00
Wang Yan 461a5fa50d
add stop sbom scanning API (#20200)
* add stop sbom scanning API

1. [UI] support to stop sbom scanning #20200
2. add type for stop scanning api, make it able to support both vulnerability and sbom.
3. refactor the db query to support multiple extra attributes.

Signed-off-by: wang yan <wangyan@vmware.com>
Signed-off-by: xuelichao <xuel@vmware.com>
Co-authored-by: xuelichao <xuel@vmware.com>
2024-04-09 16:07:47 +08:00
stonezdj(Daojun Zhang) 80a9c688fc
panic due to mark retention task error (#20161)
panic due to mark retention task error

fixes #20129

Signed-off-by: stonezdj <daojunz@vmware.com>
2024-03-26 04:52:17 +00:00
Maksym Trofimenko 996e57b511
Feature: GDPR compliant audit logs (#17396) 2023-11-20 21:31:22 +01:00
Wang Yan 5c02fd807e
add permission api (#19543)
The permission api targets to return the full set of permissons for robot to use.
And only system and project admin have the access

Signed-off-by: wang yan <wangyan@vmware.com>
2023-11-09 09:47:07 +08:00
stonezdj(Daojun Zhang) da949bfc3f
Delete project member when delete project (#19523)
Signed-off-by: stonezdj <daojunz@vmware.com>
2023-11-08 20:51:21 +08:00
Daniel Jiang 3f7c605cf5
[WIP] Remove vendor folder from harbor code base (#19508)
* Remove vendor folder from harbor code base

Signed-off-by: Daniel Jiang <jiangd@vmware.com>

* debug ut failure

* debug failure

* debug failure

---------

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2023-11-05 14:37:25 +08:00
stonezdj(Daojun Zhang) 7b0beed934
Delete tag retention rule and tag immutable rule when deleting project (#19390)
fixes #18250

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-10-24 04:28:16 +00:00
Wang Yan 3338f27189
support accessory in either order (#19375)
In certain cases, the OCI client may push the subject artifact and accessory in either order.
Therefore, it is necessary to handle situations where the client pushes the accessory ahead of the subject artifact.

Signed-off-by: wang yan <wangyan@vmware.com>
2023-09-23 02:42:58 +08:00
stonezdj(Daojun Zhang) 3de778e9f1
Fix incorrect artifact and scanned artifact count issue (#19106)
* Fix incorrect artifact and scanned artifact count issue

  fixes #19009 #19020 #19013

Signed-off-by: stonezdj <daojunz@vmware.com>

* fix issue

Signed-off-by: stonezdj <daojunz@vmware.com>

---------

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-07 08:40:26 +08:00
stonezdj(Daojun Zhang) d4aa9b13c4
Add vulnerability search API (#18924)
use q.Query to pass all query conditions

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-19 10:17:14 +08:00
stonezdj(Daojun Zhang) 93e428d0d2
Add security hub summary API (#18872)
include WithCVE, WithArtifact option

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-12 19:18:08 +08:00
stonezdj(Daojun Zhang) d84b1d07d2
Update table scan_report and extract cvss_v3_score from vendor attribute (#18854)
For better performance when query cve information, add summary information to scan_report
    Extract cve_score from vendor attribute in vulnerability_record
    SQL migrate script for the update

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-06-29 17:30:50 +08:00
Chlins Zhang 02a1c417d4
refactor: migrate the redis command keys to scan (#18825)
Refine the cache interface, migrate the Keys to Scan, change the redis
underlying keys command to scan.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-06-29 11:28:19 +08:00
Chlins Zhang fbeeaa7537
fix: add checkpoint when enqueue scan tasks for scan all (#18680)
Fix the scanAll cannot be stopped in case of large number of artifacts,
add the checkpoint before submit scan tasks, mark the scanAll stopped
flag in the redis.

Fixes: #18044

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-06-05 15:12:54 +08:00
Wang Yan 06aa87a72a
set tag pull time for proxy cache (#18731)
fixes #18708

to set the pull time of tag for the first time cache the artifact.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-05-26 13:08:37 +08:00
Chlins Zhang 362387f914
fix: improve the performance of list artifacts (#18610)
1. Change the query for listing tasks of scan which can use the db
   index.
2. Add the gin index for task.extra_attrs.report_uuids

Fixes: #18013

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-04-30 09:10:28 +08:00
Wang Yan deaecf2de5
add subject artifact repo (#18394)
add suject_artifact_repo column in the table artifact_accessory

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-22 21:48:09 +08:00
Wang Yan 2f380495bf
revert subject id in the accessory (#18377)
DO not replact id with digest and just add digest into the accessory table

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-22 10:58:30 +08:00
Shengwen YU 4efee54f07
feat: upgrade mockery version (#18332)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2023-03-11 08:16:11 +08:00
Wang Yan 295260b7a3
replace subject id with digest (#18278)
Since it has to support push subject and accessories in either order, it has to replace digest with id

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-08 17:03:12 +08:00
hyeyoung-leee 2c287cf27a
Modified to check the expiration time of the allowlist when determining the vulnerability of an artifact (#18106)
Signed-off-by: hyeyoung-lee <hyeyoung.lee@nhn.com>
2023-03-08 11:25:40 +08:00
stonezdj(Daojun Zhang) 5c0266e719
Get job log by job_id in worker (#18261)
Get job by job_id in redis

  Get the last 10MB of data if it exceeds 10MB

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-03-06 13:19:10 +08:00
Chlins Zhang 90db04e92d
refactor: refact the webhook API and life process (#18255)
refactor: refact the notification job API and life process

1. Introduce new APIs for webhook jobs management.
2. Refact legacy APIs for backforward compatible.
3. Migrate the webhook jobs process to unified execution/task framework.

Closes: #18210

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-03 10:17:47 +08:00
Chlins Zhang da1637e1d3
fix: resolve the oidc or ldap group user cannot export cve (#18219)
Remove the project filter in the scan data export job as they have been
validated by API handler, fix the oidc or ldap group users cannot export
cve.

Fixes: #18112

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-02-20 14:22:27 +08:00
Wang Yan 738fde7d3b
remove chartmuseum backend (#18191)
Harbor deprecates chartmuseum as of v2.8.0

Epic: https://github.com/goharbor/harbor/issues/17958

Discussion: https://github.com/goharbor/harbor/discussions/15057

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-02-16 18:11:05 +08:00
stonezdj(Daojun Zhang) 64c03e8679
Update the execution status after tasks stopped (#17875)
Fixes #17862

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-12-01 21:17:35 +08:00
Wang Yan 18a3373725
bump beego (#17801)
* bump beego

upgrade beego version from v1.10.12 to v2.0.5

1, beego v2 vserver/web refactor
2, beego v2 context refactor
3, beego v2 session refactor
4, beego v2 cache refactor
5, beego v2 orm refactor

Signed-off-by: MinerYang <yminer@vmware.com>
2022-11-24 18:07:42 +08:00
stonezdj(Daojun Zhang) 57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
yanggang 0f4e2daf4c
Replaced io/ioutil with "os / io" package. (#17792)
Signed-off-by: yanggang <gang.yang@daocloud.io>

update typo

Signed-off-by: yminer <yminer@vmware.com>

Signed-off-by: yminer <yminer@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2022-11-17 10:02:29 +08:00
stonezdj(Daojun Zhang) 39ca918ffe
Add jobservice monitoring api list pool, worker and stop running task (#17658)
Add REST API to list job pool, worker, stop running task

  Add jobservice handler to retrieve configuration
  Add RBAC for jobservice monitoring dashboard
  Add REST API to list pool, worker and stop running task

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-03 10:47:40 +08:00
Chlins Zhang c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
stonezdj(Daojun Zhang) 3050e856d9
Add check when adding project member (#17701)
Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-10-28 15:06:27 +08:00
Chenyu Zhang 5d7a549620
upgrade: bump up mockery to v2.14.0 (#17415)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-22 14:21:13 +08:00
Chenyu Zhang 19e73174e2
refactor: refact codebase of cve export process and filter (#17254)
1. Refact the scan export FilterProcessor interface
2. Optmize the sql template for export cve report
3. Update the process of cve export job

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-28 16:05:12 +08:00
Loong Dai 09371b48e8
lint: sort imports (#17131)
* lint: add goimports

Signed-off-by: Loong Dai <loong.dai@intel.com>
2022-07-20 11:33:08 +08:00
prahaladdarkin 130452111b
Vulnerability scan data export functionality (#15998)
Vulnerability Scan Data (CVE) Export Functionality
Proposal - goharbor/community#174
Closes - https://github.com/goharbor/harbor/issues/17150
Changes:
* CVE Data export to CSV with filtering support.
* Implement CSV data export job for creating CSVs
* APIs to trigger CSV export job executions

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-11 16:35:04 +08:00
Chenyu Zhang 1a1ce634cc
Fix the process of cache layer (#17010)
fix: fix cache layer issues (#16995,#16997,#16996,#17038)

1. Load config and initialize cache layer in jobservice(for GC)
2. Cache artifact by digest the key should contains repository name
3. Repository cache cleanup error when update
4. Skip save cache when request ctx in transaction

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 16:11:53 +08:00
stonezdj(Daojun Zhang) dcccb44db8
Add options to the user.Count method (#16285)
Fixes #16269, exclude the admin account by default
  Add excludeDefaultAdmin method -- exclude default admin by option
  Update authModeCanBeModified method -- the user count should be 0 without admin

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-06 13:57:31 +08:00
stonezdj(Daojun Zhang) aa63569de2
Fix scan log mismatch issue (#17085)
Add checks in label

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 18:12:38 +08:00
stonezdj(Daojun Zhang) c639257ba7
Support stop purge audit log job (#17033)
fixes #17006

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 11:15:11 +08:00
prahaladdarkin 7024442f25
Project name validation to prevent system artifact reserved project names (#17029)
Include system artifact size in system storage space calculation.

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-21 12:51:29 +08:00