Commit Graph

873 Commits

Author SHA1 Message Date
stonezdj(Daojun Zhang)
80a9c688fc
panic due to mark retention task error (#20161)
panic due to mark retention task error

fixes #20129

Signed-off-by: stonezdj <daojunz@vmware.com>
2024-03-26 04:52:17 +00:00
Wang Yan
2eb5464603
add type for scanner metadata (#20108)
Signed-off-by: wang yan <wangyan@vmware.com>
2024-03-25 07:02:39 +00:00
MinerYang
a269b4f31c
Update support for artifactType for both manifest and index (#20030)
add artifact_type for artifact model to support artifactType filter

Signed-off-by: yminer <yminer@vmware.com>

add 2.11 sql schema & update index artifactType omitted

Signed-off-by: yminer <yminer@vmware.com>

update UT

update migrate sql for artifact_type

Signed-off-by: yminer <yminer@vmware.com>

remove debug line
2024-03-12 13:52:56 +00:00
Wang Yan
dbe9790147
add generate sbom object utility (#20097)
* add generate sbom object utility

Leverage the go-containerregistry to generate the oci object for sbom and add it as an accessory of the subject artifact.

Signed-off-by: wang yan <wangyan@vmware.com>

* remove vendor

Signed-off-by: wang yan <wangyan@vmware.com>

* resolve comments

Signed-off-by: wang yan <wangyan@vmware.com>

* fix ut

Signed-off-by: wang yan <wangyan@vmware.com>

* resolve comments

Signed-off-by: wang yan <wangyan@vmware.com>

* remove the todo comments

Signed-off-by: wang yan <wangyan@vmware.com>

---------

Signed-off-by: wang yan <wangyan@vmware.com>
2024-03-12 12:27:34 +08:00
Wang Yan
3782bab80a
add sbom settings for project (#20069)
Add a new switcher for sbom generation, by default is false.

Signed-off-by: wang yan <wangyan@vmware.com>
2024-03-01 14:40:02 +08:00
Antoine Jouve
73c2884e58
[Token/JWT] Update to golang-jwt v5.2.0 (#19802)
* feat: update to golang-jwt v5.2.0

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* fix: module issues and robot claims

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* fix: add missing time import

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* feat: set jwt validation leeway to 60s

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* fix: update leeways that were still set to 10s

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* feat: update go.sum

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* feat: add two leeway related test cases

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* fix: correct jwt audience validation

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* fix: gofmt v2_token.go

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* feat: take into account review comments

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

* feat: use a common constant to store JWT leeway

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>

---------

Signed-off-by: Antoine Jouve <ant.jouve@gmail.com>
Signed-off-by: Antoine Jouve <an-toine@users.noreply.github.com>
Co-authored-by: MinerYang <yminer@vmware.com>
2024-02-23 11:30:13 +08:00
zycupup
ee6f61c502
feat: volc cr adapter (#19456)
feat: support volcEngine replication

Signed-off-by: zhuyuchen.1 <zhuyuchen.1@bytedance.com>
2024-01-19 14:15:49 +08:00
MinerYang
b38de22054
bump golang 1.21.5 & fix golangci-lint error (#19722)
bump golang 1.21.5

update golangci-lint && fix revive error

fix white space lint

Signed-off-by: yminer <yminer@vmware.com>
2023-12-19 09:41:26 +08:00
Chlins Zhang
43ccd2f09f
perf: optimize the performance of accessory query (#19557)
1. Add db index for subject_artifact_id in table artifact_accessory.
2. Optimize the SQL of excluding the accessory for artifact count.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-11-22 02:11:11 +00:00
Maksym Trofimenko
996e57b511
Feature: GDPR compliant audit logs (#17396) 2023-11-20 21:31:22 +01:00
stonezdj(Daojun Zhang)
307a3121aa
Return empty result when no scanner configured (#19577)
fixes #19534

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-11-17 07:29:19 +00:00
Shengwen YU
e941f3272b
fix: sorting quota (#19538)
fix: sort Project Quotas

Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2023-11-13 14:37:41 +08:00
Wang Yan
5c02fd807e
add permission api (#19543)
The permission api targets to return the full set of permissons for robot to use.
And only system and project admin have the access

Signed-off-by: wang yan <wangyan@vmware.com>
2023-11-09 09:47:07 +08:00
stonezdj(Daojun Zhang)
da949bfc3f
Delete project member when delete project (#19523)
Signed-off-by: stonezdj <daojunz@vmware.com>
2023-11-08 20:51:21 +08:00
Daniel Jiang
3f7c605cf5
[WIP] Remove vendor folder from harbor code base (#19508)
* Remove vendor folder from harbor code base

Signed-off-by: Daniel Jiang <jiangd@vmware.com>

* debug ut failure

* debug failure

* debug failure

---------

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2023-11-05 14:37:25 +08:00
stonezdj(Daojun Zhang)
b337f51e7e
Replace comma in username to avoid casbin issue (#19505)
Check username when creating user by API
   Replace comma with underscore in username for OnboardUser
   Fixes #19356

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-11-02 03:50:50 +00:00
stonezdj(Daojun Zhang)
7b0beed934
Delete tag retention rule and tag immutable rule when deleting project (#19390)
fixes #18250

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-10-24 04:28:16 +00:00
stonezdj(Daojun Zhang)
d030ab216b
Use batch to list the job id in the job queue to avoid crash redis (#19444)
fixes: #19436

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-10-18 09:31:37 +00:00
stonezdj(Daojun Zhang)
663d0b81a7
Change fixed_version to package_version in query dangerous CVE sql (#19397)
Change fixed_version to package_version
  fixes #19391

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-10-11 07:57:19 +00:00
Wang Yan
3338f27189
support accessory in either order (#19375)
In certain cases, the OCI client may push the subject artifact and accessory in either order.
Therefore, it is necessary to handle situations where the client pushes the accessory ahead of the subject artifact.

Signed-off-by: wang yan <wangyan@vmware.com>
2023-09-23 02:42:58 +08:00
Wang Yan
ed370a496b
deprecate resource label (#19349)
There is no api is using the DAO, remove it from the source code.

Signed-off-by: wang yan <wangyan@vmware.com>
2023-09-14 01:25:52 +08:00
stonezdj(Daojun Zhang)
7f191630e4
Remove job status track information from redis after stop (#19227)
Remove job status track information from redis after stop the job in the queue

  After stop in the queue:
  Remove key in {harbor_job_service_namespace}:job_track:inprogress
  Remove {harbor_job_service_namespace}:job_stats:<job_id>
  fixes #19211

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-09-05 10:30:48 +08:00
Bin Liu
0e039b255c
exporter: add field alias for count(*) func (#18840)
Add an alias so that orm can parse the column name regardless of
the type and version of the database.

Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-30 09:05:44 +08:00
stonezdj(Daojun Zhang)
298ecbbe7d
Refine total artifact and scanned artifact (#19228)
Artifact include all accessory, child artifact
  fixes #19215

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-22 16:03:33 +08:00
Chlins Zhang
c7e25295fe
fix: support customize cache db for business (#19182)
Support to configure the customized redis db for cache layer and other
misc business for core, by default the behavior is same with
previous(stored in db 0).

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-08-18 11:04:16 +08:00
stonezdj(Daojun Zhang)
83ff2b277a
Wrong artifact scanned count (#19198)
fixes #19197

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-17 14:12:07 +08:00
Chlins Zhang
52e66155d4
log: change log level to reduce the noise logs (#19146)
1. Change some logs level to reduce the noise.
2. Wrap the go-redis.Nil error as ErrNotFound to avoid confusing

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-08-17 11:11:04 +08:00
stonezdj(Daojun Zhang)
d98699b5fc
Filter artifact without CVE from top 5 dangerous artifacts (#19187)
Fixes #19186

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-17 09:29:23 +08:00
Shengwen YU
88c6018950
fix: cron string validation (#19071)
fix: cron string validation (the 1st field of a cron string must be 0 when there are 6 fields)

Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2023-08-09 10:22:54 +08:00
Shengwen YU
90de9092ce
fix: add storage_limit check (#19095)
fix: add storage_limit check (add ValidateQuotaLimit as a general method to validate quota limit value)

Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2023-08-09 09:37:07 +08:00
stonezdj(Daojun Zhang)
0e92eaea18
Sort most dangerous vulnerabilities by score and severity level (#19103)
Check it is referred by a scan_report
  Add index to report_vulnerability_record to improve performance
  Fixes #19014 #19028

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-07 10:13:34 +08:00
stonezdj(Daojun Zhang)
c62ec7daac
Add artifact digest to query condition (#19102)
Fixes #19023

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-07 09:24:44 +08:00
stonezdj(Daojun Zhang)
3de778e9f1
Fix incorrect artifact and scanned artifact count issue (#19106)
* Fix incorrect artifact and scanned artifact count issue

  fixes #19009 #19020 #19013

Signed-off-by: stonezdj <daojunz@vmware.com>

* fix issue

Signed-off-by: stonezdj <daojunz@vmware.com>

---------

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-07 08:40:26 +08:00
Chlins Zhang
8c23bcdc54
fix: fix replication list projects with pure numberic name (#19090)
Quote the project name when listing projects in the replication, resolve
the issue of pure number name of project.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-08-04 09:29:51 +08:00
stonezdj(Daojun Zhang)
d92aca9d39
Fix wrong scanned artifact count when there are multiple report for an artifact (#18975)
Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-25 14:10:10 +08:00
Chlins Zhang
970bdab936
fix: correct the operator in the webhook payload (#18906)
Fix the incorrect or meaningless operator in the webhook payload.

Fixes: #18438

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-07-19 15:40:29 +08:00
stonezdj(Daojun Zhang)
d4aa9b13c4
Add vulnerability search API (#18924)
use q.Query to pass all query conditions

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-19 10:17:14 +08:00
Wang Yan
5cce621471
add notation support (#18909)
Accept and recognize the signature of notation client

Signed-off-by: wang yan <wangyan@vmware.com>
2023-07-13 11:18:34 +08:00
stonezdj(Daojun Zhang)
93e428d0d2
Add security hub summary API (#18872)
include WithCVE, WithArtifact option

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-12 19:18:08 +08:00
stonezdj(Daojun Zhang)
d84b1d07d2
Update table scan_report and extract cvss_v3_score from vendor attribute (#18854)
For better performance when query cve information, add summary information to scan_report
    Extract cve_score from vendor attribute in vulnerability_record
    SQL migrate script for the update

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-06-29 17:30:50 +08:00
Chlins Zhang
02a1c417d4
refactor: migrate the redis command keys to scan (#18825)
Refine the cache interface, migrate the Keys to Scan, change the redis
underlying keys command to scan.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-06-29 11:28:19 +08:00
rongfu.leng
8251fd2dec
【UT】add unit test for collector system info (#18717)
add unit test for system collector test

Signed-off-by: lengrongfu <1275177125@qq.com>
2023-06-08 10:06:09 +08:00
rongfu.leng
8fe578e7ab
feat: Optimize quota checking when pushing images (#17392)
Signed-off-by: lengrongfu <1275177125@qq.com>
2023-06-05 17:19:55 +08:00
cui fliter
e19ec9623b
Fix: fix function name in comments (#18726)
fix function name in comments

Signed-off-by: cui fliter <imcusg@gmail.com>
2023-05-31 10:26:19 +08:00
guangwu
b7b0e43a21
fix: import optimization (#18727)
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-05-30 13:24:08 +08:00
stonezdj(Daojun Zhang)
1b1af4a14c
Return error when proxy cache get too many request error(429) (#18728)
Add 429 too many request error in http error
  Fixes #18707

Signed-off-by: stonezdj <stonezdj@gmail.com>
2023-05-29 13:33:46 +08:00
Wang Yan
bf7c82b9a8
remove the notary from backend (#18668)
Since harbor deprecates notary since v2.9.0, this pull request targets to remove the code related with notary.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-05-18 18:47:42 +08:00
Viktor
f8e28dbede
Changed logic search projects in gitlab adapter (#18529)
* fix(gitlab): change logic search projects usage search_namespaces

Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>

* tests(gitlab): remove old data and actualization test

Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>

* refactor(gitlab): added debug log

Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>
(cherry picked from commit 732806221b1c1633682fe1adad4cbcd3a4687d0b)

* lint(gitlab): fix import order

Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>

---------

Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>
2023-05-15 13:59:13 +08:00
Chlins Zhang
362387f914
fix: improve the performance of list artifacts (#18610)
1. Change the query for listing tasks of scan which can use the db
   index.
2. Add the gin index for task.extra_attrs.report_uuids

Fixes: #18013

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-04-30 09:10:28 +08:00
MinerYang
ebac530b46
add goheader linter settings (#18503)
fix files for goheader linter

fix copyright 2018/2019

Signed-off-by: yminer <yminer@vmware.com>
2023-04-25 11:18:42 +08:00