Commit Graph

356 Commits

Author SHA1 Message Date
Chlins Zhang
90db04e92d
refactor: refact the webhook API and life process (#18255)
refactor: refact the notification job API and life process

1. Introduce new APIs for webhook jobs management.
2. Refact legacy APIs for backforward compatible.
3. Migrate the webhook jobs process to unified execution/task framework.

Closes: #18210

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-03 10:17:47 +08:00
stonezdj(Daojun Zhang)
58054c0ce8
Change queue name for purge audit log and filter the old one (#18182)
Fixes #18121
 Refactor job name with VendorType prefix, make sure job queue name and vendor type in execution and task are identical

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-02-20 16:03:22 +08:00
stonezdj(Daojun Zhang)
d03f0dcf2d
Skip to update pull time and pull count for scanner robot account (#17807)
Add prefix for scanner robot account
   Fixes #14638

Signed-off-by: stonezdj <daojunz@vmware.com>

# Conflicts:
#	api/v2.0/swagger.yaml
#	src/common/const.go
#	src/lib/config/metadata/metadatalist.go
2023-02-20 15:09:21 +08:00
Chlins Zhang
da1637e1d3
fix: resolve the oidc or ldap group user cannot export cve (#18219)
Remove the project filter in the scan data export job as they have been
validated by API handler, fix the oidc or ldap group users cannot export
cve.

Fixes: #18112

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-02-20 14:22:27 +08:00
Wang Yan
738fde7d3b
remove chartmuseum backend (#18191)
Harbor deprecates chartmuseum as of v2.8.0

Epic: https://github.com/goharbor/harbor/issues/17958

Discussion: https://github.com/goharbor/harbor/discussions/15057

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-02-16 18:11:05 +08:00
Chlins Zhang
47137605c9
fix: update the replication API handler (#18197)
1. Check execution before stop replication execution.
2. Check execution before list replication tasks.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-02-14 13:11:27 +08:00
Maksym Trofimenko
68fb01813e
Allow the Identity Provider to be the primary auth method in Harbor. (#17627)
Allow the Identity Provider to be the primary auth method in Harbor

Signed-off-by: Maksym Trofimenko <maksym@container-registry.com>
2023-02-08 18:11:08 +01:00
Iceber Gu
7dc452ccab
controller/robot: use global regexp vars (#17915)
Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>

Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>
2022-12-06 15:59:37 +08:00
Chlins Zhang
287f334327
fix: align some configuration api fields validation with frontend (#17903)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-12-05 15:52:41 +08:00
stonezdj(Daojun Zhang)
64c03e8679
Update the execution status after tasks stopped (#17875)
Fixes #17862

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-12-01 21:17:35 +08:00
stonezdj(Daojun Zhang)
619747fc68
Stop Pending Job cannot update status (#17842)
Convert the redis range result into struct and extract job id from it
  Add more log when get redis config fails

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-24 20:29:50 +08:00
Wang Yan
18a3373725
bump beego (#17801)
* bump beego

upgrade beego version from v1.10.12 to v2.0.5

1, beego v2 vserver/web refactor
2, beego v2 context refactor
3, beego v2 session refactor
4, beego v2 cache refactor
5, beego v2 orm refactor

Signed-off-by: MinerYang <yminer@vmware.com>
2022-11-24 18:07:42 +08:00
stonezdj(Daojun Zhang)
4b7d45513e
Fix issue related to redhat registry proxy cache (#17789)
fixes #16495

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-18 17:17:39 +08:00
Chlins Zhang
321a9abfb3
fix: skip to push system artifact for empty CSV file (#17816)
1. Skip to push system artifact to the distribution when the exported CSV file is empty.
2. Add status message for cve export execution.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-18 14:16:36 +08:00
MinerYang
62223bd36d
update golangci-lint for golang1.19 (#17817)
update golaci-lint for golang1.19

Signed-off-by: yminer <yminer@vmware.com>

update ci version to v1.50.1
2022-11-18 11:40:01 +08:00
stonezdj(Daojun Zhang)
57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
yanggang
0f4e2daf4c
Replaced io/ioutil with "os / io" package. (#17792)
Signed-off-by: yanggang <gang.yang@daocloud.io>

update typo

Signed-off-by: yminer <yminer@vmware.com>

Signed-off-by: yminer <yminer@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2022-11-17 10:02:29 +08:00
Chlins Zhang
7f00a77d99
fix: pre check notification enable for artifact webhook (#17779)
Check the notification enable before in the artifact webhook handler,
avoid additional db query cost for notification_policy.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-15 09:54:36 +08:00
stonezdj(Daojun Zhang)
a47e175056
Remove email configuration (#17712)
Email related settings are deprecated
   Remove email ping API
   Fixes #17683

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-14 15:03:49 +08:00
stonezdj(Daojun Zhang)
39ca918ffe
Add jobservice monitoring api list pool, worker and stop running task (#17658)
Add REST API to list job pool, worker, stop running task

  Add jobservice handler to retrieve configuration
  Add RBAC for jobservice monitoring dashboard
  Add REST API to list pool, worker and stop running task

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-03 10:47:40 +08:00
Chlins Zhang
bf4cfe9e1e
fix: add human friendly message when export cve job failure (#17726)
Add human friendly when export CVE in the condition of empty CSV file,
because this file will be stored as system artifact and pushed to
distribution, but it will leads to error when push empty blob to S3
storage driver.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 15:04:07 +08:00
Chlins Zhang
c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
stonezdj(Daojun Zhang)
3050e856d9
Add check when adding project member (#17701)
Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-10-28 15:06:27 +08:00
Eng Zer Jun
cd0fa06a32
test: use T.Setenv to set env vars in tests (#17670)
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-25 10:56:26 +08:00
stonezdj(Daojun Zhang)
b08b5fe617
Add check on the audit log forward and skip audit log database (#17575)
fixes #17524
Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-09-21 15:24:57 +08:00
Chenyu Zhang
5d7a549620
upgrade: bump up mockery to v2.14.0 (#17415)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-22 14:21:13 +08:00
MinerYang
daf0874ed5
project userid not found log level into debug (#17343)
Signed-off-by: yminer <yminer@vmware.com>
2022-08-08 14:20:13 +08:00
Chenyu Zhang
49999ab1c0
fix: replication webhook lost when src namespace different with dest (#17312)
Fix the replication webhook notification lost when the rule is
pull-based and src namespace different with dest.

Closes: #17298

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-04 15:10:19 +08:00
Chenyu Zhang
70a95a9696
fix: reduce the high db cpu usage for tag retention (#17296)
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: #14708

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-04 10:08:02 +08:00
Wang Yan
914d9a5526
fix copy artifact accessory quota issue (#17293)
It needs to request quota for accessory on copying artifact.

Signed-off-by: wang yan <wangyan@vmware.com>
2022-08-03 20:02:20 +08:00
MinerYang
722d1a19ea
update image extra attributes author in labels (#17291)
update image extra attributes author

Signed-off-by: yminer <yminer@vmware.com>

adding spec comment

Signed-off-by: yminer <yminer@vmware.com>
2022-08-03 11:50:54 +08:00
Chenyu Zhang
bff4e13087
fix: export cve adds resource check and project validation (#17265)
1. Add resource permission check for API handler
2. Validate export cve params project
3. Optimize friendly human message when execution status is error

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-29 19:01:46 +08:00
Wang Yan
02eae9dede
Filter out accessory from list artifact results (#17231)
Fixed #17145
1, Filter out the accessory from the artifact list.
2, Disable the display func of the accessory interface, currently this will not impact any kind of accessory, like signature and nydus. If we'd like to introduce it, it needs to resolve the pagiation issue of artifact list.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-26 00:38:05 +08:00
Abirdcfly
2fece8c9ea
delete duplicate import and use ST1019 in golangci-lint (#17211)
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-07-25 14:12:00 +08:00
Loong Dai
09371b48e8
lint: sort imports (#17131)
* lint: add goimports

Signed-off-by: Loong Dai <loong.dai@intel.com>
2022-07-20 11:33:08 +08:00
prahaladdarkin
3f383e3ffd
Improvements to Vulnerability Data Export functionality. (#17161)
Closes:
* https://github.com/goharbor/harbor/issues/17152
* https://github.com/goharbor/harbor/issues/17153
Better error handling in case of task executions not found in the system

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-14 15:08:25 +08:00
prahaladdarkin
130452111b
Vulnerability scan data export functionality (#15998)
Vulnerability Scan Data (CVE) Export Functionality
Proposal - goharbor/community#174
Closes - https://github.com/goharbor/harbor/issues/17150
Changes:
* CVE Data export to CSV with filtering support.
* Implement CSV data export job for creating CSVs
* APIs to trigger CSV export job executions

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-11 16:35:04 +08:00
Roooocky
7e4b26b220
Add new feature for supporting WebAssembly artifact (#16931)
support wasm

Signed-off-by: ln23415 <ln23415@hotmail.com>
2022-07-08 23:08:28 +08:00
MinerYang
efd9632e96
add nydus middleware (#17126)
Signed-off-by: yminer <yminer@vmware.com>

remove comments

Signed-off-by: yminer <yminer@vmware.com>

update ut manifest

Signed-off-by: yminer <yminer@vmware.com>

modify comment manifest

Signed-off-by: yminer <yminer@vmware.com>

updtae ut testcase

Signed-off-by: yminer <yminer@vmware.com>

fixwhitespace lint

Signed-off-by: yminer <yminer@vmware.com>

update isNydus judgement && define annotation var

Signed-off-by: yminer <yminer@vmware.com>

whitespace lint

Signed-off-by: yminer <yminer@vmware.com>
2022-07-08 09:56:10 +08:00
Wang Yan
bd8d66c68d
resolve robot authgen password format issue (#17134)
In some cases, the robot automatically generates passwords that do not meet confidentiality requirements.
The fix adds retry for auto generating passwords, and the timeout is 1 minute.

The requirement: the secret must longer than 8 chars with at least 1 uppercase letter, 1 lowercase letter and 1 number

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-08 02:05:32 +08:00
stonezdj(Daojun Zhang)
dcccb44db8
Add options to the user.Count method (#16285)
Fixes #16269, exclude the admin account by default
  Add excludeDefaultAdmin method -- exclude default admin by option
  Update authModeCanBeModified method -- the user count should be 0 without admin

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-06 13:57:31 +08:00
Wang Yan
6917021b41
resolve copy failure for artifact with multiple accessories (#17123)
If the artifact has more than one signaure, it will currently copy the last one. After the fix, all accessories can be copied to the target project.

Signed-off-by: wang yan <wangyan@vmware.com>
2022-07-05 20:10:22 +08:00
Chenyu Zhang
927f9f5af3
fix: attach labels for replication event (#17108)
Attach labels for replication event, list labels by artifact ID so for
event-based replication rule filter by label can work as expect.

Closes: #17014

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-05 19:57:13 +08:00
Shengwen YU
315d8a8993
fix: update code in compliance with golangci-lint revive (#17087)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-07-01 14:52:47 +08:00
Wang Yan
b4ef1dbfd5
Unify retention schedule/task retrieve and update (#17083)
Add not-found handling for non exist items in retention operation.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-01 01:58:14 +08:00
stonezdj(Daojun Zhang)
aa63569de2
Fix scan log mismatch issue (#17085)
Add checks in label

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 18:12:38 +08:00
stonezdj(Daojun Zhang)
c639257ba7
Support stop purge audit log job (#17033)
fixes #17006

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 11:15:11 +08:00
MinerYang
aab320591f
golangci-lint fix ineffassign (#17027)
Signed-off-by: yminer <yminer@vmware.com>

delete src/lib/redis/helper.go

Signed-off-by: yminer <yminer@vmware.com>
2022-06-21 12:01:59 +08:00
Maksym Trofimenko
9a3cb4a041
GDPR compliant deletion of Users (#16859)
fixes #16697

Signed-off-by: Maksym Trofimenko <maksym@container-registry.com>
2022-06-16 14:28:15 +02:00
Shengwen YU
e9fca3de45
fix: refactor code for golangci-lint whitespace (#17005)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-06-16 17:42:46 +08:00