Commit Graph

790 Commits

Author SHA1 Message Date
stonezdj(Daojun Zhang)
c4a58433cf
Update the log information in proxy cache (#17936)
fixes #17933

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-12-08 13:50:37 +08:00
Iceber Gu
a90181e312
fix the setting of data slice length for tablewriter (#17918)
Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>
2022-12-07 13:19:08 +08:00
little_an
8cc16de180
fix dragonfly response when p2p preheat (#15916)
* fix dragonfly response when p2p preheat

Signed-off-by: anyiwei <anyiwei@didichuxing.com>
2022-12-06 16:12:21 +08:00
stonezdj(Daojun Zhang)
64c03e8679
Update the execution status after tasks stopped (#17875)
Fixes #17862

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-12-01 21:17:35 +08:00
Jianwei Guo
cb11540a14
add parameters for PostgreSQL (#16641)
Signed-off-by: sayaoailun <guojianwei007@126.com>
2022-11-30 19:08:08 +08:00
stonezdj(Daojun Zhang)
402363d50b
Remove job parameters and schedule extra attribute (#17878)
Remove job parameters from job

  Remove extra attribute and cron type from schedule
  fixes #17866

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-28 17:45:26 +08:00
stonezdj(Daojun Zhang)
619747fc68
Stop Pending Job cannot update status (#17842)
Convert the redis range result into struct and extract job id from it
  Add more log when get redis config fails

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-24 20:29:50 +08:00
Wang Yan
18a3373725
bump beego (#17801)
* bump beego

upgrade beego version from v1.10.12 to v2.0.5

1, beego v2 vserver/web refactor
2, beego v2 context refactor
3, beego v2 session refactor
4, beego v2 cache refactor
5, beego v2 orm refactor

Signed-off-by: MinerYang <yminer@vmware.com>
2022-11-24 18:07:42 +08:00
Chlins Zhang
321a9abfb3
fix: skip to push system artifact for empty CSV file (#17816)
1. Skip to push system artifact to the distribution when the exported CSV file is empty.
2. Add status message for cve export execution.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-18 14:16:36 +08:00
MinerYang
62223bd36d
update golangci-lint for golang1.19 (#17817)
update golaci-lint for golang1.19

Signed-off-by: yminer <yminer@vmware.com>

update ci version to v1.50.1
2022-11-18 11:40:01 +08:00
stonezdj(Daojun Zhang)
57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
yanggang
0f4e2daf4c
Replaced io/ioutil with "os / io" package. (#17792)
Signed-off-by: yanggang <gang.yang@daocloud.io>

update typo

Signed-off-by: yminer <yminer@vmware.com>

Signed-off-by: yminer <yminer@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2022-11-17 10:02:29 +08:00
stonezdj(Daojun Zhang)
a47e175056
Remove email configuration (#17712)
Email related settings are deprecated
   Remove email ping API
   Fixes #17683

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-14 15:03:49 +08:00
stonezdj(Daojun Zhang)
cc69b1e951
Add OIDC group filter (#17736)
Filter out the OIDC group which doesn't match the regular expression
  Fixes #17130

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-11 09:55:11 +08:00
Chlins Zhang
77df4d92fd
feat: support customize registry http client timeout (#17768)
Support override the registry http client timeout from env `REGISTRY_HTTP_CLIENT_TIMEOUT` (in minutes), the timeout is 30 minutes by default.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-09 17:20:47 +08:00
stonezdj(Daojun Zhang)
39ca918ffe
Add jobservice monitoring api list pool, worker and stop running task (#17658)
Add REST API to list job pool, worker, stop running task

  Add jobservice handler to retrieve configuration
  Add RBAC for jobservice monitoring dashboard
  Add REST API to list pool, worker and stop running task

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-03 10:47:40 +08:00
Chlins Zhang
c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
Eng Zer Jun
cd0fa06a32
test: use T.Setenv to set env vars in tests (#17670)
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-25 10:56:26 +08:00
Vincent Ni
9c0275deb6
Fix ecr authorization token issue (#17645)
Signed-off-by: Vincent Ni <kangqini@amazon.com>
2022-10-18 11:09:58 +08:00
Vincent Ni
805a36e7f0
Fix Replication to Cross-account AWS ECR (#17583)
Replication to Cross-account AWS ECR
2022-09-30 09:07:47 +02:00
stonezdj(Daojun Zhang)
b6c978c7f7
Change ldap.Search to ldap.SearchWithPaging (#17534)
fixes #16390

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-09-19 11:04:54 +08:00
Vincent Ni
a3d96000f5
Fix cross account issue (#17533)
Signed-off-by: Vincent Ni <kangqini@amazon.com>
2022-09-15 14:25:41 +08:00
Chenyu Zhang
5d7a549620
upgrade: bump up mockery to v2.14.0 (#17415)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-22 14:21:13 +08:00
Chenyu Zhang
24d60eb119
enhancement: add error log for create task (#17397)
Signed-off-by: chlins <chenyuzh@vmware.com>

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-16 11:35:05 +08:00
Chenyu Zhang
49d73fa57d
fix: export cve for image manifest list (#17333)
1. Fix export cve for image manifest list
2. Remove row_id column in csv file
3. Update cve execution swagger API description

Closes: #17331,#17330,#17335,#17334

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-05 20:07:02 +08:00
prahaladdarkin
c776ea8bfe
Export CVE data Score column empty values (#17321)
Closes: https://github.com/goharbor/harbor/issues/17189

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-08-05 10:03:42 +08:00
Chenyu Zhang
70a95a9696
fix: reduce the high db cpu usage for tag retention (#17296)
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: #14708

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-04 10:08:02 +08:00
Adam
bf741ad381
feature: Use RegisteredClaims instead of deprecated staruct StandardClaims (#16206)
Signed-off-by: wujw39640 <wujw39640@hundsun.com>
2022-08-01 10:23:47 +08:00
Chenyu Zhang
bff4e13087
fix: export cve adds resource check and project validation (#17265)
1. Add resource permission check for API handler
2. Validate export cve params project
3. Optimize friendly human message when execution status is error

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-29 19:01:46 +08:00
Chenyu Zhang
19e73174e2
refactor: refact codebase of cve export process and filter (#17254)
1. Refact the scan export FilterProcessor interface
2. Optmize the sql template for export cve report
3. Update the process of cve export job

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-28 16:05:12 +08:00
prahaladdarkin
d53af792ad
Fixes for various bugs/issues logged as a part of the test day. (#17232)
Closes:
* CVE Data Export API IDOR issue
* https://github.com/goharbor/harbor/issues/17199
* https://github.com/goharbor/harbor/issues/17193
* https://github.com/goharbor/harbor/issues/17188
* https://github.com/goharbor/harbor/issues/17184

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-26 16:50:54 +08:00
Wang Yan
02eae9dede
Filter out accessory from list artifact results (#17231)
Fixed #17145
1, Filter out the accessory from the artifact list.
2, Disable the display func of the accessory interface, currently this will not impact any kind of accessory, like signature and nydus. If we'd like to introduce it, it needs to resolve the pagiation issue of artifact list.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-26 00:38:05 +08:00
stonezdj(Daojun Zhang)
8e876d847c
Add resource type in forward audit log (#17216)
Add resource type when forward audit log

  fixes #17201

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-22 09:54:07 +08:00
Loong Dai
09371b48e8
lint: sort imports (#17131)
* lint: add goimports

Signed-off-by: Loong Dai <loong.dai@intel.com>
2022-07-20 11:33:08 +08:00
prahaladdarkin
f3edb03b49
Fix to CVE Data Export functionality for images pushed by docker push (#17182)
Closes:
* https://github.com/goharbor/harbor/issues/17167

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-18 09:22:15 +08:00
prahaladdarkin
3f383e3ffd
Improvements to Vulnerability Data Export functionality. (#17161)
Closes:
* https://github.com/goharbor/harbor/issues/17152
* https://github.com/goharbor/harbor/issues/17153
Better error handling in case of task executions not found in the system

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-14 15:08:25 +08:00
prahaladdarkin
130452111b
Vulnerability scan data export functionality (#15998)
Vulnerability Scan Data (CVE) Export Functionality
Proposal - goharbor/community#174
Closes - https://github.com/goharbor/harbor/issues/17150
Changes:
* CVE Data export to CSV with filtering support.
* Implement CSV data export job for creating CSVs
* APIs to trigger CSV export job executions

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-11 16:35:04 +08:00
MinerYang
efd9632e96
add nydus middleware (#17126)
Signed-off-by: yminer <yminer@vmware.com>

remove comments

Signed-off-by: yminer <yminer@vmware.com>

update ut manifest

Signed-off-by: yminer <yminer@vmware.com>

modify comment manifest

Signed-off-by: yminer <yminer@vmware.com>

updtae ut testcase

Signed-off-by: yminer <yminer@vmware.com>

fixwhitespace lint

Signed-off-by: yminer <yminer@vmware.com>

update isNydus judgement && define annotation var

Signed-off-by: yminer <yminer@vmware.com>

whitespace lint

Signed-off-by: yminer <yminer@vmware.com>
2022-07-08 09:56:10 +08:00
Chenyu Zhang
1a1ce634cc
Fix the process of cache layer (#17010)
fix: fix cache layer issues (#16995,#16997,#16996,#17038)

1. Load config and initialize cache layer in jobservice(for GC)
2. Cache artifact by digest the key should contains repository name
3. Repository cache cleanup error when update
4. Skip save cache when request ctx in transaction

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 16:11:53 +08:00
stonezdj(Daojun Zhang)
dcccb44db8
Add options to the user.Count method (#16285)
Fixes #16269, exclude the admin account by default
  Add excludeDefaultAdmin method -- exclude default admin by option
  Update authModeCanBeModified method -- the user count should be 0 without admin

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-06 13:57:31 +08:00
stonezdj(Daojun Zhang)
c9d6ba1469
Revert "Skip to refresh execution status when unchanged (#17023)" (#17111)
This reverts commit 88b8a7ac2c.

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-04 18:10:59 +08:00
Shengwen YU
315d8a8993
fix: update code in compliance with golangci-lint revive (#17087)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-07-01 14:52:47 +08:00
Wang Yan
b4ef1dbfd5
Unify retention schedule/task retrieve and update (#17083)
Add not-found handling for non exist items in retention operation.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-01 01:58:14 +08:00
Wenkai Yin(尹文开)
ab74e853ee
Enhance the read-only API to avoid deleting operations during the job running (#17055)
Enhance the read-only API to avoid deleting operations during the job running

Fixes #16901

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2022-06-29 08:00:17 +08:00
Chenyu Zhang
1c3eb6974c
fix: update preheat api handler and DAO (#17079)
1. fix preheat dao Get method
2. update preheat tasks and getLog api

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-28 19:01:08 +08:00
Shengwen YU
44bfb992f4
fix: update code for golangci-lint varcheck (#17045)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-06-21 16:56:36 +08:00
prahaladdarkin
7024442f25
Project name validation to prevent system artifact reserved project names (#17029)
Include system artifact size in system storage space calculation.

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-21 12:51:29 +08:00
MinerYang
aab320591f
golangci-lint fix ineffassign (#17027)
Signed-off-by: yminer <yminer@vmware.com>

delete src/lib/redis/helper.go

Signed-off-by: yminer <yminer@vmware.com>
2022-06-21 12:01:59 +08:00
stonezdj(Daojun Zhang)
88b8a7ac2c
Skip to refresh execution status when unchanged (#17023)
fixes #16640

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-20 13:56:19 +08:00
Maksym Trofimenko
9a3cb4a041
GDPR compliant deletion of Users (#16859)
fixes #16697

Signed-off-by: Maksym Trofimenko <maksym@container-registry.com>
2022-06-16 14:28:15 +02:00