Commit Graph

544 Commits

Author SHA1 Message Date
stonezdj(Daojun Zhang)
c4a58433cf
Update the log information in proxy cache (#17936)
fixes #17933

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-12-08 13:50:37 +08:00
stonezdj(Daojun Zhang)
402363d50b
Remove job parameters and schedule extra attribute (#17878)
Remove job parameters from job

  Remove extra attribute and cron type from schedule
  fixes #17866

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-28 17:45:26 +08:00
stonezdj(Daojun Zhang)
80999f28ee
Keep CheckInAt time empty when it is empty (#17877)
Fixes #17860

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-28 14:02:30 +08:00
Wang Yan
18a3373725
bump beego (#17801)
* bump beego

upgrade beego version from v1.10.12 to v2.0.5

1, beego v2 vserver/web refactor
2, beego v2 context refactor
3, beego v2 session refactor
4, beego v2 cache refactor
5, beego v2 orm refactor

Signed-off-by: MinerYang <yminer@vmware.com>
2022-11-24 18:07:42 +08:00
stonezdj(Daojun Zhang)
f8628258bc
Add Cron, CronType and Update Time for schedule (#17844)
Remove create time because update time is added

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-24 10:38:52 +08:00
Chlins Zhang
321a9abfb3
fix: skip to push system artifact for empty CSV file (#17816)
1. Skip to push system artifact to the distribution when the exported CSV file is empty.
2. Add status message for cve export execution.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-18 14:16:36 +08:00
MinerYang
62223bd36d
update golangci-lint for golang1.19 (#17817)
update golaci-lint for golang1.19

Signed-off-by: yminer <yminer@vmware.com>

update ci version to v1.50.1
2022-11-18 11:40:01 +08:00
stonezdj(Daojun Zhang)
57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
yanggang
0f4e2daf4c
Replaced io/ioutil with "os / io" package. (#17792)
Signed-off-by: yanggang <gang.yang@daocloud.io>

update typo

Signed-off-by: yminer <yminer@vmware.com>

Signed-off-by: yminer <yminer@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
2022-11-17 10:02:29 +08:00
stonezdj(Daojun Zhang)
ea27197a82
Sort group search result by most match order (#17708)
fixes #17535

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-15 20:38:44 +08:00
stonezdj(Daojun Zhang)
a47e175056
Remove email configuration (#17712)
Email related settings are deprecated
   Remove email ping API
   Fixes #17683

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-14 15:03:49 +08:00
Maksym Trofimenko
6b46b8bf26
fix permission denied error for non existing projects (#17586) (#17619)
Signed-off-by: Maksym Trofimenko <maksym@container-registry.com>
2022-11-14 12:01:20 +08:00
Wang Yan
4fb0a19efb
add cli randomly for empty update (#17740)
give an random cli secret when client gives empty update.

Signed-off-by: Wang Yan <wangyan@vmware.com>

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-11-03 21:04:29 +08:00
stonezdj(Daojun Zhang)
39ca918ffe
Add jobservice monitoring api list pool, worker and stop running task (#17658)
Add REST API to list job pool, worker, stop running task

  Add jobservice handler to retrieve configuration
  Add RBAC for jobservice monitoring dashboard
  Add REST API to list pool, worker and stop running task

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-11-03 10:47:40 +08:00
Chlins Zhang
bf4cfe9e1e
fix: add human friendly message when export cve job failure (#17726)
Add human friendly when export CVE in the condition of empty CSV file,
because this file will be stored as system artifact and pushed to
distribution, but it will leads to error when push empty blob to S3
storage driver.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 15:04:07 +08:00
Chlins Zhang
c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
Eng Zer Jun
cd0fa06a32
test: use T.Setenv to set env vars in tests (#17670)
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-25 10:56:26 +08:00
stonezdj(Daojun Zhang)
b6bdb4b57f
Add default project when pulling from dockerhub without project name (#17537)
Check project is default project format in dockerhub
 Redirect blob/manifest request to default url
 Fixes #12719

Signed-off-by: stonezdj <stonezdj@gmail.com>

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-10-17 17:37:44 +08:00
Chenyu Zhang
31a3c6d18c
fix: validate the existence of project for export cve api (#17437)
Validate the existence of the project for export CVE API handler, return
not found error if the project is not exist.

Closes: #17430

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-22 15:44:57 +08:00
Chenyu Zhang
1e13999fff
fix: validate export cve request params (#17341)
1. Validate export cve request params in the API handler
2. Trim space for request in the scan export job

Closes: #17326

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-08 11:07:05 +08:00
Chenyu Zhang
70a95a9696
fix: reduce the high db cpu usage for tag retention (#17296)
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: #14708

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-04 10:08:02 +08:00
Wang Yan
914d9a5526
fix copy artifact accessory quota issue (#17293)
It needs to request quota for accessory on copying artifact.

Signed-off-by: wang yan <wangyan@vmware.com>
2022-08-03 20:02:20 +08:00
Thomas Maroschik
0559ce7e28
fix: oidc token is not respected in /service/token endpoint (#16442)
Signed-off-by: Thomas Maroschik <tmaroschik@dfau.de>
2022-07-30 13:01:51 +02:00
Eike David Lenz
0fe5eed55d
fix cve allowlist projectmeta validation (#15800)
Signed-off-by: Eike David Lenz <e.lenz@mittwald.de>
2022-07-30 16:59:18 +08:00
Chenyu Zhang
bff4e13087
fix: export cve adds resource check and project validation (#17265)
1. Add resource permission check for API handler
2. Validate export cve params project
3. Optimize friendly human message when execution status is error

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-29 19:01:46 +08:00
Wang Yan
02c41573d6
fix update robot regression (#17248)
Only check the project id for the project level robot

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-27 20:13:46 +08:00
MinerYang
f7ebf20d2e
Disable Nydus middleware for v2.6 (#17233)
disable nydus middleware for v2.6

Signed-off-by: yminer <yminer@vmware.com>

remove nyduus module import in main.go

Signed-off-by: yminer <yminer@vmware.com>
2022-07-27 08:41:33 +08:00
prahaladdarkin
d53af792ad
Fixes for various bugs/issues logged as a part of the test day. (#17232)
Closes:
* CVE Data Export API IDOR issue
* https://github.com/goharbor/harbor/issues/17199
* https://github.com/goharbor/harbor/issues/17193
* https://github.com/goharbor/harbor/issues/17188
* https://github.com/goharbor/harbor/issues/17184

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-26 16:50:54 +08:00
stonezdj(Daojun Zhang)
712419778a
Return bad request if audit log retention hour > 240000 hour (#17217)
Cap the retention hour to 240000 hour
  Fixes #17198

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-22 15:14:10 +08:00
Loong Dai
09371b48e8
lint: sort imports (#17131)
* lint: add goimports

Signed-off-by: Loong Dai <loong.dai@intel.com>
2022-07-20 11:33:08 +08:00
prahaladdarkin
3f383e3ffd
Improvements to Vulnerability Data Export functionality. (#17161)
Closes:
* https://github.com/goharbor/harbor/issues/17152
* https://github.com/goharbor/harbor/issues/17153
Better error handling in case of task executions not found in the system

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-14 15:08:25 +08:00
prahaladdarkin
130452111b
Vulnerability scan data export functionality (#15998)
Vulnerability Scan Data (CVE) Export Functionality
Proposal - goharbor/community#174
Closes - https://github.com/goharbor/harbor/issues/17150
Changes:
* CVE Data export to CSV with filtering support.
* Implement CSV data export job for creating CSVs
* APIs to trigger CSV export job executions

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-07-11 16:35:04 +08:00
MinerYang
efd9632e96
add nydus middleware (#17126)
Signed-off-by: yminer <yminer@vmware.com>

remove comments

Signed-off-by: yminer <yminer@vmware.com>

update ut manifest

Signed-off-by: yminer <yminer@vmware.com>

modify comment manifest

Signed-off-by: yminer <yminer@vmware.com>

updtae ut testcase

Signed-off-by: yminer <yminer@vmware.com>

fixwhitespace lint

Signed-off-by: yminer <yminer@vmware.com>

update isNydus judgement && define annotation var

Signed-off-by: yminer <yminer@vmware.com>

whitespace lint

Signed-off-by: yminer <yminer@vmware.com>
2022-07-08 09:56:10 +08:00
Wang Yan
bd8d66c68d
resolve robot authgen password format issue (#17134)
In some cases, the robot automatically generates passwords that do not meet confidentiality requirements.
The fix adds retry for auto generating passwords, and the timeout is 1 minute.

The requirement: the secret must longer than 8 chars with at least 1 uppercase letter, 1 lowercase letter and 1 number

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-08 02:05:32 +08:00
Chenyu Zhang
1a1ce634cc
Fix the process of cache layer (#17010)
fix: fix cache layer issues (#16995,#16997,#16996,#17038)

1. Load config and initialize cache layer in jobservice(for GC)
2. Cache artifact by digest the key should contains repository name
3. Repository cache cleanup error when update
4. Skip save cache when request ctx in transaction

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 16:11:53 +08:00
Shengwen YU
315d8a8993
fix: update code in compliance with golangci-lint revive (#17087)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-07-01 14:52:47 +08:00
Wang Yan
b4ef1dbfd5
Unify retention schedule/task retrieve and update (#17083)
Add not-found handling for non exist items in retention operation.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-01 01:58:14 +08:00
Chenyu Zhang
1c3eb6974c
fix: update preheat api handler and DAO (#17079)
1. fix preheat dao Get method
2. update preheat tasks and getLog api

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-28 19:01:08 +08:00
stonezdj(Daojun Zhang)
aa63569de2
Fix scan log mismatch issue (#17085)
Add checks in label

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 18:12:38 +08:00
stonezdj(Daojun Zhang)
c639257ba7
Support stop purge audit log job (#17033)
fixes #17006

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-28 11:15:11 +08:00
Wang Yan
ea2ed27a7d
add url raw query check middleware (#17070)
The middleware can give a uniform url validation and raised error early.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-06-24 17:57:46 +08:00
Chenyu Zhang
8e99c83b19
fix: fix the update of retention policy (#17064)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-23 19:12:48 +08:00
Chenyu Zhang
c343232081
fix: revise the process of policy update (#17021)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-22 18:30:06 +08:00
stonezdj(Daojun Zhang)
e6eb7821d0
Unify the process of job schedule/task retrieve and update (#17012)
Unify the process of jobservice execution/task retrieve and update

   Change regular expression in robot account

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-22 18:22:33 +08:00
prahaladdarkin
7024442f25
Project name validation to prevent system artifact reserved project names (#17029)
Include system artifact size in system storage space calculation.

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-21 12:51:29 +08:00
MinerYang
aab320591f
golangci-lint fix ineffassign (#17027)
Signed-off-by: yminer <yminer@vmware.com>

delete src/lib/redis/helper.go

Signed-off-by: yminer <yminer@vmware.com>
2022-06-21 12:01:59 +08:00
Wenkai Yin(尹文开)
6c515b04d4
Support stop GC execution (#17004)
Support stop GC execution

Fixes 16902

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2022-06-17 20:03:24 +08:00
Shengwen YU
e9fca3de45
fix: refactor code for golangci-lint whitespace (#17005)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-06-16 17:42:46 +08:00
Shengwen YU
c2a9f5de64
fix: update code for golangci-lint gosimple (#16974)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-06-14 13:44:09 +08:00
Chenyu Zhang
e08ad05659
fix(swagger): append scan report version 1.1 to swagger docs (#16965)
fix(swagger): append scan report version 1.1 to swagger docs default value

The default value of swagger header X-Accept-Vulnerabilities should contains
scan report version 1.0 and 1.1.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-10 16:38:43 +08:00