feat: Store vulnerability report from scanner into a relational format
Convert vulnerability report JSON obtained from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145
Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
Fixes#13740
Update ManifestExist to return Descriptor instead of digest
For docker 20.10 or containerd, it HEAD the manifest before pull, then
it GET the manifest with digest, add logic to handle this scenario and
correlate the tag between the digest in proxy cache
Signed-off-by: stonezdj <stonezdj@gmail.com>
This reverts commit e6f80259
This reverts commit 918fe125
Signed-off-by: stonezdj <stonezdj@gmail.com>
Revert "Add content type and length in header"
This reverts commit ca379111
Signed-off-by: stonezdj <stonezdj@gmail.com>
Fixes#13566: Quota of dockerhub is still used in v2.1.1 after the image is cached
Cache manifest list in redis cache.
Trade off between efficiency and data integrity, it might cause the proxy cache return the full content of a manifest list instead of the actual manifest list saved in the Harbor storage, which is a part of the manifest list. but this change doesn't break any /v2/ API, just caches full manifest list.
Signed-off-by: stonezdj <stonezdj@gmail.com>
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.
Signed-off-by: wang yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* updates on robot accounts
1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account
Signed-off-by: wang yan <wangyan@vmware.com>
* update code per review comments
1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.
Signed-off-by: Wang Yan <wangyan@vmware.com>
default is true and doesn't break any existing api, and when to set it to false, the api only return the basic project infor
without meta, CVE settings and etc of the project.
Signed-off-by: wang yan <wangyan@vmware.com>
author Wang Yan <wangyan@vmware.com> 1605849192 +0800
committer Wang Yan <wangyan@vmware.com> 1606361046 +0800
update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
1. Clean the dirty data in quota/quota_usage.
2. Ignore the fail when getting the reference of quota.
Closes#13387
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* refactor: remove core/promgr pkg
Remove `core/promgr` package and use `controller/project` instead of it.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Use `project.Controller` instead of `promgr.ProjectManager` in security
implementations because we will remove `promgr` package later.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Compare the local digest and the remote digest when pull by tag
Use HEAD request (ManifestExist) instead of GET request (GetManifest) to avoid been throttled
For manifest list, it can avoid GET request because cached manifest list maybe different with the original manifest list
Make RemoteInterface public
Fixes#13112
Signed-off-by: stonezdj <stonezdj@gmail.com>
There is requirement that show the cron type(daily, weekly, etc.) on the UI, this commit adds the support for storing the cron type in the schedule model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Add a new method "StopAndWait" which stops the execution and wait until the execution stopped or get an error
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1. Abstract the "config" property(which contains labels) of config layer into the extra attributes for images
2. Try to get the author information from the "maintainer" label
fixes 12066
fixes 12734
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, remove the pkg dependency on v2.0/server/models
2, remove the controller dependency on v2.0/server/models
Signed-off-by: wang yan <wangyan@vmware.com>
- use content trust settings of project configurations
- use vulnerability severity settings of project configurations
Signed-off-by: Steven Zou <szou@vmware.com>
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule
Signed-off-by: Wenkai Yin <yinw@vmware.com>
- fix npe issue in create/update policy
- fix issue of missing schedule job id in the preheat policy
Signed-off-by: Steven Zou <szou@vmware.com>
- increase the client timeout
- add job stop check points in preheat job
- add missing digest property for the preheat request sent to the provider
Signed-off-by: Steven Zou <szou@vmware.com>
- fix issue of missing handling error in the preheat event handler
- change preheat artifact logic to reduce health check times
- publish pushed events only for the tagged artifacts
Signed-off-by: Steven Zou <szou@vmware.com>
- fix invalid data type of vulnerability filter param
- add more debug logs
- add more logs in the preheat job
- fix issue of getting empty list when doing querying artifacts
Signed-off-by: Steven Zou <szou@vmware.com>
1, The update blob status method should udpate the blob version of the blob object as well, otherwise the GC job cannot handle the blob status transform(none - delete - deleting - deletefailed)
as the method is using version equals as the query condition.
2, For the deleting blob which marked for more than 2 hours, it should be set to delete failed in head blob & put manifest request
Signed-off-by: wang yan <wangyan@vmware.com>
Read security settings from the project configurations and override the preheat policy settings if necessary.
Check the project security settings and override the related settings in the policy if necessary.
NOTES: if the security settings (relevant with signature and vulnerability) are set at the project configuration,
they will have the highest priority and override the related settings of the preheat policy.
e.g (use signature as an example, similar case to vulnerability severity part):
if policy.signature = false and project.config.signature = true; then policy.signature = true
if policy.signature = true and project.config.signature = true; then policy.signature = true
if policy.signature = true and project.config.signature = false; then policy.signature = true
if policy.signature = false and project.config.signature = false; then policy.signature = false
Signed-off-by: Steven Zou <szou@vmware.com>
Signed-off-by: Steven Zou <szou@vmware.com>
- use real provider instance manager
- move mock insatnce manager to testing/pkg
- modify kraken deriver implementation to remove digest fetcher
- update related UT cases
Signed-off-by: Steven Zou <szou@vmware.com>
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
- define policy enforcer interface
- implement the default enforcer
- registrer P2P preheat job to JS
- add the missing mock manager&controller in the src/testing pkg
- Add UT cases for enforcer
- fix#12285
- left one TODO: query provider instance by instance Manager
Signed-off-by: Steven Zou <szou@vmware.com>
- add new selector based on vulnerability severity criteria
- add new selector based on signature(signed) criteria
- do change to the select factory method definition
- do changes to selector.Candidate model
- add preheat policy filter interface and default implementation
- add UT cases to cover new code
Signed-off-by: Steven Zou <szou@vmware.com>
misspelling
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.
Signed-off-by: wang yan <wangyan@vmware.com>
In the delete repository scenario, the repository has already been removed from the DB, the repository cannot be retrived.
Just let the creation time as empty.
Signed-off-by: wang yan <wangyan@vmware.com>
Fixes#9704
As we do want to unify error handling, so just decreprates pkg errors, use lib/errors instead for Harbor internal used errors model.
1, The lib/errors can cover all of funcs of pkg/errors, and also it has code attribute to define the http return value.
2, lib/errors can give a OCI standard error format, like {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized"}]}
If you'd like to use pkg/errors, use lib/errors instead. If it cannot meet your request, enhance it.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Remove `common/quota` package.
2. Remove functions about quota in `common/dao` package.
3. Move `Quota` and `QuotaUsage` models from `common/models` to
`pkg/quota/dao`.
4. Add `Count` and `List` methods to `quota.Controller`.
5. Use `quota.Controller` to implement quota APIs.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Ignore limitation when refresh quota for project.
2. Return 403 when quota errors occurred.
3. Add test for Refresh method of quota controller.
Closes#11512
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Copy artifact will not return 409 anymore.
2. Make sure the tags of source artifact exist in the target artifact
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Fixes#11241
1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.
Signed-off-by: wang yan <wangyan@vmware.com>
* fix(scan): fix ScanStatus when merge NativeReportSummary
1. Running and success status is high priority when merge ScanStatus of
NativeReportSummary, otherwise chose the bigger status.
2. Merge scan logs of referenced artifacts when get the scan logs of
image index.
Closes#11265
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* fix(portal): fix the annotation for the scan completed percent in scan overview
Signed-off-by: He Weiwei <hweiwei@vmware.com>
For pull a public resource, there is no need to login, give the access name to anonymous in the audit logs
Signed-off-by: wang yan <wangyan@vmware.com>
The function GetRegistrationByProject should not return err when Ping
return err. The return value 'registration' has 'Health' field which
shows the scanner health status.
Resolves: #11051
See also: #9788, #9807
Signed-off-by: qinshaoxuan <qinshaoxuan@baidu.com>
Fixes#11016
1. src/pkg/q->src/internal/q
2. src/internal->src/lib (internal is a reserved package name of golang)
3. src/api->src/controller
Signed-off-by: Wenkai Yin <yinw@vmware.com>
