- update Job interface to introdcue MaxCurrency method for declaring the max currency of the specified job
- change the downstream jobs to implement the new interface method
- GC and sample jobs are set to 1
- other jobs are set to 0 that means unlimited
- add max currency optiot when doing job registration
- resolve issue #11586
- probably resolve issue #11281
- resolve issue #11570
Signed-off-by: Steven Zou <szou@vmware.com>
1. Remove `common/quota` package.
2. Remove functions about quota in `common/dao` package.
3. Move `Quota` and `QuotaUsage` models from `common/models` to
`pkg/quota/dao`.
4. Add `Count` and `List` methods to `quota.Controller`.
5. Use `quota.Controller` to implement quota APIs.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
fixes#11533
GC jobs will use the filter results to call registry API to delete manifest.
In the current imple, the filter function in some case does not return the deleted artifact as it's using digest as the filter condition.
Like: If one artifact is deleted, but there is another project/repo has a image with same digest with the deleted one, filter func will
not mark the deleted artifact as candidate. It results in, GC job does not call API to remove the manifest.
To fix it, update the filter to use both digest and repository name to filter candidate.
Signed-off-by: wang yan <wangyan@vmware.com>
* Update tags related APIs
1. Remove API for listing tags of repository
2. Add API for listing tags of artifact
3. Support filter artifact by tag name
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* [OCI] modify artifact tag name check
1. switch api get tag list
2. modify artifact tag name check
Signed-off-by: Yogi_Wang <yawang@vmware.com>
Co-authored-by: Yogi_Wang <yawang@vmware.com>
Fixes#11241
1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Prevent the pull action when scan report status is not successfuly.
2. Bypass the checking when no vulnerabilities not found.
3. Improve the returned message when prevented the pull action.
Closes#11202
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* fix(scan): fix ScanStatus when merge NativeReportSummary
1. Running and success status is high priority when merge ScanStatus of
NativeReportSummary, otherwise chose the bigger status.
2. Merge scan logs of referenced artifacts when get the scan logs of
image index.
Closes#11265
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* fix(portal): fix the annotation for the scan completed percent in scan overview
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Add timeout when transter artifacts
2. Check 404 error when unschedule the policy
3. Add line to mark the job failure in job log
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Fixes#11016
1. src/pkg/q->src/internal/q
2. src/internal->src/lib (internal is a reserved package name of golang)
3. src/api->src/controller
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* replication webhook support
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
* replication webhook support with ut fixed
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
The query string is encoded by UI, and we have to unescape the "=" in "q=tag=nil",
otherwise, the query doesn't work, and returns 400
Signed-off-by: wang yan <wangyan@vmware.com>
1. Skip vulnerability prevention checking when artifact is not
scannable.
2. Skip vulnerability prevention checking when artifact is image index
and its type is `IMAGE` or `CNAB`.
3. Skip vulnerability prevention checking when the artifact is pulling
by the scanner.
4. Change `hasCapability` from blacklist to whitelist.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Filter the pulling manifest request from replication service so that the audit log will not record the pulling action
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, enable audit logs for notifications
2, move the handler and meatadata into API
3, use the notification middleware to send out notification
Signed-off-by: wang yan <wangyan@vmware.com>
1. Use quota controller to implement the internal quota APIs.
2. The internal quota APIs can exceed the quota limitations.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
the notification is for send out the event after DB transaction complete.
It's safe to send hook as this middleware is after transaction in the response path.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Introduce `Evaluator` interface which do the permission checking.
2. `admin`, `lazy`, `rbac`, `namespace` and `evaluartor` set are implemented the
`Evaluator` interface.
3. Move project rbac implemention from `project` to `rbac` pkg to reduce
the name conflict with project instance of model.
4. Do permission checking in security context by `Evaluator`.
5. Cache the regexp in rbac evaluator for casbin.
6. Cache evaluator in namespace evaluator to improve performance.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
"-1" means the robot account is a permanent account, no expiration time set.
The ExpiresAt claim is optional, so if it's not set, it will still be considered a valid claim
Signed-off-by: wang yan <wangyan@vmware.com>
1, enable dao test for artifact trash
2, set default flush trash table to false
3, hanlder empty parameter in API call
4, add registry auth info into jobservice container
Signed-off-by: wang yan <wangyan@vmware.com>
This commit defines the API query string format and provides the builders to build query string to query model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, add API entry for get audit logs
2. add audit log manager to hanlder CRUD
Use the new format of audit log to cover differernt resource, artifact/tag/repostory/project
Signed-off-by: wang yan <wangyan@vmware.com>
1. Add Checker to check the scannable status of the artifact.
2. Only set vulnerabilities addition link when the artifact scanable in the
project.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
use the tag controller to handle CRUD of tags, especially the delete scenario, it could validate
the immutable and signature. And move the code of tag handling from artifact controller to tag controller
Signed-off-by: wang yan <wangyan@vmware.com>
As we store the repository name in the artifact table, we can use it direclty in the code to reduce the database query
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Assemble scan overview to artifact when scanner enabled in the
project of the artifact.
2. Set addition link for vulnerabilities to artifact when scanner
enabled in the project of the artifact.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Add middleware to record the accepted blob size for stream blob
upload.
2. Add middleware to create blob and associate it with project after blob upload
complete.
3. Add middleware to sync blobs, create blob for manifest and associate blobs
with the manifest after put manifest.
4. Add middleware to associate blob with project after mount blob.
5. Cleanup associations for the project when artifact deleted.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* refactor(scan,scanner): move scan and scanner controllers to api pkg
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* feat(scan-all-job): move artifacts query from job to notification
Move artifact query from scan all job to its notification handler to
ensure that the components in pkg will not call controllers in api.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* Populate signature status in artifact API
This Commit add signature status into response of list artifact API.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. As "List" method of artifact DAO doesn't return the artifacts that referenced by other and without tag, so we introduce a new method "GetByDigest" to check the existence of artifact
2. The "Www-Authenticate" header is needed to be returned when the request is unauthorized. This is required in the OCI distribution spec and is needed when pushing CNAB
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit add supporting for adding/removing label to/from artifacts and populates labels when listing artifacts
Signed-off-by: Wenkai Yin <yinw@vmware.com>
As we only provide the API to get artifact information via project name, repository name and digest, the digest of child artifact must be returned
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1. Upgrade beego to v1.12.0
2. Add RequestID middleware to all HTTP requests.
3. Add Orm middleware to v2 and v2.0 APIs.
4. Remove OrmFilter from all HTTP requests.
5. Fix some test cases which cause panic in API controllers.
6. Enable XSRF for test cases of CommonController.
7. Imporve ReadOnly middleware.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1, Leverage go v1.13 new error feature
2, Define genernal error OCI format, so that /v2 API could return a OCI compatible error
Signed-off-by: wang yan <wangyan@vmware.com>
1. Define the controller/manager interface for artifact and tag
2. Provide a null implementation for artifact manager
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commits does some basic init work for supporting OCI:
1. Create the artifact and tag model
2. Create database tables
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit removes the attribute to control case-sensitivity from
authproxy setting.
The result in token review status will be used as the single source of
truth, regardless the case of the letters in group names and user names.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit add an attribute to configurations, whose value is the
certificate of authproxy server. When this attribute is set Harbor will
pin to this cert when connecting authproxy.
This value will also be part of the response of systemInfo API.
This commit will be cherrypicked to 1.10 and 1.9 branch.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit fixes#9869
It has some refactor to make sure the group is populated when user is
authenticated via tokenreview workflow.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
BREAKING CHANGE: the value negligible of severity in project metadata will change to none in the responses of project APIs
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Fix name conflict when install internal clair adapter.
2. Remove all internal adapters when reinstall harbor without --with-clair flag
Signed-off-by: He Weiwei <hweiwei@vmware.com>
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job
fix#9705fix#9722fix#9670
Signed-off-by: Steven Zou <szou@vmware.com>
1. Remove ping action when initialize clair adapter installed by harbor.
2. Remvoe the `IsDefault` property when initialize clair adapter that
make it switch to auto detecting.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
- add a common error pkg to support error with code and AsError check
- replace some errors in scan with coded errors
- fix#9668
Signed-off-by: Steven Zou <szou@vmware.com>
1, replace the UIVERSION file with ldflags, which is generarted by make to inject into the UI core.
2, inject additional ldflags for harbor compiler
Signed-off-by: wang yan <wangyan@vmware.com>
- add new endpoint for getting scanner candidates of specified project
- adjust the permission granting functions
- fix#9608
Signed-off-by: Steven Zou <szou@vmware.com>
1. Upgrade clair adapter to v1.0.0.
2. Make the clair adapter which installed by harbor immutable and using internal registry address.
3. Add support to build clair adapter image from binary.
4. Switch to ScannerPull action when make authorization for the scan request.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
- do changes to the sql schema
- add `UseInternalAddr` and `Immutable` properties to scanner registration
- support multiple authentication type
- basic
- bearer token
Signed-off-by: Steven Zou <szou@vmware.com>
1, the commit is for internal robot to bypass policy check, like vul and signature checking.
2, add a bool attribute into registry token, decode it in the harbor core and add the status into request context.
3, add a bool attribut for robot API controller, but API will not use it.y
Signed-off-by: wang yan <wangyan@vmware.com>
- add LCM control to the robot account generated for scanning
- improve the scan webhook
- remove reprots when related artifact is deleted
- update report manager/scan controller and other components to support above cases
- add artifact manager/comtroller to list artifacts
Signed-off-by: Steven Zou <szou@vmware.com>
To contain various vulnerabilities in the CVE whitelist, this commit
removes the validation.
Fixes#9242
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema
Signed-off-by: Steven Zou <szou@vmware.com>
