Refactor the logic for updating the status of execution when receiving
the hook from jobservice, avoid the optimistic lock due to the multiple
tasks update one execution by refreshing the status asynchronously. But
still retain the old way by specifying the flag from ENV.
Fixes: #17584
Signed-off-by: chlins <chenyuzh@vmware.com>
Wrap orm tx when the scheduler try to create the task because submit job
maybe failure depends on the jobservice.
Fixes: #18452
Signed-off-by: chlins <chenyuzh@vmware.com>
1. Change the the value of CloudEvents id from requestid to uuid
2. Add additional requestid to trace to event
Fixes: #18448
Signed-off-by: chlins <chenyuzh@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
Change the JobId param type from int to string, use int will bring some
problems for encode/decode type mismatch which generate the invalid
repository name.
Fixes: #18380
Signed-off-by: chlins <chenyuzh@vmware.com>
refactor: refactor the old goroutine execution sweep to global execution sweep job
1. Delete the old goroutine execution sweeper when create execution.(in the case of high concurrency can cause goroutine backlogs, affect the performance of core)
2. Introduce the new way to sweep executions, a global scheduled job will take the work.
Signed-off-by: chlins <chenyuzh@vmware.com>
As for the distribution spec 1.1, it supports client to push an manifest with subject field. By leverging this fidle, harbor could build up the linkage between the subject artifact and it's accessories.
Signed-off-by: wang yan <wangyan@vmware.com>
refactor: refact the notification job API and life process
1. Introduce new APIs for webhook jobs management.
2. Refact legacy APIs for backforward compatible.
3. Migrate the webhook jobs process to unified execution/task framework.
Closes: #18210
Signed-off-by: chlins <chenyuzh@vmware.com>
Fixes#18121
Refactor job name with VendorType prefix, make sure job queue name and vendor type in execution and task are identical
Signed-off-by: stonezdj <daojunz@vmware.com>
Remove the project filter in the scan data export job as they have been
validated by API handler, fix the oidc or ldap group users cannot export
cve.
Fixes: #18112
Signed-off-by: chlins <chenyuzh@vmware.com>
1. Change the Export CVE temporary file directory to /tmp.
2. Remove the scan data export volume in Dockerfile and docker-compose
yaml.
Fixes: #18067
Signed-off-by: chlins <chenyuzh@vmware.com>
Remove job parameters from job
Remove extra attribute and cron type from schedule
fixes#17866
Signed-off-by: stonezdj <daojunz@vmware.com>
Signed-off-by: stonezdj <daojunz@vmware.com>
Convert the redis range result into struct and extract job id from it
Add more log when get redis config fails
Signed-off-by: stonezdj <daojunz@vmware.com>
Signed-off-by: stonezdj <daojunz@vmware.com>
1. Skip to push system artifact to the distribution when the exported CSV file is empty.
2. Add status message for cve export execution.
Signed-off-by: chlins <chenyuzh@vmware.com>
Add queue manager and redis client
Update scheduler to add count and list
Signed-off-by: stonezdj <daojunz@vmware.com>
Signed-off-by: stonezdj <daojunz@vmware.com>
Filter out the OIDC group which doesn't match the regular expression
Fixes#17130
Signed-off-by: stonezdj <stonezdj@gmail.com>
Signed-off-by: stonezdj <stonezdj@gmail.com>
Support override the registry http client timeout from env `REGISTRY_HTTP_CLIENT_TIMEOUT` (in minutes), the timeout is 30 minutes by default.
Signed-off-by: chlins <chenyuzh@vmware.com>
Add REST API to list job pool, worker, stop running task
Add jobservice handler to retrieve configuration
Add RBAC for jobservice monitoring dashboard
Add REST API to list pool, worker and stop running task
Signed-off-by: stonezdj <stonezdj@gmail.com>
Signed-off-by: stonezdj <stonezdj@gmail.com>
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler
Signed-off-by: chlins <chenyuzh@vmware.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.
Closes: #14708
Signed-off-by: chlins <chenyuzh@vmware.com>
1. Add resource permission check for API handler
2. Validate export cve params project
3. Optimize friendly human message when execution status is error
Signed-off-by: chlins <chenyuzh@vmware.com>
1. Refact the scan export FilterProcessor interface
2. Optmize the sql template for export cve report
3. Update the process of cve export job
Signed-off-by: chlins <chenyuzh@vmware.com>
Fixed#17145
1, Filter out the accessory from the artifact list.
2, Disable the display func of the accessory interface, currently this will not impact any kind of accessory, like signature and nydus. If we'd like to introduce it, it needs to resolve the pagiation issue of artifact list.
Signed-off-by: Wang Yan <wangyan@vmware.com>
fix: fix cache layer issues (#16995,#16997,#16996,#17038)
1. Load config and initialize cache layer in jobservice(for GC)
2. Cache artifact by digest the key should contains repository name
3. Repository cache cleanup error when update
4. Skip save cache when request ctx in transaction
Signed-off-by: chlins <chenyuzh@vmware.com>
Fixes#16269, exclude the admin account by default
Add excludeDefaultAdmin method -- exclude default admin by option
Update authModeCanBeModified method -- the user count should be 0 without admin
Signed-off-by: stonezdj <stonezdj@gmail.com>
Fix azurecr use ACR token failed to list tags, the root cause is the
scope action of acr token is 'metadata_read' not 'pull' when list v2
tags API.
Signed-off-by: chlins <chenyuzh@vmware.com>
The tag/lable filter only works on the subject manifest, and if the subject manifest is mathed, all the accessories are marked as matched.
Signed-off-by: Wang Yan <wangyan@vmware.com>
Implement cache layer for resource manifest, it will read manifest
from cache instead of proxying to distribution if enabled.
Signed-off-by: chlins <chenyuzh@vmware.com>
