Add pkg/member/dao
Add pkg/member/models
Add pkg/member/manager
Add controller/member
Remove the old project member API
Signed-off-by: stonezdj <stonezdj@gmail.com>
This commit moves the legacy apis related to users to new model.
Some funcs under common/dao are left b/c they are used by other module,
which should also be shifted to leverage managers.
We'll handle them separately.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Improve the performance of replication by introducing a new API to check whether the blob can be mounted directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context
Signed-off-by: stonezdj <stonezdj@gmail.com>
This commit adds the attribute "http_authproxy_admin_usernames", which
is string that contains usernames separated by comma, when a user logs
in and the username in the tokenreview status matches the setting of
this attribute, the user will have administrator permission.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* Refactor registry API
Refactor registry API
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* Fix bugs of replications
1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit updates the API POST /api/v2.0/system/oidc/ping to new
programming model, in which the code will be generated by go-swagger.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* update blob list query
Deprecate blob list parameters, and use the query for instead.
Signed-off-by: wang yan <wangyan@vmware.com>
* update per review comments
Signed-off-by: Wang Yan <wangyan@vmware.com>
1. Remove the default execution sweeper count for execution vendor.
2. Set the execution sweeper count for gc, preheat, replication,
retention to 50.
3. Disable sweep for the executions of the scan job.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Don't ignore the NotFoundErr when handling the status hook of tasks to avoid the status out of sync
Fixes#14016
Signed-off-by: Wenkai Yin <yinw@vmware.com>
The server to handle token-review may have a limitation for the size of
the header. When the token is huge the token review may fail.
This commit remove the necessary header to harden the flow.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Remove the duplicate CVE records in the report/summary for the image
index.
2. Add scanner field in the scan overview for the API.
Closes#13913
Signed-off-by: He Weiwei <hweiwei@vmware.com>
feat: Store vulnerability report from scanner into a relational format
Convert vulnerability report JSON obtained from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145
Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
Fixes#13740
Update ManifestExist to return Descriptor instead of digest
For docker 20.10 or containerd, it HEAD the manifest before pull, then
it GET the manifest with digest, add logic to handle this scenario and
correlate the tag between the digest in proxy cache
Signed-off-by: stonezdj <stonezdj@gmail.com>
This reverts commit e6f80259
This reverts commit 918fe125
Signed-off-by: stonezdj <stonezdj@gmail.com>
Revert "Add content type and length in header"
This reverts commit ca379111
Signed-off-by: stonezdj <stonezdj@gmail.com>
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit adds admin_groups into the configuration of http_auth
settings, it's a string in the form of "group1, group2". If the token
review result shows the user is in one of the groups in the setting he
will have the administrator role in Harbor.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.
Signed-off-by: wang yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* updates on robot accounts
1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account
Signed-off-by: wang yan <wangyan@vmware.com>
* update code per review comments
1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.
Signed-off-by: Wang Yan <wangyan@vmware.com>
author Wang Yan <wangyan@vmware.com> 1605849192 +0800
committer Wang Yan <wangyan@vmware.com> 1606361046 +0800
update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Use `project.Controller` instead of `promgr.ProjectManager` in security
implementations because we will remove `promgr` package later.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit refactors the approach to encode a token in handler of /service/token,
by reusing pkg/token to avoid inconsistency.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
There is requirement that show the cron type(daily, weekly, etc.) on the UI, this commit adds the support for storing the cron type in the schedule model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Add a new method "StopAndWait" which stops the execution and wait until the execution stopped or get an error
Signed-off-by: Wenkai Yin <yinw@vmware.com>
We know the user id when query projects by member, so use the user id
as entity_id directly in project_member, no need to join harbor_user
table.
Closes#12968
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1, remove the pkg dependency on v2.0/server/models
2, remove the controller dependency on v2.0/server/models
Signed-off-by: wang yan <wangyan@vmware.com>
fixes#12849
1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.
2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.
Signed-off-by: wang yan <wangyan@vmware.com>
Refresh the status of execution for every status changing of task to support filtering executions by status directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
fixes#12720
The GC job doesn't remove the manifest of scheme1.MediaTypeSignedManifest as it's recognized by GC job.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule
Signed-off-by: Wenkai Yin <yinw@vmware.com>
- fix npe issue in create/update policy
- fix issue of missing schedule job id in the preheat policy
Signed-off-by: Steven Zou <szou@vmware.com>
- increase the client timeout
1, update typo in the update blob status sql, the typo will not impact the sql result.
2, correct blob status in the middleware & GC job log.
Signed-off-by: wang yan <wangyan@vmware.com>
- add job stop check points in preheat job
- add missing digest property for the preheat request sent to the provider
Signed-off-by: Steven Zou <szou@vmware.com>
- fix invalid data type of vulnerability filter param
- add more debug logs
- add more logs in the preheat job
- fix issue of getting empty list when doing querying artifacts
Signed-off-by: Steven Zou <szou@vmware.com>
* add debugging env for GC time window
For debugging, the tester/users wants to run GC to delete the removed artifact immediately instead of waitting for two hours, add the env(GC_BLOB_TIME_WINDOW) to meet this.
Signed-off-by: wang yan <wangyan@vmware.com>
1, The update blob status method should udpate the blob version of the blob object as well, otherwise the GC job cannot handle the blob status transform(none - delete - deleting - deletefailed)
as the method is using version equals as the query condition.
2, For the deleting blob which marked for more than 2 hours, it should be set to delete failed in head blob & put manifest request
Signed-off-by: wang yan <wangyan@vmware.com>
This commit provides the secret manager for proxy cache.
The secret is used for pushing blobs to local when it's proxied from
remote registry.
Each secret can be used only once and has a relatively short expiration
time.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
two phases:
1, mark, select the gc candidates bases on the DB and mark them as status delete.
2, sweep, select the candidate and mark it as status deleting and remove it from backend and database.
Signed-off-by: wang yan <wangyan@vmware.com>
- use real provider instance manager
- move mock insatnce manager to testing/pkg
- modify kraken deriver implementation to remove digest fetcher
- update related UT cases
Signed-off-by: Steven Zou <szou@vmware.com>
Add support list blob with update time.
As introduces the time window in GC, it wants to list the blobs less than specific time.
Signed-off-by: wang yan <wangyan@vmware.com>
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
- define policy enforcer interface
- implement the default enforcer
- registrer P2P preheat job to JS
- add the missing mock manager&controller in the src/testing pkg
- Add UT cases for enforcer
- fix#12285
- left one TODO: query provider instance by instance Manager
Signed-off-by: Steven Zou <szou@vmware.com>
* add get GC candidate
select non referenced blobs from table blob and exclude the ones in the time windows.
Signed-off-by: wang yan <wangyan@vmware.com>
- add new selector based on vulnerability severity criteria
- add new selector based on signature(signed) criteria
- do change to the select factory method definition
- do changes to selector.Candidate model
- add preheat policy filter interface and default implementation
- add UT cases to cover new code
Signed-off-by: Steven Zou <szou@vmware.com>
misspelling
- define preheat driver interface
- implement dragonfly driver
- implememt kraken driver
- add related UT cases with testify framework
- fix#10870#10871
- some code are picked up from the original P2P feat branch
Signed-off-by: Steven Zou <szou@vmware.com>
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.
Signed-off-by: wang yan <wangyan@vmware.com>
Fixes#9704
As we do want to unify error handling, so just decreprates pkg errors, use lib/errors instead for Harbor internal used errors model.
1, The lib/errors can cover all of funcs of pkg/errors, and also it has code attribute to define the http return value.
2, lib/errors can give a OCI standard error format, like {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized"}]}
If you'd like to use pkg/errors, use lib/errors instead. If it cannot meet your request, enhance it.
Signed-off-by: wang yan <wangyan@vmware.com>
- update Job interface to introdcue MaxCurrency method for declaring the max currency of the specified job
- change the downstream jobs to implement the new interface method
- GC and sample jobs are set to 1
- other jobs are set to 0 that means unlimited
- add max currency optiot when doing job registration
- resolve issue #11586
- probably resolve issue #11281
- resolve issue #11570
Signed-off-by: Steven Zou <szou@vmware.com>
1. Remove `common/quota` package.
2. Remove functions about quota in `common/dao` package.
3. Move `Quota` and `QuotaUsage` models from `common/models` to
`pkg/quota/dao`.
4. Add `Count` and `List` methods to `quota.Controller`.
5. Use `quota.Controller` to implement quota APIs.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
fixes#11533
GC jobs will use the filter results to call registry API to delete manifest.
In the current imple, the filter function in some case does not return the deleted artifact as it's using digest as the filter condition.
Like: If one artifact is deleted, but there is another project/repo has a image with same digest with the deleted one, filter func will
not mark the deleted artifact as candidate. It results in, GC job does not call API to remove the manifest.
To fix it, update the filter to use both digest and repository name to filter candidate.
Signed-off-by: wang yan <wangyan@vmware.com>
* Update tags related APIs
1. Remove API for listing tags of repository
2. Add API for listing tags of artifact
3. Support filter artifact by tag name
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* [OCI] modify artifact tag name check
1. switch api get tag list
2. modify artifact tag name check
Signed-off-by: Yogi_Wang <yawang@vmware.com>
Co-authored-by: Yogi_Wang <yawang@vmware.com>
Fixes#11241
1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Prevent the pull action when scan report status is not successfuly.
2. Bypass the checking when no vulnerabilities not found.
3. Improve the returned message when prevented the pull action.
Closes#11202
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* fix(scan): fix ScanStatus when merge NativeReportSummary
1. Running and success status is high priority when merge ScanStatus of
NativeReportSummary, otherwise chose the bigger status.
2. Merge scan logs of referenced artifacts when get the scan logs of
image index.
Closes#11265
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* fix(portal): fix the annotation for the scan completed percent in scan overview
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Add timeout when transter artifacts
2. Check 404 error when unschedule the policy
3. Add line to mark the job failure in job log
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Fixes#11016
1. src/pkg/q->src/internal/q
2. src/internal->src/lib (internal is a reserved package name of golang)
3. src/api->src/controller
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* replication webhook support
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
* replication webhook support with ut fixed
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
