This commit moves the legacy apis related to users to new model.
Some funcs under common/dao are left b/c they are used by other module,
which should also be shifted to leverage managers.
We'll handle them separately.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Improve the performance of replication by introducing a new API to check whether the blob can be mounted directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context
Signed-off-by: stonezdj <stonezdj@gmail.com>
This commit adds the attribute "http_authproxy_admin_usernames", which
is string that contains usernames separated by comma, when a user logs
in and the username in the tokenreview status matches the setting of
this attribute, the user will have administrator permission.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* Refactor registry API
Refactor registry API
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* Fix bugs of replications
1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit updates the API POST /api/v2.0/system/oidc/ping to new
programming model, in which the code will be generated by go-swagger.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* update blob list query
Deprecate blob list parameters, and use the query for instead.
Signed-off-by: wang yan <wangyan@vmware.com>
* update per review comments
Signed-off-by: Wang Yan <wangyan@vmware.com>
1. Add operation id in ctx in baseapi before prepare
2. add operation id for registry proxy request
3. use url for other request
Signed-off-by: DQ <dengq@vmware.com>
Don't ignore the NotFoundErr when handling the status hook of tasks to avoid the status out of sync
Fixes#14016
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Correct the completed count in the metrics of the scan all to be the sum
of success, error and stopped count.
Closes#14001
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1, introduce & define the system resources.
2, replace the IsSysAdmin judge method.
3, give the robot the system access capability.
Signed-off-by: Wang Yan <wangyan@vmware.com>
Add X-Accept-Vulnerabilities header to the list/get artifact and get
artifact vulnerability addition APIs, and these APIs will traverse the
mime types in this header and return the first report and summary found
from the mime type.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
feat: Store vulnerability report from scanner into a relational format
Convert vulnerability report JSON obtained from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145
Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
Add the trigger to the metrics of the scan all job so that the customer
can know who trigger the latest scan all job.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit add a handler to handle the request to
"/c/authproxy/redirect". Harbor is configured to authenticate against
an authproxy, if a request with query string `?token=xxxx`
is sent to this URI, the handler will do tokenreview according to the
setting of authproxy and simulate a `login` workflow based on the result
of token review.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Fixes#13740
Update ManifestExist to return Descriptor instead of digest
For docker 20.10 or containerd, it HEAD the manifest before pull, then
it GET the manifest with digest, add logic to handle this scenario and
correlate the tag between the digest in proxy cache
Signed-off-by: stonezdj <stonezdj@gmail.com>
* fix robot account update issue
enable the update method to support both v1 & v2 robot update
Signed-off-by: Wang Yan <wangyan@vmware.com>
* resolve review comments
Signed-off-by: Wang Yan <wangyan@vmware.com>
This reverts commit e6f80259
This reverts commit 918fe125
Signed-off-by: stonezdj <stonezdj@gmail.com>
Revert "Add content type and length in header"
This reverts commit ca379111
Signed-off-by: stonezdj <stonezdj@gmail.com>
Fixes#13566: Quota of dockerhub is still used in v2.1.1 after the image is cached
Cache manifest list in redis cache.
Trade off between efficiency and data integrity, it might cause the proxy cache return the full content of a manifest list instead of the actual manifest list saved in the Harbor storage, which is a part of the manifest list. but this change doesn't break any /v2/ API, just caches full manifest list.
Signed-off-by: stonezdj <stonezdj@gmail.com>
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit adds admin_groups into the configuration of http_auth
settings, it's a string in the form of "group1, group2". If the token
review result shows the user is in one of the groups in the setting he
will have the administrator role in Harbor.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.
Signed-off-by: wang yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* updates on robot accounts
1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account
Signed-off-by: wang yan <wangyan@vmware.com>
* update code per review comments
1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.
Signed-off-by: Wang Yan <wangyan@vmware.com>
This commit cover the cases when the port is set in one of the Host of
request or the core URL, to make sure the comparison works as expected
when the default port (80, 443) is added in only one of them.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
default is true and doesn't break any existing api, and when to set it to false, the api only return the basic project infor
without meta, CVE settings and etc of the project.
Signed-off-by: wang yan <wangyan@vmware.com>
author Wang Yan <wangyan@vmware.com> 1605849192 +0800
committer Wang Yan <wangyan@vmware.com> 1606361046 +0800
update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Remove nop close body in the BeforeRequest helper function.
Middleware must make the request body re-readable itself when it wants
read the body in the middleware.
Closes#13556
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* refactor: remove core/promgr pkg
Remove `core/promgr` package and use `controller/project` instead of it.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Add configs in prepare
2. Add models and config items in Core
3. Encapdulate getting metric in commom package
4. Add a middleware for global request to collect 3 metrics
Signed-off-by: DQ <dengq@vmware.com>
The ping endpoint will be blocked when DB conns reach the max open conns
of the sql.DB which will make ping request timeout,
so skip the middlewares which will require DB conn.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Add oidc_admin_group to configuration, and make sure a token with the
group name in group claim has the admin authority.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Use `project.Controller` instead of `promgr.ProjectManager` in security
implementations because we will remove `promgr` package later.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit refactors the approach to encode a token in handler of /service/token,
by reusing pkg/token to avoid inconsistency.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Compare the local digest and the remote digest when pull by tag
Use HEAD request (ManifestExist) instead of GET request (GetManifest) to avoid been throttled
For manifest list, it can avoid GET request because cached manifest list maybe different with the original manifest list
Make RemoteInterface public
Fixes#13112
Signed-off-by: stonezdj <stonezdj@gmail.com>
This reverts commit 6fc0c9d75a.
Because this erases the AdminRoleInAuth attribute in user model as it is
not stored in DB and it will break the admin group of LDAP.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
We know the user id when query projects by member, so use the user id
as entity_id directly in project_member, no need to join harbor_user
table.
Closes#12968
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit makes a change so that the user id will be stored in sessoin
after user login instead of user model to avoid data inconsistency when
user model changes.
Fixes#12934
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit ensures that when CLI is pulling a tag, the content trust middleware check the data in notary to ensure the particular tag is signed, not only the digest.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, remove the pkg dependency on v2.0/server/models
2, remove the controller dependency on v2.0/server/models
Signed-off-by: wang yan <wangyan@vmware.com>
When the remote repo is offline or network issue or credential issue, fall back to local repo.
Fixes#12853
Signed-off-by: stonezdj <stonezdj@gmail.com>
The current approach will prevent the effectiveness of `Cache-Control`
header and gorilla's library add `Vary:Cookie` header in all responses.
We will set the token in a header of response so the response can be
cached when needed.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Refresh the status of execution for every status changing of task to support filtering executions by status directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Return 403 when trying to push artifacts into the proxy cache project to avoid the retrying in the docker client
fixes#12731
Signed-off-by: Wenkai Yin <yinw@vmware.com>
fixes#11522
use the format: date-time as the format of audit op_time, then it could be rendered by FF and Chrome.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit makes OIDC CLI secret filter allow more URLs so that the
OIDC CLI secret can be used for replication
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, update typo in the update blob status sql, the typo will not impact the sql result.
2, correct blob status in the middleware & GC job log.
Signed-off-by: wang yan <wangyan@vmware.com>
* add debugging env for GC time window
For debugging, the tester/users wants to run GC to delete the removed artifact immediately instead of waitting for two hours, add the env(GC_BLOB_TIME_WINDOW) to meet this.
Signed-off-by: wang yan <wangyan@vmware.com>
1, The update blob status method should udpate the blob version of the blob object as well, otherwise the GC job cannot handle the blob status transform(none - delete - deleting - deletefailed)
as the method is using version equals as the query condition.
2, For the deleting blob which marked for more than 2 hours, it should be set to delete failed in head blob & put manifest request
Signed-off-by: wang yan <wangyan@vmware.com>
When to call,
~~~ REQUEST ~~~
GET /v2/conformance/testrepo/manifests/.INVALID_MANIFEST_NAME
Per OCI distribution spec, it has to return 404, instead of 400 (project name required)
Signed-off-by: wang yan <wangyan@vmware.com>
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
* handle blob status chanage in put blob middlware
After blob is uploaded success, the middleware will update the blob status accordingly.
Signed-off-by: wang yan <wangyan@vmware.com>
* move send error to source lib
Move the sendError into library in case the cycle dependency as regsitry and core are now the consumers.
Signed-off-by: wang yan <wangyan@vmware.com>
* Add api to delete blob and manifest
Enable the capability of registry controller to delete blob and manifest
Signed-off-by: wang yan <wangyan@vmware.com>
When scanner like trivy handles the auth flow to pull image, it pings
the /v2 and access the token service url in response body, by default it
will be external endpoint of Harbor.
There will be problem when Harbor is deployed on a single node with hairpinning not
supported.
This commit makes sure the address of token service in the challenge is
internal url of core component when the request is from internal.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
The v2 catalog API needs to filter out the empty repository and the repository which artifacts are all with no tags.
1，In v2.0.0, Harbor does not delete repository even there is no artifact, it's different with v1.10.0
2, Compares with docker distribution, it doesn't return the respository with untagged images.
Signed-off-by: wang yan <wangyan@vmware.com>
Fixes#9704
As we do want to unify error handling, so just decreprates pkg errors, use lib/errors instead for Harbor internal used errors model.
1, The lib/errors can cover all of funcs of pkg/errors, and also it has code attribute to define the http return value.
2, lib/errors can give a OCI standard error format, like {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized"}]}
If you'd like to use pkg/errors, use lib/errors instead. If it cannot meet your request, enhance it.
Signed-off-by: wang yan <wangyan@vmware.com>
Customize the json output with message with err.Error(). Otherwise, the wrappged message will be lost
in the final errors object.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit removes the EnsureArtifactDigest as its implementation is
problematic: the artifactinfo in context is immutable.
When the content trust middleware needs the digest it will retrieve it
via artifact controller.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Ignore limitation when refresh quota for project.
2. Return 403 when quota errors occurred.
3. Add test for Refresh method of quota controller.
Closes#11512
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* Update tags related APIs
1. Remove API for listing tags of repository
2. Add API for listing tags of artifact
3. Support filter artifact by tag name
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* [OCI] modify artifact tag name check
1. switch api get tag list
2. modify artifact tag name check
Signed-off-by: Yogi_Wang <yawang@vmware.com>
Co-authored-by: Yogi_Wang <yawang@vmware.com>
Fixes#11241
1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Prevent the pull action when scan report status is not successfuly.
2. Bypass the checking when no vulnerabilities not found.
3. Improve the returned message when prevented the pull action.
Closes#11202
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Docker CLI fails if it's not logged in upon seeing "basic" realm challenging while pinging the "/v2" endpoint. (#11266)
Some CLI will send HEAD to artifact endpoint before pushing (#11188)(#11271)
To fix such problems, this commit re-introduce the token auth flow to the CLIs.
For a HEAD request to "/v2/xxx" with no "Authoirzation" header, the v2_auth middleware populates the
"Www-Authenticate" header to redirect it to token endpoint with proper
requested scope.
It also adds security context to based on the content of the JWT which has the claims of the registry.
So a request from CLI carrying a token signed by the "/service/token" will have proper permissions.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Fixes#11206
1, fix middleware doesn't work for docker pull without auth
2, fix middleware doesn't bypass scanner pull
Signed-off-by: wang yan <wangyan@vmware.com>
That was added to support core process sending request to `/v2/xxx`.
It's no longer needed after reworking the flow.
This commit removes this.
Fixes#10602, as it's not a case we need to support for now.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Fixes#11315
When specify no pagination in listing artifact request, the go-swagger will set the default value for them, so we need to iterate the link header to get all of artifacts
Signed-off-by: Wenkai Yin <yinw@vmware.com>
As we don't support bearer token in Harbor 2.0, the URL checking logic in auth proxy security generator should be updated
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Fixes#11016
1. src/pkg/q->src/internal/q
2. src/internal->src/lib (internal is a reserved package name of golang)
3. src/api->src/controller
Signed-off-by: Wenkai Yin <yinw@vmware.com>
The query string is encoded by UI, and we have to unescape the "=" in "q=tag=nil",
otherwise, the query doesn't work, and returns 400
Signed-off-by: wang yan <wangyan@vmware.com>
1, enable user view log api test case
2, update project logs api permission check
3, use project ctl instead in permission check base method
Signed-off-by: wang yan <wangyan@vmware.com>
1. Skip vulnerability prevention checking when artifact is not
scannable.
2. Skip vulnerability prevention checking when artifact is image index
and its type is `IMAGE` or `CNAB`.
3. Skip vulnerability prevention checking when the artifact is pulling
by the scanner.
4. Change `hasCapability` from blacklist to whitelist.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
