As we introduce a new table to record the relationship between the artifacts and labels, the current way to list label's resources doesn't work anymore, and the API isn't needed by any features, remove it in 2.0
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, enable audit logs for notifications
2, move the handler and meatadata into API
3, use the notification middleware to send out notification
Signed-off-by: wang yan <wangyan@vmware.com>
1. Use quota controller to implement the internal quota APIs.
2. The internal quota APIs can exceed the quota limitations.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
the notification is for send out the event after DB transaction complete.
It's safe to send hook as this middleware is after transaction in the response path.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Introduce `Evaluator` interface which do the permission checking.
2. `admin`, `lazy`, `rbac`, `namespace` and `evaluartor` set are implemented the
`Evaluator` interface.
3. Move project rbac implemention from `project` to `rbac` pkg to reduce
the name conflict with project instance of model.
4. Do permission checking in security context by `Evaluator`.
5. Cache the regexp in rbac evaluator for casbin.
6. Cache evaluator in namespace evaluator to improve performance.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit replaces beego's CSRF mechanism with gorilla's csrf library.
The criteria for requests to skip the csrf check remain the same.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, enable dao test for artifact trash
2, set default flush trash table to false
3, hanlder empty parameter in API call
4, add registry auth info into jobservice container
Signed-off-by: wang yan <wangyan@vmware.com>
1. Add Checker to check the scannable status of the artifact.
2. Only set vulnerabilities addition link when the artifact scanable in the
project.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
As we swtich to basic auth for /v2/* API
The CLI secret should handle /v2/* API so that OIDC user can use the
secret to do push/pull
This commit makes such change.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.
Signed-off-by: wang yan <wangyan@vmware.com>
* refactor(scan,scanner): move scan and scanner controllers to api pkg
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* feat(scan-all-job): move artifacts query from job to notification
Move artifact query from scan all job to its notification handler to
ensure that the components in pkg will not call controllers in api.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* Populate signature status in artifact API
This Commit add signature status into response of list artifact API.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Escape and clean request path for `legacyAPISkipper`.
2. Escape and clean request path for `MethodAndPathSkipper`.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Add `NewContext` and `FromContext` funcs in security pkg.
2. Add `Name` func in `security.Context` interface to make the checking
for the `/api/internal/configurations` API clear.
3. Get the security from the context to prepare change the security
filter to middleware.
4. Remove `GetSecurityContext` in filter pkg.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth. #10604 is opened
to track the follow up work.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
When the registry shifts from token auth to basic auth, we'll use the middleware to check permission.
This commit add middlewares for populate the artifact info and check
permission based on request to /v2/* api via security context
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Upgrade beego to v1.12.0
2. Add RequestID middleware to all HTTP requests.
3. Add Orm middleware to v2 and v2.0 APIs.
4. Remove OrmFilter from all HTTP requests.
5. Fix some test cases which cause panic in API controllers.
6. Enable XSRF for test cases of CommonController.
7. Imporve ReadOnly middleware.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Get and set orm from context.
2. Add WithTransaction decorator make func run in transaction.
3. Support nested transaction by Savepoint.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit updates `prepare` and templates to populate the credential
for registry for basic authentication.
A temporary flag `registry_use_basic_auth` was added to avoid breakage.
It MUST be removed before the release.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
We'll support manifest list, so this commit removes the manifest list middleware who blocks the manifest list pushing
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit disables XSRF check for "service/token" so that when
containerd sends `POST` it will not return 403 and containerd can
fallback to `GET` to complete the workflow.
Fixes#10305
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit will make the "tokenreview" security filter onboard
user if the request carries a valid token. If the "skipsearch" flag in
http_auth setting is set to false the onboard will fail.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, Leverage go v1.13 new error feature
2, Define genernal error OCI format, so that /v2 API could return a OCI compatible error
Signed-off-by: wang yan <wangyan@vmware.com>
Seperate the HasAdminRole(In DB) with the privileges from external auth, and use user.HasAdminPrivilege to check
Signed-off-by: stonezdj <stonezdj@gmail.com>
In order to replace travis.
Implement 5 CI jobs
- UTTEST
- APITEST_DB
- APITEST_LDAP
- OFFLINE
- UI_UT
Signed-off-by: Ziming Zhang <zziming@vmware.com>
statistic API use security Context to list project rather than calling
projectmanager directly, such that the group membership will be taken
into account.
fixes#10230
It should be cherry picked to 1.9.x and 1.10.x branches
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit removes the attribute to control case-sensitivity from
authproxy setting.
The result in token review status will be used as the single source of
truth, regardless the case of the letters in group names and user names.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit add an attribute to configurations, whose value is the
certificate of authproxy server. When this attribute is set Harbor will
pin to this cert when connecting authproxy.
This value will also be part of the response of systemInfo API.
This commit will be cherrypicked to 1.10 and 1.9 branch.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit fixes#9869
It has some refactor to make sure the group is populated when user is
authenticated via tokenreview workflow.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit fixes#9946, that when creating a project the minimum length
should be 1, not 2.
This commit should be cherry picked to 1.9.x and 1.10.x branch .
We need to double check if this change impacts the creation of replication
rule.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit refactors the flow to populate user info and verify CLI
secret in OIDC authentication.
It will call the `userinfo` backend of OIDC backend and fallback to
using the ID token if userinfo is not supported by the backend.
It also makes sure the token will be persisted if it's refreshed during
this procedure.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
BREAKING CHANGE: the value negligible of severity in project metadata will change to none in the responses of project APIs
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Fix name conflict when install internal clair adapter.
2. Remove all internal adapters when reinstall harbor without --with-clair flag
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit make case sensitivity configurable when the authentication
backend is auth proxy.
When the "http_authproxy_case_sensitive" is set to false, the name of
user/group will be converted to lower-case when onboarded to Harbor, so
as long as the authentication is successful there's no difference regardless
upper or lower case is used. It will be mapped to one entry in Harbor's
User/Group table.
Similar to auth_mode, there is limitation that once there are users
onboarded to Harbor's DB this attribute is not configurable.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit fixes#9771
It compares the roles to return the one with highest permission in the
response of `GET /api/projects`.
In addition to that, it adds the role list to the response, because a
user can have multiple roles in a project.
It also removes the togglable attribute as it's not used anywhere.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit is to fix https://github.com/goharbor/harbor/issues/9780.
To align with OCI spec, when a docker pull request without bearer token in header comes in, Harbor should not intecepte it(return a 412 if check fail)
when the policy check is enabled. As the 401 is expected by the docker/caller, and then to ask token service which url is in the 401 header.
Signed-off-by: wang yan <wangyan@vmware.com>
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job
fix#9705fix#9722fix#9670
Signed-off-by: Steven Zou <szou@vmware.com>
The severity saved in db is lowercase but the severities in vuln pkg
begin with upper letter, this fix use func to transform project severity
value from db to vuln.Severity.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
As the quota sync is default called by harbor-core on every launch, and it will break the launch process if any failure throwed.
1, The commit is to provide an switcher for the system admin to bypass the quota sync.
2, In case Harbor goes into the restarting cycle.
Harbor already provides an internal API to sync quota data, in the failure case,
system admin can launch harbor and call the /api/internal/syncquota to sync quota.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Remove ping action when initialize clair adapter installed by harbor.
2. Remvoe the `IsDefault` property when initialize clair adapter that
make it switch to auto detecting.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
- add a common error pkg to support error with code and AsError check
- replace some errors in scan with coded errors
- fix#9668
Signed-off-by: Steven Zou <szou@vmware.com>
1, replace the UIVERSION file with ldflags, which is generarted by make to inject into the UI core.
2, inject additional ldflags for harbor compiler
Signed-off-by: wang yan <wangyan@vmware.com>
- add new endpoint for getting scanner candidates of specified project
- adjust the permission granting functions
- fix#9608
Signed-off-by: Steven Zou <szou@vmware.com>
1, eat the unknown manifest error, and log it. The migration process will not crashed on it.
2, enable to persist DB of sync quota API.
3, add empty project support.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Upgrade clair adapter to v1.0.0.
2. Make the clair adapter which installed by harbor immutable and using internal registry address.
3. Add support to build clair adapter image from binary.
4. Switch to ScannerPull action when make authorization for the scan request.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
- do changes to the sql schema
- add `UseInternalAddr` and `Immutable` properties to scanner registration
- support multiple authentication type
- basic
- bearer token
Signed-off-by: Steven Zou <szou@vmware.com>
1, the commit is for internal robot to bypass policy check, like vul and signature checking.
2, add a bool attribute into registry token, decode it in the harbor core and add the status into request context.
3, add a bool attribut for robot API controller, but API will not use it.y
Signed-off-by: wang yan <wangyan@vmware.com>
As we introduce the pluggable scanner, users can add the external scanners, so we remove the Clair from the health check API
Signed-off-by: Wenkai Yin <yinw@vmware.com>
- add LCM control to the robot account generated for scanning
- improve the scan webhook
- remove reprots when related artifact is deleted
- update report manager/scan controller and other components to support above cases
- add artifact manager/comtroller to list artifacts
Signed-off-by: Steven Zou <szou@vmware.com>
1. Only show project member info when has member list permission.
2. Only show quota info when has quota read permission.
3. Add quota read permission for all roles of project.
4. Refactor permission service in portoal.
5. Clear cache when clear session.
Closes#8697
Signed-off-by: He Weiwei <hweiwei@vmware.com>
To contain various vulnerabilities in the CVE whitelist, this commit
removes the validation.
Fixes#9242
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1) Disassociate id token from user session
2) Some OIDC providers do not return id_token in the response of refresh
request:
https://openid.net/specs/openid-connect-core-1_0.html#RefreshTokenResponse
When validating the CLI secret it will not validate the id token,
instead it will check the expiration of the access token, and try to
refresh it.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema
Signed-off-by: Steven Zou <szou@vmware.com>
- refactor the db schema \
- refactor permission checking in API handlers \
to follow the latest code/interface changes
Signed-off-by: Steven Zou <szou@vmware.com>
1, add API controller for robot account, make it callable internally
2, add Manager to handler dao releate operation
Signed-off-by: wang yan <wangyan@vmware.com>
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases
- update robot account making content
- hidden credential in the job log
Commnet scan related API test cases which will be re-activate later
fix#8985
fix the issues found by codacy
Signed-off-by: Steven Zou <szou@vmware.com>
This commit is to fix the issue for the following scenario:
1, user success migrate harbor to v1.9.0 from a previous version
2, add a project, push images into the project.
3, delete images and then to delete the project.
4, re-launch harbor.
After that, it still execute the quota migration as the condition doesn't consider the deleted projects usage.
And in this case, the harbor core crashes with a duplicate sql err, and unable to launch.
[Workaroud]
Clean table of project_blob with: TRUNCATE TABLE project_blob, and re-launch harbor, wait for quota sync success.
Signed-off-by: wang yan <wangyan@vmware.com>
- change error collection in scan job
- add dead client checking in client pool
- change key word type to interface{} for q.Query
- update bearer authorizer
- add required UT cases
Signed-off-by: Steven Zou <szou@vmware.com>
This commit returns a common error message when testing the webhook endpoint and prints the detail in the log for debug
Signed-off-by: Wenkai Yin <yinw@vmware.com>
- add scanner rest API v1 spec
- implement v1 client which is used to talk to scanner adapter
- adjust data/orm models
- adjust code package structure
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan client which is used to talk to scanner adapter
- implement scan job which take the work of communicating with scanner
- update scanner mgmt API routes
- add corresponding UT cases
This commit mitigates the Session Fixation issue by making sure a new
session ID is generated each time user logs in to Harbor
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit enable project admin to add group as project member when
Harbor is configured against OIDC as AuthN backend.
It populates the information of groups from ID Token based on the claim
that is set in OIDC settings.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- add DAO layer for scanner registration
- add CURD manager for scanner registration
- add API controller for plug scanner
- add REST APIs for CURD of plug scanner
- add migration sql:0011_1.10.0
- add scan interface definition (no implementations)
- add related UT cases with testify
fix#8979#8990
Signed-off-by: Steven Zou <szou@vmware.com>
This commit add the new setting "oidc_groups_claim" to Harbor's
configurations.
And add "group_claim" to OIDCSetting struct.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Sync blobs from manifest for image with foreign layers.
2. Ignore size of foreign layers when compute size quota.
3. Fix repo info of artifact when upgrade from 1.8 version.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
If wrong OIDC scopes are defined, or there are some configuration errors, the OIDC callback query string might contain "error=..." with an error message. Intercept this case and show an error to the user instead of trying to exchange the token with a missing "code" parameter.
Signed-off-by: Iradier, AlvaroJose <AlvaroJose.Iradier@adidas.com>
Change error variable name
Signed-off-by: Iradier, AlvaroJose <AlvaroJose.Iradier@adidas.com>
the image may has the same blobs as the references, which causes the artifact & blobs
can not be inserted by unique constraint
Signed-off-by: wang yan <wangyan@vmware.com>
The foreign layer won't be counted into project quota
NOTE: the foreign layer will be dumped from the registry in the migration
Signed-off-by: wang yan <wangyan@vmware.com>
1, fix#8802, update the error formet
2, fix#8807, raise the real retag error to UI
3, fix#8832, raise the real chart error to chart client & ut
Signed-off-by: wang yan <wangyan@vmware.com>
