Commit Graph

199 Commits

Author SHA1 Message Date
Qian Deng
a15983432c Add trace for core
* Add trace related lib
* Add trace middleware for core
* add rid for middleware

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
He Weiwei
d00024ab36 refactor: initialize the remote helper using ctx from http request
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-09-09 11:13:53 +00:00
He Weiwei
06f2414d1c
fix: use ctx from http request for middlewares (#15523)
1. Use ctx from http request for the readonly middleware.
2. Refactor the AuthenticateHelper to let it get orm from ctx of the http request.
3. Change to use ctx from http request for oidc and authproxy http handlers.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-09-02 19:05:35 +08:00
Wang Yan
14f7274989
support robot to list project (#15431)
1, add permission check for API of List Projects
2, add permission check for API of List Repositories
3, use the self defined query to handle both names and public query

Signed-off-by: wang yan <wangyan@vmware.com>
2021-08-17 16:35:36 +08:00
Wang Yan
f7a4401dcb
deprecate version 1 robot account (#15296)
1, deprecate support for version 1 robot support, the robotv1 cannot be used anymore.
2, reserve the /project/{id_or_name}/robots api.

After the PR, user cannot use the robotv1 to login, and do any interaction with Harbor,
but still can view & delete them with UI or API.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-07-13 13:39:44 +08:00
Wang Yan
b158086642
fix conformance failure (#15261)
fixes #15252

Give 404 for invalid digest when to get/head manifest

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-07-06 15:11:13 +08:00
Wang Yan
2a1c9ec96a
update the log level of robot v1 context (#15037)
The robotv1 context uses the robot$ as a hardcoded prefix to identify robot account, it will raise error
for a valid robotv2 account with this prefix.

Update the log level to avoid the redundant logs for the default installation.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-06-01 18:30:40 +08:00
Daniel Jiang
238bea2066
Merge pull request #15010 from reasonerjt/v2auth-enhancement
Make v2auth more strict
2021-05-31 13:10:31 +08:00
Daniel Jiang
486554caa1
Merge pull request #15011 from reasonerjt/merge-slash-middleware
Add merge slash middleware
2021-05-31 13:09:39 +08:00
Wenkai Yin(尹文开)
4ed5fee681
Merge pull request #14949 from stonezdj/21may22_fallback_localregistry
Fall back to local registry when upstream registry is not working
2021-05-31 11:00:43 +08:00
Daniel Jiang
f4ac81b710 Add merge slash middleware
There are code in the core component to conditionally execute code based
on the pattern of url path, and different ingress controller or reverse
proxy may handle the dup slashes in the url path differently.
This commit merge dup slashes in the url paths to make things more
consistent.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-05-28 17:24:48 +08:00
Daniel Jiang
d3b8c613fd Make v2auth more strict
This commit enhances the v2auth middleware, such that any un-recognized
request sent to /v2/ will be blocked.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-05-28 16:41:48 +08:00
Wang Yan
42a9d0d905
remove common project code (#14939)
move project model from common to pkg

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-25 11:01:19 +08:00
stonezdj
d7d26c0966 Fall back to local registry when upstream registry is not working
Fixes #14822
When upstream registry not working, but status might stay healthy because the health check interval is 5 minutes, if a pull request comes before registry status turns to unhealthy, the proxy cache middleware might proxy the request to the upstream registry and get a 401 error and this 401 error might translate to a http 500 error to the client eventually.

To solve this issue, it fall back all error to local registry when proxying manifest except the NotFoundError from the local registry.

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-05-24 14:29:05 +08:00
Wenkai Yin(尹文开)
0fe551274d
Merge pull request #14921 from ywk253100/210519_db
Clean up tech debt codes
2021-05-20 13:44:10 +08:00
Wenkai Yin
998e392bb4 Clean up tech debt codes
Clean up tech debt codes

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-20 11:47:46 +08:00
Daniel Jiang
294ee52d7a
Merge pull request #14900 from reasonerjt/rm-common-dao-getuser-onboard
Remove GetUser and Onboard from common/dao
2021-05-19 11:21:00 +08:00
Daniel Jiang
952644e23f Remove GetUser and Onboard from common/dao
Replaced by funcs in src/pkg/user and src/controller/user

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-05-18 15:28:41 +08:00
He Weiwei
0c315d8aee
refactor: remove allowlist in GetSummary of scan controller (#14836)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-05-18 14:01:59 +08:00
Daniel Jiang
fa46b0d736
Merge pull request #14095 from jsoref/tokens
Tokens
2021-05-13 13:25:58 +08:00
stonezdj(Daojun Zhang)
0dfc801a50
Error string should not be capitalized (#14840)
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-05-12 16:53:37 +08:00
Daniel Jiang
6d0e391740 Move user related funcs from common/dao
This commit moves more user related funcs, such as ChangePassword,
Login, ChangeUserProfile from common/dao to rely on /pkg/user and
pkg/oidc.
It also removes the code for resetting user's password as it's disabled.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-05-12 01:12:57 +08:00
Daniel Jiang
c2ab1769b3
Merge pull request #14768 from reasonerjt/fix-14711
Return 401 for GET request to /v2 API for public artifacts.
2021-04-29 15:23:45 +08:00
Daniel Jiang
6a379d398c Return 401 for GET request to /v2 API for public artifacts.
This commits make sure when the request does not carry authorization
headers, the HEAD and GET will get the same response code.  This change
should be made due to #14711

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-04-28 09:14:43 +08:00
Daniel Jiang
5b526b8dc7
Remove dependencies from pkg/oidc to common/dao (#14739)
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-04-26 10:56:49 +08:00
stonezdj
60478f4990 Move common config api to lib/config
Register all config managers, and get it by getConfigManger()

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-13 19:43:33 +08:00
stonezdj
107e468b60 Refactor configure api to new programming model
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-09 08:10:11 +08:00
Daniel Jiang
ad8eee8623 Add attribute admin username for authproxy
This commit adds the attribute "http_authproxy_admin_usernames", which
is string that contains usernames separated by comma, when a user logs
in and the username in the tokenreview status matches the setting of
this attribute, the user will have administrator permission.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-04-07 18:14:59 +08:00
Wenkai Yin(尹文开)
28596c3ffb
Refactor registry API (#14528)
* Refactor registry API

Refactor registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Fix bugs of replications

1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-31 15:49:23 +08:00
Wang Yan
85254ccc22
refactor immutable rule (#14371)
Migrate immutable realted APIs to v2 swagger

Signed-off-by: wang yan <wangyan@vmware.com>
2021-03-08 17:10:12 +08:00
Daniel Jiang
e96c1cbced Switch API to ping OIDC endpoint to new model
This commit updates the API POST /api/v2.0/system/oidc/ping to new
programming model, in which the code will be generated by go-swagger.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-03-04 15:44:11 +08:00
Wang Yan
21d35f9702
update blob list query (#14195)
* update blob list query

Deprecate blob list parameters, and use the query for instead.

Signed-off-by: wang yan <wangyan@vmware.com>

* update per review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-02-09 17:08:26 +08:00
Wenkai Yin(尹文开)
120d88a0dd
Merge pull request #14116 from reasonerjt/reduce-oidc-warning
Reduce warning logs in OIDC middleware
2021-02-01 09:55:41 +08:00
DQ
15ad870262 Fix: unkonw metrics issue
Signed-off-by: DQ <dengq@vmware.com>
2021-01-29 18:07:06 +08:00
Daniel Jiang
2dd499bacf Reduce warning logs in OIDC middleware
If the request does not have bearer token in the header, do not decode
the empty string.
Fixes #12261

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-28 18:08:28 +08:00
Josh Soref
5be895cb39 Check return from token.DefaultTokenOptions()
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2021-01-26 20:39:55 -05:00
DQ
28ae77e5c6 Aggregate metrics
1. Add operation id in ctx in baseapi before prepare
2. add operation id for registry proxy request
3. use url for other request

Signed-off-by: DQ <dengq@vmware.com>
2021-01-25 09:59:10 +08:00
Wang Yan
77347c54cf fix robot issues
fixes #13980
fixes #13981

1, add the robot prefix to the audit log
2, add duration maximum checking

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-13 18:06:43 +08:00
Wang Yan
2d4456c630
refractor project rbac (#13924)
As the system rbac introduced, move the code of project rbac into project directory

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-11 11:27:26 +08:00
Daniel Jiang
efa63d905a
Update the reg-exp to match v2/catalog api (#13941)
A more strict check is applied such that all requests to
/v2/_catalog/...  will be verified.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-09 13:17:51 +08:00
Wang Yan
0cf43d766c
enable system resource access (#13826)
1, introduce & define the system resources.
2, replace the IsSysAdmin judge method.
3, give the robot the system access capability.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-07 15:45:04 +08:00
stonezdj(Daojun Zhang)
b748852ee8
Merge pull request #13805 from stonezdj/201218_add_content_type_length
Add content type and length in header
2021-01-06 15:02:34 +08:00
stonezdj
1d50be31aa Refactor Manifest cache process
Separate manifest, manifest list and image index.

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-01-06 14:01:02 +08:00
Wang Yan
0271efd3f7
enable visible when to list/create robot (#13840)
1, enable the visible attribute when to create/list robots
2, rename package name from robot2 to robot

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-04 10:24:31 +08:00
prahaladdarkin
a890b28e1e
Store vulnerability data from scanner into a relational format (#13616)
feat: Store vulnerability report from scanner into a relational format

Convert vulnerability report JSON obtained  from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2020-12-25 08:47:46 +08:00
stonezdj
aa3002e7a5 Add content type and length in header
Fixes #13740
Update ManifestExist to return Descriptor instead of digest
For docker 20.10 or containerd, it HEAD the manifest before pull, then
it GET the manifest with digest, add logic to handle this scenario and
correlate the tag between the digest in proxy cache

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-12-21 20:21:28 +08:00
stonezdj
3334defd92 Revert "fix issue"
This reverts commit e6f80259
This reverts commit 918fe125

Signed-off-by: stonezdj <stonezdj@gmail.com>

Revert "Add content type and length in header"

This reverts commit ca379111

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-12-18 12:35:39 +08:00
stonezdj
ca37911113 Add content type and length in header
Fixes #13740
Update ManifestExist to return Descriptor instead of digest

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-12-17 15:42:49 +08:00
stonezdj(Daojun Zhang)
1eb0287ecb
Merge pull request #13709 from stonezdj/201209_dockerhub_limit2
Cache manifest list for proxy cache
2020-12-15 14:03:39 +08:00
stonezdj
670a94835b Cache manifest list for proxy cache
Fixes #13566: Quota of dockerhub is still used in v2.1.1 after the image is cached
Cache manifest list in redis cache.
Trade off between efficiency and data integrity, it might cause the proxy cache return the full content of a manifest list instead of the actual manifest list saved in the Harbor storage, which is a part of the manifest list. but this change doesn't break any /v2/ API, just caches full manifest list.

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-12-15 11:30:30 +08:00