Commit Graph

834 Commits

Author SHA1 Message Date
Wang Yan
6e03c8a54e
Merge pull request #9896 from heww/owner-check-for-project-member-robot-account
fix(robot,project-member): check owner of member, robot when update, …
2019-11-15 16:53:22 +08:00
Wang Yan
7b12ed14a1
Merge pull request #9852 from stonezdj/remove_tedious_msg
Change log level to avoid tedious error in log
2019-11-15 10:42:28 +08:00
He Weiwei
5bd1cfdbf2 fix(robot,project-member): check owner of member, robot when update, delete
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-11-14 07:00:44 +00:00
Wang Yan
29be93725d
Merge pull request #9860 from reasonerjt/authproxy-case-sensitive-master
Authproxy case sensitive master
2019-11-14 14:03:53 +08:00
Daniel Jiang
8933ab8074 Add configuration "case sensitive" to HTTP auth proxy
This commit make case sensitivity configurable when the authentication
backend is auth proxy.
When the "http_authproxy_case_sensitive" is set to false, the name of
user/group will be converted to lower-case when onboarded to Harbor, so
as long as the authentication is successful there's no difference regardless
upper or lower case is used.  It will be mapped to one entry in Harbor's
User/Group table.
Similar to auth_mode, there is limitation that once there are users
onboarded to Harbor's DB this attribute is not configurable.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-11-13 15:00:05 +08:00
stonezdj
dc5cb3504c Change log level to avoid tedious error in log
change from error to debug

Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-11-13 11:15:00 +08:00
stonezdj
4d822e0a19 Fix review comments on PR9749
Fix review comments on PR9749
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-11-12 13:05:11 +08:00
Wang Yan
407417ce7b
Merge pull request #9810 from stonezdj/bug9479
Populate group from auth provider to Harbor when user login
2019-11-11 19:52:31 +08:00
stonezdj
0c011ae717 Populate group from auth provider to Harbor DB when user login
Fix #9749, change include LDAP auth, OIDC auth, HTTP auth

Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-11-11 14:41:35 +08:00
Daniel Jiang
64dc5122e6 Add role list in project response
This commit fixes #9771

It compares the roles to return the one with highest permission in the
response of `GET /api/projects`.
In addition to that, it adds the role list to the response, because a
user can have multiple roles in a project.
It also removes the togglable attribute as it's not used anywhere.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-11-11 14:03:50 +08:00
Daniel Jiang
7d04eab63c
Merge pull request #9593 from qyqcswill/code_improve
promote code quality
2019-11-08 18:28:46 +08:00
Steven Zou
ee31418e8e revoke scan permission from the developer role
Signed-off-by: Steven Zou <szou@vmware.com>
2019-11-06 17:57:48 +08:00
Steven Zou
ebc5d2482b do improvements to the scan all job
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job

fix #9705
fix #9722
fix #9670

Signed-off-by: Steven Zou <szou@vmware.com>
2019-11-05 15:12:07 +08:00
He Weiwei
ae8931e816 fix(policy-checker): add func to transform project severity to vuln.Severity
The severity saved in db is lowercase but the severities in vuln pkg
begin with upper letter, this fix use func to transform project severity
value from db to vuln.Severity.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-31 14:11:44 +00:00
He Weiwei
3c80832341 fix(quota): order by quotas only on support resources
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-30 02:42:34 +00:00
Daniel Jiang
b17711abbf
Merge pull request #9592 from qyqcswill/code_clean
remove useless code
2019-10-29 15:08:59 +08:00
Steven Zou
5b2ab34e03 permission grant for scanner related actions are not correctly
- add new endpoint for getting scanner candidates of specified project
- adjust the permission granting functions
- fix #9608

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-28 18:20:47 +08:00
Wenkai Yin(尹文开)
f007a62b04
Merge pull request #9588 from stonezdj/fix_ldap_group_sql
Fix User Group Search SQL error
2019-10-28 11:22:14 +08:00
hao.cheng
29e905271d promote code quality
Signed-off-by: hao.cheng <hao.cheng@daocloud.io>
2019-10-25 15:37:35 +08:00
hao.cheng
94bc8c2f5c remove useless code
Signed-off-by: hao.cheng <hao.cheng@daocloud.io>
2019-10-25 15:20:25 +08:00
stonezdj
f402db380b Fix User Group Search SQL error
User Group Query SQL error in some cases

Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-10-25 14:00:45 +08:00
Wang Yan
d18678a48d
Merge pull request #9506 from wy65701436/token-sevice
Enable robot account to support scan pull case
2019-10-24 19:52:33 +08:00
wang yan
71c769ec97 remvoe bypass to scanner pull
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-24 17:49:20 +08:00
wang yan
a6ad1b2db8 update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 20:05:51 +08:00
wang yan
2fa85aefca fix per comments
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 18:45:30 +08:00
wang yan
5996189bb0 update per comments and fix govet error
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 18:45:30 +08:00
wang yan
22b4ea0f89 Enable robot account bypass policy check
1, the commit is for internal robot to bypass policy check, like vul and signature checking.
2, add a bool attribute into registry token, decode it in the harbor core and add the status into request context.
3, add a bool attribut for robot API controller, but API will not use it.y

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 18:45:30 +08:00
Wenkai Yin
9d896d4d72 Remove the health checker for Clair in health check API
As we introduce the pluggable scanner, users can add the external scanners, so we remove the Clair from the health check API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-10-23 12:55:03 +08:00
stonezdj(Daojun Zhang)
4dcd323b4a
Merge pull request #9475 from wy65701436/immu-delete-repo
add immutable match in the repository/tag delete api
2019-10-22 17:28:15 +08:00
Wang Yan
fc106e218c
Merge pull request #9503 from heww/issue-9308
fix(configuration): E notation support for int64 and quota types
2019-10-22 11:50:06 +08:00
Wang Yan
3772ccc163
Merge pull request #9493 from stonezdj/remove_nested_group
Remove nested group search
2019-10-21 17:45:50 +08:00
He Weiwei
7c8f5426ed fix(configuration): E notation support for int64 and quota types
Closes #9308

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-21 08:15:27 +00:00
wang yan
424f11e697 add immutable match in the repository/tag delete api
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-21 15:53:24 +08:00
stonezdj
b148ffe6a8 Remove the nested group search
Remove the code change in #8378, because the previous code change caused issues: #9092, #9110, #9326

Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-10-21 14:34:53 +08:00
He Weiwei
e254fe3095
fix(permissions): permissions checking for member and quota info (#9490)
1. Only show project member info when has member list permission.
2. Only show quota info when has quota read permission.
3. Add quota read permission for all roles of project.
4. Refactor permission service in portoal.
5. Clear cache when clear session.

Closes #8697

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-21 14:03:52 +08:00
He Weiwei
bf6a14c9ad
feat(role): introduce a limited guest role (#9403)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-20 14:21:28 +08:00
Wenkai Yin(尹文开)
f98196e5ba
Merge pull request #9435 from reasonerjt/oidc-refresh-refine
Update OIDC token refresh process
2019-10-18 19:43:34 +08:00
Steven Zou
0f16913635 rebase: resolve the code confilcts with master
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-17 17:42:41 +08:00
Wenkai Yin(尹文开)
97ddff2ac8
Merge pull request #9434 from heww/clair-adapter
build(clair): internal clair adapter when install with clair
2019-10-17 16:06:10 +08:00
He Weiwei
8964a8697a build(clair): internal clair adapter when install with clair
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-17 12:00:51 +08:00
Daniel Jiang
f0cb16cb86 Update OIDC token refresh process
1) Disassociate id token from user session

2) Some OIDC providers do not return id_token in the response of refresh
request:
https://openid.net/specs/openid-connect-core-1_0.html#RefreshTokenResponse
When validating the CLI secret it will not validate the id token,
instead it will check the expiration of the access token, and try to
refresh it.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-10-17 11:26:18 +08:00
Steven Zou
f18afc0a3f do changes to let the vul policy check compatiable with new framework
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-16 23:15:26 +08:00
stonezdj(Daojun Zhang)
2973ddcf6b
Merge pull request #9428 from stonezdj/disable_self_reg
Update default self_registration=false
2019-10-16 17:41:21 +08:00
stonezdj
3636a1afa5 Update default self_registration=false
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-10-16 01:33:48 -07:00
He Weiwei
d9a539807b perf(test): speed up TestAddBlobsToProject test in dao pkg
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-14 16:46:01 +00:00
Daniel Jiang
ee9e92b6dd
Merge pull request #9157 from phin1x/master
Escape user dn in ldap group search filter
2019-10-14 16:41:27 +08:00
Wenkai Yin(尹文开)
7d0505593f
Merge pull request #8556 from chlins/feat/image-replication-adapter-for-quay.io
Feat/image replication adapter for quay.io
2019-10-14 09:16:45 +08:00
Steven Zou
a86afd6ebc Merge branch 'master' into feature/pluggable_scanner_s3_merge 2019-10-12 15:18:06 +08:00
wang yan
6f6f113f0f refactor robot api
1, add API controller for robot account, make it callable internally
2, add Manager to handler dao releate operation

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-11 17:26:18 +08:00
Steven Zou
58afd8e14b [stage3] support pluggable scanner
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces

Signed-off-by: Steven Zou <szou@vmware.com>

- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases

- update robot account making content
- hidden credential in the job log

Commnet scan related API test cases which will be re-activate later
fix #8985

fix the issues found by codacy

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-11 12:53:02 +08:00
chlins
4ab3b864ae feat: add image replication adapter for quay.io
Signed-off-by: chlins <chlins.zhang@gmail.com>
2019-10-11 10:00:07 +08:00
stonezdj(Daojun Zhang)
a2938c5d78
Merge pull request #9274 from wy65701436/immu-refatctor
refactor immutable dao code to align the new structure under pkg
2019-10-10 10:38:22 +08:00
He Weiwei
4ce72e37c4 fix(robot): robot account improvement for policies
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-09-27 03:07:58 +00:00
wang yan
7c4fd79b5c refactor immutable dao code to align the new structure under pkg
1, add manager
2, move model dao to /pkg/dao

Signed-off-by: wang yan <wangyan@vmware.com>
2019-09-26 20:35:58 +08:00
stonezdj
cc22a175b9 Add immutable tag API
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-09-25 15:53:56 +08:00
stonezdj(Daojun Zhang)
ec559b0585
Merge pull request #9123 from stonezdj/immutable_tags
Add DAO for immutable tags
2019-09-23 21:46:07 +08:00
stonezdj
29d2bcce99 Add DAO for immutable tags
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-09-23 16:45:07 +08:00
Fabian
1467f4bbb1 Escape User DN
Signed-off-by: Fabian Weber <fa.weber@enbw.com>
2019-09-19 14:29:09 +02:00
Daniel Jiang
b21f9dc6f1 Support OIDC groups
This commit enable project admin to add group as project member when
Harbor is configured against OIDC as AuthN backend.

It populates the information of groups from ID Token based on the claim
that is set in OIDC settings.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-09-19 17:49:31 +08:00
Daniel Jiang
f36efa4dcd Add groups claim to OIDC configuration
This commit add the new setting "oidc_groups_claim" to Harbor's
configurations.
And add "group_claim" to OIDCSetting struct.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-09-16 15:54:14 +08:00
Daniel Jiang
753219834e
Merge pull request #8960 from ninjadq/upgrade_hash_alg_for_pswd
Upgrade hash alg for pswd
2019-09-12 11:22:39 +08:00
Wenkai Yin
089eb4c449 Add the port 8080 to the default URL of portal to avoid the health check API failure
We changed the listenning port of portal from 80 to 8080 to run the process as non-root user, but the change didn't update the default URL of portal in source code, this causes the health check API fail.

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-09-11 10:45:55 +08:00
DQ
ea5c27fcd5 Enhance: Upgrade encrypt alg to sha256
previous sha1 will still used for old password

Signed-off-by: DQ <dengq@vmware.com>
2019-09-09 21:48:21 +08:00
Wenkai Yin
3b07be5a72 Check the status behind error when trying to update the scan schedule
Check the status behind error when trying to update the scan schedule

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-09-09 13:31:10 +08:00
Wang Yan
2194834b41
Merge pull request #8910 from heww/foreign-layers
fix(quota): correct size quota for image with foreign layers
2019-09-03 00:29:24 +08:00
He Weiwei
f44b75f398 fix(quota): correct size quota for image with foreign layers
1. Sync blobs from manifest for image with foreign layers.
2. Ignore size of foreign layers when compute size quota.
3. Fix repo info of artifact when upgrade from 1.8 version.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-09-02 14:10:58 +00:00
wang yan
d3f7d01a69 fix int out of range when to set usage in GC job
Signed-off-by: wang yan <wangyan@vmware.com>
2019-09-02 18:48:10 +08:00
wang yan
c28920c84f fix #8807
Format the error of mount blob, return a http error so that the core can parse it.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-30 16:29:47 +08:00
wang yan
16b910e1cf fix(quota/sync) #8886
The foreign layer won't be counted into project quota
NOTE: the foreign layer will be dumped from the registry in the migration

Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-29 17:29:40 +08:00
Wang Yan
db5781bf78
Merge pull request #8860 from wy65701436/fix-quota-sync
fix quota sync issues
2019-08-29 13:45:38 +08:00
wang yan
5decb56369 update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-29 12:46:42 +08:00
Wenkai Yin(尹文开)
5da4286ef4 Hard delete project metadata (#8856)
Hard delete project metadata

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-08-29 12:14:39 +08:00
wang yan
942e793f20 fix quota sync issues
1, fix #8858, add retry to ping backend service
2, fix #8859, split the blobs data when larger then 65535

Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-28 18:59:25 +08:00
He Weiwei
2c1c816941
fix(database): generate db url by url.URL for schema upgrade (#8852)
Closes #7948

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-08-28 16:59:22 +08:00
Ziming
94138137d5
add valid for rule (#8846)
Change-Id: I82215a0cf1ec32a253c8db9bfafe7e25b26c9ad9
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2019-08-28 16:58:49 +08:00
wang yan
19f543a025 fix sql in remove blob from project
the project id is missing in the method, that makes GC to clean all of items,
and if quota will not compute twice for the existing manifest.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-28 00:38:11 +08:00
Wenkai Yin(尹文开)
7262cc4c1a
Merge pull request #8836 from wy65701436/update-quota-error
Revise quota errors to make it more readable
2019-08-27 11:34:36 +08:00
wang yan
f343b2ec45 Revise quota errors to make it more readable
1, fix #8802, update the error formet
2, fix #8807, raise the real retag error to UI
3, fix #8832, raise the real chart error to chart client & ut

Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-26 17:00:31 +08:00
wang yan
e3155e00d6 fix #8815 :add remove untagged blob record in table project_blob
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-26 15:57:19 +08:00
mmpei
d5f87063e4
Merge branch 'master' into official-wehook-events-20190811 2019-08-22 22:07:12 -05:00
wang yan
2d569192ab fix quota count size for same manifest in different repo
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-23 00:56:45 +08:00
Wenkai Yin(尹文开)
21f8290110
Merge pull request #8777 from heww/issue-8635
fix(rbac): NewProjectNamespace in rbac only accept projectID
2019-08-22 17:52:27 +08:00
Wenkai Yin(尹文开)
6198ed2634
Merge pull request #8758 from heww/issue-8681
refactor(quota,middleware): skip overflow error when subtract resources
2019-08-22 13:54:01 +08:00
He Weiwei
8effdc6f18 fix(rbac): NewProjectNamespace in rbac only accept projectID
Closes #8635

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-08-21 19:37:28 +00:00
wang yan
e91ded65cb fix quota size usage in gc job, issue #https://github.com/goharbor/harbor/issues/8699
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-21 16:51:31 +08:00
Wang Yan
4bccb17236
Merge pull request #8749 from heww/issue-8493
fix(quota-driver): owner name of project quota reference object
2019-08-21 13:47:17 +08:00
He Weiwei
c22bf2539e refactor(quota,middleware): skip overflow error when subtract resources
1. Skip overflow error when subtract resources
2. Take up resources before handle request and put it back when handle
failed for add action in quota interceptor
3. Free resources only after handle success for subtract action in quota
interceptor

Closes #8681

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-08-20 14:41:55 +00:00
He Weiwei
8eb17be13c fix(quota-driver): owner name of project quota reference object
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-08-20 07:03:11 +00:00
stonezdj
7c7b6d2710 Normalize LDAP filter for user filter and group filter
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-08-20 10:55:30 +08:00
Daniel Jiang
f10fb67d6d
Merge pull request #8662 from stonezdj/email_sec2
Set default email to null if not provided
2019-08-20 09:01:50 +08:00
He Weiwei
75772aae11
refactor(quota): new error types for quota checking (#8726)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-08-19 19:00:29 +08:00
stonezdj
5fa8eb7854 Set default email to null if not provided
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-08-19 15:20:44 +08:00
Daniel Jiang
b3abd0316b
Merge pull request #8713 from reasonerjt/fix-8702
Avoid overwriting system CVE whitelist by mistake
2019-08-19 01:42:58 +08:00
Daniel Jiang
504202ecfd
Merge pull request #8378 from Typositoire/ldap/nested-groups
Search for LDAP_MATCHING_RULE_IN_CHAIN groups
2019-08-18 16:07:16 +08:00
Wang Yan
7a41d89ac8 Add quota sync api toi to sync quota data with backend storage
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-16 14:55:46 +08:00
Daniel Jiang
30bb2ddcdf Avoid overwriting system CVE whitelist by mistake
Fixes #8702
Also enforce the code to mitigate the potential risk.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-08-16 13:28:16 +08:00
Qian Deng
89aed1a1ea
Merge pull request #8672 from ywk253100/190815_content_length
Set content length when pushing blobs
2019-08-15 12:45:35 +08:00
Wenkai Yin
b94a99dded Set content length when pushing blobs
Set content length when pushing blobs

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-08-15 10:52:08 +08:00
Wang Yan
bf0b5a3fd0
Merge pull request #8663 from wy65701436/fix-quota-api
Fix quota switch fail to get project size
2019-08-15 10:49:49 +08:00
wang yan
a947a4259d Fix quota switch fail to get project size
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-14 22:32:32 +08:00