Commit Graph

604 Commits

Author SHA1 Message Date
stonezdj
15898f2069 Avoid to create duplicated immutable tag rules in the same project
Fix #9681, add constraint on immutable_tag_rule and catch the error

Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-11-15 14:46:23 +08:00
Wang Yan
88773436c9
Merge pull request #9865 from wy65701436/quota-event
add quota exceed event imple
2019-11-15 11:37:19 +08:00
Wang Yan
4bec9bbfc6
Merge pull request #9875 from wy65701436/middleware-policy-checker
enable policy checker in response handler
2019-11-14 18:31:50 +08:00
wang yan
a39e1a2a34 enable policy checker in response handler
Signed-off-by: wang yan <wangyan@vmware.com>
2019-11-14 15:39:29 +08:00
He Weiwei
5bd1cfdbf2 fix(robot,project-member): check owner of member, robot when update, delete
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-11-14 07:00:44 +00:00
Wang Yan
29be93725d
Merge pull request #9860 from reasonerjt/authproxy-case-sensitive-master
Authproxy case sensitive master
2019-11-14 14:03:53 +08:00
Daniel Jiang
6f0b4a139a
Merge pull request #9838 from stonezdj/fix_review
Fix review comments on PR9749
2019-11-14 13:12:56 +08:00
Wang Yan
10850a06d8
Merge pull request #9859 from ywk253100/191113_subresource_1.10
Refine the implementation of replication execution API
2019-11-14 11:30:10 +08:00
wang yan
f8390c5ec1 add quota exceed event imple
Signed-off-by: wang yan <wangyan@vmware.com>
2019-11-14 10:27:18 +08:00
stonezdj
a3c298e9fd Refactor immutable tag rule
Change implementation
Fix some nil pointer issue

Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-11-13 19:09:31 +08:00
Daniel Jiang
8933ab8074 Add configuration "case sensitive" to HTTP auth proxy
This commit make case sensitivity configurable when the authentication
backend is auth proxy.
When the "http_authproxy_case_sensitive" is set to false, the name of
user/group will be converted to lower-case when onboarded to Harbor, so
as long as the authentication is successful there's no difference regardless
upper or lower case is used.  It will be mapped to one entry in Harbor's
User/Group table.
Similar to auth_mode, there is limitation that once there are users
onboarded to Harbor's DB this attribute is not configurable.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-11-13 15:00:05 +08:00
Wenkai Yin
54c5811974 Update the test cases of user API
Update the test cases of user API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-11-13 14:51:29 +08:00
Wenkai Yin
05ffb7a3c5 Refine the implementation of replication execution API
Remove the duplicated code in replication execution API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-11-13 14:26:57 +08:00
stonezdj
4d822e0a19 Fix review comments on PR9749
Fix review comments on PR9749
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-11-12 13:05:11 +08:00
wang yan
c6fecf75d8 update immutable tag error message format
Signed-off-by: wang yan <wangyan@vmware.com>
2019-11-12 12:51:17 +08:00
Wang Yan
407417ce7b
Merge pull request #9810 from stonezdj/bug9479
Populate group from auth provider to Harbor when user login
2019-11-11 19:52:31 +08:00
stonezdj
0c011ae717 Populate group from auth provider to Harbor DB when user login
Fix #9749, change include LDAP auth, OIDC auth, HTTP auth

Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-11-11 14:41:35 +08:00
Daniel Jiang
64dc5122e6 Add role list in project response
This commit fixes #9771

It compares the roles to return the one with highest permission in the
response of `GET /api/projects`.
In addition to that, it adds the role list to the response, because a
user can have multiple roles in a project.
It also removes the togglable attribute as it's not used anywhere.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-11-11 14:03:50 +08:00
wang yan
415bdfa61f Disable policy check when pull without bearer token
This commit is to fix https://github.com/goharbor/harbor/issues/9780.
To align with OCI spec, when a docker pull request without bearer token in header comes in, Harbor should not intecepte it(return a 412 if check fail)
when the policy check is enabled. As the 401 is expected by the docker/caller, and then to ask token service which url is in the 401 header.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-11-08 13:59:30 +08:00
Wenkai Yin(尹文开)
d60979cdd0
Merge pull request #9762 from steven-zou/fix/add_status_2_scan-call_metrics
add status data in the scan all metrics
2019-11-08 10:30:25 +08:00
Steven Zou
a1d8c01cea add status data in the scan all metrics
Signed-off-by: Steven Zou <szou@vmware.com>

Signed-off-by: Steven Zou <szou@vmware.com>
2019-11-07 14:28:11 +08:00
Daniel Jiang
06e4e124d8
Refine request handle process (#9760)
* Refine request handle process

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-11-07 13:02:17 +08:00
Steven Zou
ee31418e8e revoke scan permission from the developer role
Signed-off-by: Steven Zou <szou@vmware.com>
2019-11-06 17:57:48 +08:00
Steven Zou
ebc5d2482b do improvements to the scan all job
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job

fix #9705
fix #9722
fix #9670

Signed-off-by: Steven Zou <szou@vmware.com>
2019-11-05 15:12:07 +08:00
Steven Zou
a928928a43
Merge pull request #9686 from heww/fix-project-severity-mapping
fix(policy-checker): add func to transform project severity to vuln.Severity
2019-11-01 11:12:13 +08:00
Daniel Jiang
f2beee16b1
Merge pull request #9673 from steven-zou/fix/issue_#9668_status_conflicts
return more clear error message for scan related API
2019-11-01 11:08:43 +08:00
Wenkai Yin(尹文开)
2101d7125c
Merge pull request #9680 from heww/install-clair-adapter-imporvement
fix(scanner): imporve clair adapter initializing
2019-11-01 10:42:04 +08:00
He Weiwei
ae8931e816 fix(policy-checker): add func to transform project severity to vuln.Severity
The severity saved in db is lowercase but the severities in vuln pkg
begin with upper letter, this fix use func to transform project severity
value from db to vuln.Severity.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-31 14:11:44 +00:00
Steven Zou
eb8ec49f4f add UT cases for the common error pkg
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-31 21:55:27 +08:00
Daniel Jiang
bc65609a10
Merge pull request #9657 from wy65701436/quota-sync-switcher
add a switcher for quota sync on core launch
2019-10-31 19:22:23 +08:00
wang yan
c46d7e856a add a switcher for quota sync on core launch
As the quota sync is default called by harbor-core on every launch, and it will break the launch process if any failure throwed.

1, The commit is to provide an switcher for the system admin to bypass the quota sync.
2, In case Harbor goes into the restarting cycle.

Harbor already provides an internal API to sync quota data, in the failure case,
system admin can launch harbor and call the /api/internal/syncquota to sync quota.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-31 16:17:27 +08:00
He Weiwei
7170485a9b fix(scanner): imporve clair adapter initializing
1. Remove ping action when initialize clair adapter installed by harbor.
2. Remvoe the `IsDefault` property when initialize clair adapter that
make it switch to auto detecting.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-31 08:03:37 +00:00
Steven Zou
afb46188b2 return more clear error message for scan related API
- add a common error pkg to support error with code and AsError check
- replace some errors in scan with coded errors
- fix #9668

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-31 11:35:55 +08:00
wang yan
253e87d186 inject ldflags for harbor compiler and linker
1, replace the UIVERSION file with ldflags, which is generarted by make to inject into the UI core.
2, inject additional ldflags for harbor compiler

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-30 18:31:42 +08:00
Steven Zou
cb8d4d0daf fix the bug of returning errors nothing is updated
- bug details: #9629
- root cause: the preconditions for updating may not be matched

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-29 15:34:34 +08:00
Steven Zou
10c30fde3d
Merge pull request #9618 from steven-zou/fix/bug_#9608
permission grant for scanner related actions are not correctly
2019-10-28 22:12:56 +08:00
Steven Zou
5b2ab34e03 permission grant for scanner related actions are not correctly
- add new endpoint for getting scanner candidates of specified project
- adjust the permission granting functions
- fix #9608

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-28 18:20:47 +08:00
Wang Yan
e0fd4cd609
Merge pull request #9599 from wy65701436/fix-quota-migration
Fix quota sync crash on getting the unknown mainfest
2019-10-28 17:47:46 +08:00
wang yan
fc347fc4eb Fix quota sync crash on getting the unknown mainfest
1, eat the unknown manifest error, and log it. The migration process will not crashed on it.
2, enable to persist DB of sync quota API.
3, add empty project support.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-28 15:17:08 +08:00
He Weiwei
28e0c0693b Upgrade clair adapter to v1.0.0
1. Upgrade clair adapter to v1.0.0.
2. Make the clair adapter which installed by harbor immutable and using internal registry address.
3. Add support to build clair adapter image from binary.
4. Switch to ScannerPull action when make authorization for the scan request.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-26 17:25:36 +00:00
Steven Zou
eba1a01ac2
Merge pull request #9595 from steven-zou/fix/update_scanner_failure
property use_internal_addr can not be updated
2019-10-25 17:39:34 +08:00
Steven Zou
9d0263fc9a property use_internal_addr can not be updated
- pick up `use_internal_addr` in the update API

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-25 15:35:43 +08:00
wang yan
f9996663d8 update immutable rule API
1, unify disable and enable
2, fix update rule error

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-25 14:11:07 +08:00
Steven Zou
868851cc51
Merge pull request #9571 from steven-zou/fix/support_internal_addr_scanner
support using internal registry addr to perform scan
2019-10-24 20:52:27 +08:00
Wang Yan
d18678a48d
Merge pull request #9506 from wy65701436/token-sevice
Enable robot account to support scan pull case
2019-10-24 19:52:33 +08:00
Steven Zou
cb59ba3bbc support using internal registry addr to perform scan
- do changes to the sql schema
- add `UseInternalAddr` and `Immutable` properties to scanner registration
- support multiple authentication type
  - basic
  - bearer token

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-24 18:28:35 +08:00
wang yan
71c769ec97 remvoe bypass to scanner pull
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-24 17:49:20 +08:00
Ziming
eb6708ed76
Merge pull request #8174 from bitsf/refact_replication_info
refact replication info to support special endpoints and credential
2019-10-24 15:34:01 +08:00
Steven Zou
956d9db1d5
Merge pull request #9528 from steven-zou/fix/pluggable_scanner_improvments
improve the scan controlling
2019-10-24 15:17:06 +08:00
Wang Yan
4baa35bc17
Merge pull request #9520 from ywk253100/191022_health_check
Remove the health checker for Clair in health check API
2019-10-24 14:50:01 +08:00
Ziming Zhang
1801bac03d refact replication adapter
Change-Id: Ic28854089b8dcfcbc7e42065df5c19c64d5b85e7
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2019-10-24 12:24:55 +08:00
Steven Zou
7fad103e46 - fix API test cases failures
Signed-off-by: Steven Zou <szou@vmware.com>

- fix scan report dao bug
2019-10-23 20:44:01 +08:00
wang yan
a6ad1b2db8 update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 20:05:51 +08:00
wang yan
2fa85aefca fix per comments
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 18:45:30 +08:00
wang yan
5996189bb0 update per comments and fix govet error
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 18:45:30 +08:00
wang yan
22b4ea0f89 Enable robot account bypass policy check
1, the commit is for internal robot to bypass policy check, like vul and signature checking.
2, add a bool attribute into registry token, decode it in the harbor core and add the status into request context.
3, add a bool attribut for robot API controller, but API will not use it.y

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 18:45:30 +08:00
Ziming Zhang
5419e1a844 refact replication info to support special endpoints and credential
Change-Id: I2f7a51d3aaf57bb6d1942526184f4e62ce3afeab
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2019-10-23 17:19:56 +08:00
Wenkai Yin(尹文开)
bd28ba43c0
Merge pull request #9478 from ywk253100/191018_test
Populate public metadata into the event
2019-10-23 15:21:53 +08:00
Steven Zou
38395e015c fix api test case failure
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-23 13:27:03 +08:00
wang yan
182e557bf2 update query in the immutable delete manifest middleware
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-23 13:26:19 +08:00
Wenkai Yin
9d896d4d72 Remove the health checker for Clair in health check API
As we introduce the pluggable scanner, users can add the external scanners, so we remove the Clair from the health check API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-10-23 12:55:03 +08:00
Steven Zou
962bafb7ce fix go imports order issues
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-23 09:34:47 +08:00
wang yan
3e826c4e80 update query in the immutable delete manifest middleware
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-22 19:37:20 +08:00
Steven Zou
370a364c29 fix code conflict and rebase with master
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-22 18:39:37 +08:00
Steven Zou
dff1ee07fc improve the scan controlling
- add LCM control to the robot account generated for scanning
- improve the scan webhook
- remove reprots when related artifact is deleted
- update report manager/scan controller and other components to support above cases
- add artifact manager/comtroller to list artifacts

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-22 18:24:46 +08:00
wang yan
424f11e697 add immutable match in the repository/tag delete api
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-21 15:53:24 +08:00
Wang Yan
71bb8815bf
Merge pull request #9461 from reasonerjt/rm-validator-cve
Remove validation for item in CVE whitelist
2019-10-21 14:52:08 +08:00
He Weiwei
e254fe3095
fix(permissions): permissions checking for member and quota info (#9490)
1. Only show project member info when has member list permission.
2. Only show quota info when has quota read permission.
3. Add quota read permission for all roles of project.
4. Refactor permission service in portoal.
5. Clear cache when clear session.

Closes #8697

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-21 14:03:52 +08:00
He Weiwei
bf6a14c9ad
feat(role): introduce a limited guest role (#9403)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-20 14:21:28 +08:00
Wenkai Yin(尹文开)
f98196e5ba
Merge pull request #9435 from reasonerjt/oidc-refresh-refine
Update OIDC token refresh process
2019-10-18 19:43:34 +08:00
Wenkai Yin
db7025a504 Populate public metadata into the event
Fixes #9455. Populate the public metadata into the event when doing the replication based on event

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-10-18 17:40:19 +08:00
Daniel Jiang
1a9cebd5e8 Remove validation for item in CVE whitelist
To contain various vulnerabilities in the CVE whitelist, this commit
removes the validation.
Fixes #9242

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-10-18 15:08:32 +08:00
Wenkai Yin(尹文开)
5c5e475da4
Merge pull request #9415 from steven-zou/fix/pluggable_scanner_policy_check
do changes to let the vul policy check compatible with new framework
2019-10-18 09:39:20 +08:00
Wang Yan
51d3134e4f
Merge pull request #9427 from wy65701436/immutable-middleware
add immutable tag middleware
2019-10-17 20:28:34 +08:00
Steven Zou
0f16913635 rebase: resolve the code confilcts with master
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-17 17:42:41 +08:00
wang yan
da02b820ad add immutable tag middleware
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-17 16:58:11 +08:00
Wenkai Yin(尹文开)
97ddff2ac8
Merge pull request #9434 from heww/clair-adapter
build(clair): internal clair adapter when install with clair
2019-10-17 16:06:10 +08:00
He Weiwei
8964a8697a build(clair): internal clair adapter when install with clair
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-17 12:00:51 +08:00
Daniel Jiang
f0cb16cb86 Update OIDC token refresh process
1) Disassociate id token from user session

2) Some OIDC providers do not return id_token in the response of refresh
request:
https://openid.net/specs/openid-connect-core-1_0.html#RefreshTokenResponse
When validating the CLI secret it will not validate the id token,
instead it will check the expiration of the access token, and try to
refresh it.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-10-17 11:26:18 +08:00
Steven Zou
f18afc0a3f do changes to let the vul policy check compatiable with new framework
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-16 23:15:26 +08:00
Wenkai Yin(尹文开)
32a2c41c3b
Merge pull request #9273 from gavinfish/typo
Fix typos in core package
2019-10-16 17:10:35 +08:00
Wenkai Yin(尹文开)
372875ad64
Merge pull request #9393 from wy65701436/immutable-match
add immutable match
2019-10-15 18:51:43 +08:00
stonezdj(Daojun Zhang)
ff04b2c930
Merge pull request #9411 from wy65701436/fix-list-robot
fix list robot account API return an internal error
2019-10-15 17:47:38 +08:00
wang yan
288e4cc193 igonre the duplicate error when to insert project_blobs on quota syncing
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-15 14:14:58 +08:00
wang yan
5e8f7297f5 fix list robot account API return an internal error
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-15 12:29:44 +08:00
wang yan
a3546478eb add immutable match
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-14 16:21:32 +08:00
wang yan
25f638a989 Merge branch 'master' of https://github.com/goharbor/harbor into robot-invisiable 2019-10-14 14:35:45 +08:00
Wang Yan
17ad4aca55
Merge pull request #9343 from ywk253100/191009_bug_fix
Check the existence of project before the next action
2019-10-14 11:27:34 +08:00
Steven Zou
a86afd6ebc Merge branch 'master' into feature/pluggable_scanner_s3_merge 2019-10-12 15:18:06 +08:00
wang yan
3e81bd7f1d add visible attribute to robot account
The commit is to make robot controller could create invisible robot account for internal use

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-12 00:51:48 +08:00
Wang Yan
40d80f82ba
Merge pull request #9368 from reasonerjt/set-cli-secret-api
API for user to set the CLI secret
2019-10-11 18:38:58 +08:00
Wang Yan
d7375992b5
Merge pull request #9363 from wy65701436/robot-refactor
refactor robot api
2019-10-11 18:16:28 +08:00
Steven Zou
9fd8b6306c refactor code to reflect code review comments
- refactor the db schema \
- refactor  permission checking in API handlers \

to follow the latest code/interface changes

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-11 18:07:47 +08:00
wang yan
6f6f113f0f refactor robot api
1, add API controller for robot account, make it callable internally
2, add Manager to handler dao releate operation

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-11 17:26:18 +08:00
Daniel Jiang
53a13e165d API for user to set the CLI secret
This commit replace the API to generate CLI secret with a new API to
update the secret

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-10-11 14:15:08 +08:00
Steven Zou
58afd8e14b [stage3] support pluggable scanner
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces

Signed-off-by: Steven Zou <szou@vmware.com>

- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases

- update robot account making content
- hidden credential in the job log

Commnet scan related API test cases which will be re-activate later
fix #8985

fix the issues found by codacy

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-11 12:53:02 +08:00
wang yan
8c155e0c50 fix quota migration still execute on launch even data sync success
This commit is to fix the issue for the following scenario:
1, user success migrate harbor to v1.9.0 from a previous version
2, add a project, push images into the project.
3, delete images and then to delete the project.
4, re-launch harbor.

After that, it still execute the quota migration as the condition doesn't consider the deleted projects usage.
And in this case, the harbor core crashes with a duplicate sql err, and unable to launch.

[Workaroud]
Clean table of project_blob with: TRUNCATE TABLE project_blob, and re-launch harbor, wait for quota sync success.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-11 01:25:21 +08:00
stonezdj(Daojun Zhang)
a2938c5d78
Merge pull request #9274 from wy65701436/immu-refatctor
refactor immutable dao code to align the new structure under pkg
2019-10-10 10:38:22 +08:00
wang yan
8317100cda continue refactor API
Signed-off-by: wang yan <wangyan@vmware.com>
2019-10-09 23:06:35 +08:00
Wenkai Yin
a61c928e34 Check the existence of project before the next action
Fixes #8234, check the existence of project to avoid the panic

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-10-09 16:54:10 +08:00