Commit Graph

272 Commits

Author SHA1 Message Date
Daniel Jiang
45210f7c40 Fix condition for OIDC security filter (#7645)
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-05-06 16:30:09 +08:00
Daniel Jiang
28871b78ae Remove IndexController
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-05-06 14:04:06 +08:00
Daniel Jiang
6c3df3c8ce Disable CA download when hosted via HTTP
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-05-06 11:14:39 +08:00
Wenkai Yin
12d58370ad
Merge pull request #7503 from ywk253100/190424_stop_execution
Check the task status of execution whose status is running when deleting the policy
2019-04-30 11:28:49 +08:00
wang yan
02c7cbeec2 Fix get log issue of Periodic job
Use the latest error or success execution as the periodic job log

Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-29 15:30:05 +08:00
Wenkai Yin
7e4c227318 Check the task status of execution whose status is running when deleting the policy
Check the task status of execution whose status is running when deleting the policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-29 15:04:41 +08:00
Daniel Jiang
56c7d55c73
Merge pull request #7593 from reasonerjt/ext-url-systeminfo
Add Ext URL to response of systeminfo API
2019-04-29 14:51:40 +08:00
Wenkai Yin
c53d73775a
Merge pull request #7590 from reasonerjt/oidc-wrong-secret-err
Return more details for error in exchange token
2019-04-29 14:22:37 +08:00
Daniel Jiang
02cf75c142 Add Ext URL to response of systeminfo API
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-29 13:31:52 +08:00
wang yan
2b99e148d9 Add gc parameters when to update gc schedule
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-28 18:17:08 +08:00
Wang Yan
c26f655bce
add periodic job UUID to upstream job id and use execution log as the… (#7530)
* add periodic job UUID to upstream job id and use execution log as the periodic log

Signed-off-by: wang yan <wangyan@vmware.com>

* add comments to fix codacy

Signed-off-by: wang yan <wangyan@vmware.com>

* Update code per comments

Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-28 15:09:56 +08:00
Wenkai Yin
7af679af7e
Merge pull request #7567 from reasonerjt/oidc-google-refresh-token
Persist the new token in DB after login
2019-04-28 14:12:25 +08:00
Daniel Jiang
80176cc354 Check whether user is nil in Prepare() of users API (#7507)
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-28 14:06:35 +08:00
Daniel Jiang
15626fcae0 Return more details for error in exchange token
This commit update the response off OIDC callback when there's error in exchange token.
Additionally add comments to clarify that by default 500 error will not
contain any details.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-28 13:41:53 +08:00
Wenkai Yin
2a463016a9 Upgrade the distribution and notary library (#7516)
* Return 404 when the log of task doesn't exist

Return 404 when the log of task doesn't exist

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Upgrade the distribution and notary library

Upgrade the distribution library to 2.7.1, the notary library to 0.6.1

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-28 12:00:26 +08:00
Daniel Jiang
473fed5689 Persist the new token in DB after login
This commit make sure the token is persist to DB after every time after
a user logs in via OIDC provider, to make sure the secret is usable for
the OIDC providers that don't provide refresh token.

It also updates the authorize URL for google to make sure the refresh
token will be returned.

Also some misc refinement included, including add comment to the
OIDC onboarded user, preset the username in onboard dialog.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-27 23:03:59 +08:00
Daniel Jiang
b9f5f1027c
Merge pull request #7504 from reasonerjt/reload-auth-proxy-cert-verify
Update Transport of HTTP cient in auth proxy client
2019-04-26 23:24:33 +08:00
Daniel Jiang
07d15a8553 Update Transport of HTTP cient in auth proxy client
This commit ensures that the TLS config of the HTTP client for auth
proxy is updated when the configuration is changed.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-26 18:39:28 +08:00
Wenkai Yin
6511417ba6
Merge pull request #7495 from stonezdj/const_debts
Replace string with const in metadatalist.go
2019-04-25 17:41:04 +08:00
stonezdj
504eab56c3 Replace string with const in metadatalist.go
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-04-25 17:01:43 +08:00
Wenkai Yin
66087aac82
Merge pull request #7493 from stonezdj/tech_debts
Remove adminserver in sourcecode
2019-04-24 16:24:59 +08:00
Steven Zou
9bd2de3e35
Merge pull request #7452 from steven-zou/fix_issues_for_jobservice
Fix issues for jobservice
2019-04-24 16:15:43 +08:00
Wenkai Yin
d8310cc708 Fix replication bugs (#7470)
1. Only return the event based trigger for local Harbor
2. Valid the trigger pattern and cron string when creating/updating policies
3. Set the schema as "http" if it isn't specified when creating/updating registries

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-23 19:34:29 +08:00
Steven Zou
823d9c04a9
Merge pull request #7497 from wy65701436/fix-replc-500
refine chart clint http response
2019-04-23 19:30:36 +08:00
wang yan
0d563fda9c refine chart clint http response
Chart client eats the http error if not status ok, after refactor, the
real http response will be catched in core api.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-23 18:22:23 +08:00
Steven Zou
69d9a28860
Merge pull request #7482 from wy65701436/chart-upload
Fix chart upload issue on event based
2019-04-23 17:33:08 +08:00
stonezdj(Daojun Zhang)
e4506604e2 fix error message (#7459)
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-04-23 15:53:34 +08:00
stonezdj
d7798a12d2 Remove adminserver in sourcecode
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-04-23 15:05:29 +08:00
wang yan
a3763466b3 Update err message to general information
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-23 10:57:31 +08:00
wang yan
1b4c75af25 Add event into upload ctx
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-23 10:57:31 +08:00
wang yan
df6e0600c9 Fix chart upload issue on event based
Use chart API to load the uploaded chart file to get the name and version

Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-23 10:57:31 +08:00
Steven Zou
3937c8b0dc Merge branch 'master' into fix_issues_for_jobservice 2019-04-22 19:26:51 +08:00
Daniel Jiang
1fdc2e6ba9 Provide API to generate CLI secret
This commit provide an API to allow a user that is onboarded via OIDC
authn update his CLI secret.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-22 13:34:12 +08:00
Steven Zou
8e734407c0 Merge branch 'master' into fix_issues_for_jobservice 2019-04-19 21:15:21 +08:00
Steven Zou
e92164c886
Merge pull request #7442 from ywk253100/190418_replication_bug_fix
Fix bug in replication
2019-04-19 17:04:34 +08:00
stonezdj(Daojun Zhang)
36d13e8243
Merge pull request #7328 from stonezdj/debts
Fix issue 6450 Test LDAP server error without save configuration
2019-04-19 16:51:57 +08:00
Daniel Jiang
6b45b5ef7c
Merge pull request #7451 from reasonerjt/oidc-logout
Skip verifying OIDC token for local user
2019-04-19 14:55:26 +08:00
Steven Zou
f8feaa192e add get scheduled and periodic executions APIs
Signed-off-by: Steven Zou <szou@vmware.com>
2019-04-19 13:54:23 +08:00
Wenkai Yin
cf5cd5902f Fix bug in replication
1. Fix bug when creating the namespace
2. Keep the same logic for hiding access secret
3. Filter only push mode policies for event trigger

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-19 13:34:04 +08:00
Daniel Jiang
5292aea89e Skip verifying OIDC token for local user
If a user does not have OIDC meta data in DB, it means he's not
onboarded via OIDC authn, hence, we should not check the token.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-19 13:07:06 +08:00
Wenkai Yin
059b75e97c
Merge pull request #7392 from reasonerjt/oidc-logout
Handle OIDC user invalidation from OIDC provider.
2019-04-19 12:46:36 +08:00
Daniel Jiang
239b33c5fb Handle OIDC user invalidation from OIDC provider.
Ths commmit ensures that when user's token is invalidated OIDC provider, he
cannot access protected resource in Harbor with the user info in his session.
We share the code path with secret verification b/c the refresh token
can be used only once, so it has to be stored in one place.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-19 01:27:31 +08:00
Steven Zou
1f481e492c Refactor job servcie primary logic to fix related bugs
Signed-off-by: Steven Zou <szou@vmware.com>
2019-04-18 16:02:49 +08:00
stonezdj
41a574e55c Fix issue 6450 Test LDAP server error without save configuration
Signed-off-by: stonezdj <stonezdj@gmail.com>
2019-04-18 14:24:21 +08:00
Wenkai Yin
a5cc228781
Merge pull request #7420 from ywk253100/190417_revert_local_harbor
Update the migration sql
2019-04-17 19:58:31 +08:00
wang yan
ddec7bd645 fix error handlering in job notification
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-17 18:19:01 +08:00
wang yan
e017294f71 merge with master latest
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-17 17:52:39 +08:00
Wenkai Yin
547c2337de Update the migration sql
1. Update the migration sql
2. Rename the ResourceRepository from repository to image

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-17 17:18:03 +08:00
Wang Yan
a6af9e9972
Support well-formatted error returned from the REST APIs. (#6957)
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-17 16:43:06 +08:00
Wenkai Yin
6e0d892963 Support creating project with service account
This commit introduces a solution to workaround the restriction of project creation API: only normal users can create projects

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-17 14:50:32 +08:00
Wenkai Yin
3f7884d9d2 Revert "Add new registry type: LocalHarbor"
This reverts commit 94cacf762a.

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-17 14:50:24 +08:00
Wenkai Yin
94cacf762a Add new registry type: LocalHarbor
The "LocalHarbor" is the type of registry where the replication service is running on

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-16 11:42:36 +08:00
Wenkai Yin
2f1d2257d5 Remove the namespace concept in replication
Update the replication logic to remove the "namespace"

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-16 08:36:42 +08:00
wang yan
7a373c2eed Add event trigger to helm upload/deletion replication
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-15 19:02:33 +08:00
Wenkai Yin
ba038eb883 Support replication all projects in Harbor
Support replication all projects in Harbor

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-13 20:23:19 +08:00
Wenkai Yin
c222f18fa7 Update replication
1. Refine the health check of docker hub
2. Remove the GetNamespace method from adapter interface

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-13 15:20:06 +08:00
Daniel Jiang
f92bc8076d "Skip verify cert" to "verify cert"
This commit tweaks the attribute for auth proxy mode and OIDC auth mode.
To change it from "Skip verify cert" to "verify cert" so they are more
consistent with other modes.
Additionally it removes a workaround in `SearchUser` in auth proxy
authenticator.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-12 23:25:54 +08:00
Wenkai Yin
1d16e18dff Remove "ng" from source code
Remove "ng" from source code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-12 22:42:27 +08:00
cd1989
8ca5e17c58 Correct some typos and do some small adjustments
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-12 21:17:33 +08:00
cd1989
a9fa22269c Check health status when add/update registry
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-12 17:29:05 +08:00
Wenkai Yin
188d66d875
Merge pull request #7350 from ywk253100/190411_bugfix
Fix bug of replication
2019-04-12 08:22:59 +08:00
Wenkai Yin
bc0123662b Fix bug of replication
1. check the disable/enable status before starting the replication
2. process the support_namespace property

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-12 07:59:57 +08:00
Wenkai Yin
1f574e7d10
Merge pull request #7354 from wy65701436/replication_ng_namespace
Add api to get namespaces of registry
2019-04-11 23:44:24 +08:00
wang yan
117c36d52c Add api to get namespaces of registry
To query the namespace of the registry according to its ID.

Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-11 23:17:33 +08:00
Wenkai Yin
a2fcb41b31 Fix bug in ping registry API
Fix bug in ping registry API: accept both ID and other properties

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-11 18:00:44 +08:00
Daniel Jiang
763c5df010 Add UT
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-11 15:30:19 +08:00
Wenkai Yin
b73acde051 Support the migration for scheduled replication rule from previous version of Harbor
Support the migration for scheduled replication rule from previous version of Harbor

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-11 13:14:32 +08:00
Wenkai Yin
5a047a7eb6 Update the adapter interface
Add ConvertResourceMetadata and PrepareForPush methods

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-10 23:37:11 +08:00
Daniel Jiang
0d18e6c82f Update according to comments
For more context see PR #7335

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-10 19:38:12 +08:00
Daniel Jiang
0a2343f542 Support secret for docker CLI
As CLI does not support oauth flow, we'll use secret for help OIDC user
to authenticate via CLI.
Add column to store secret and token, and add code to support
verify/refresh token associates with secret.  Such that when the user is
removed from OIDC provider the secret will no longer work.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-10 19:38:11 +08:00
Daniel Jiang
08e00744be Fix misc bugs for e2e OIDC user onboard process
This commit adjust the code and fix some bugs to make onboard process
work.
Only thing missed is that the UI will need to initiate the redirection,
because the request of onboarding a user was sent via ajax call and didn't
handle the 302.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-10 19:38:11 +08:00
Wenkai Yin
5a65480594 Handle the policy from previous versions
Handle the policy from previous versions

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-09 17:43:07 +08:00
Wenkai Yin
7ff46acd25
Merge pull request #7289 from cd1989/add-registry-ping
Add registry ping API
2019-04-08 14:08:53 +08:00
cd1989
5a2d03593f Add helth check method to registry adapter
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-08 10:03:28 +08:00
cd1989
f71a110bec Add registry ping API
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-06 17:00:52 +08:00
cd1989
07139684ce Wait randomly before registry health checking
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-05 20:46:29 +08:00
cd1989
fe004e1bfc Init replication in core
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-04 21:58:31 +08:00
cd1989
2450dacecb Use policy controller in registry deletion
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-04 21:26:27 +08:00
Wenkai Yin
4116433de8
Merge pull request #7306 from ywk253100/190404_cleanup
Remove the useless replication code
2019-04-04 21:18:04 +08:00
Wenkai Yin
c2f702be2a Remove the useless replication code
This commit removes the useless replication code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-04 20:56:25 +08:00
Wenkai Yin
58a73de3e5
Merge pull request #7299 from ywk253100/190404_sync
Sync with master branch
2019-04-04 17:33:11 +08:00
Wenkai Yin
1c735a7464 Filter the events triggerred by replication
Filter the events triggerred by replication pull

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-04 16:03:38 +08:00
Wenkai Yin
b66b1f341e Merge remote-tracking branch 'upstream/master' into 190404_sync 2019-04-04 14:55:09 +08:00
Wenkai Yin
48f02d0605
Merge pull request #7291 from cd1989/registry-with-empty-credential
Handle registry with empty credential
2019-04-03 21:35:32 +08:00
Yan
da0e20ec60
Add controller to onboard oidc user (#7286)
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-03 20:47:22 +08:00
cd1989
e2c86f8f59 Handle registry with empty credential
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-03 20:33:26 +08:00
cd1989
8968e82675 Allow edit registry description
Signed-off-by: cd1989 <chende@caicloud.io>
2019-04-03 19:47:14 +08:00
wang yan
dcf1d704e6 fix dao UT issue and refine the error of onboard OIDC user
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-03 14:05:18 +08:00
wang yan
41018041f7 remove oidc controller and add more UTs
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-03 09:54:21 +08:00
Yan
0de5999f52 add the controller for ocdi onboard user
Signed-off-by: wang yan <wangyan@vmware.com>
2019-04-03 09:52:22 +08:00
Wenkai Yin
5219073c49 Call Harbor API to delete the images in Harbor adapter
Call Harbor API to delete the images in Harbor adapter to avoid the inconsistent between the different versions of Harbor

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-02 17:02:15 +08:00
Wenkai Yin
4484bca756 Fix replication related issues
1. Add operation property for tasks
2. Add trigger property for executions
3. Update the getting registry info API to allow passing 0 as ID to get the info of local Harbor registry

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-04-02 14:26:17 +08:00
Daniel Jiang
587acd33ad Add callback controller for OIDC
This commit add callback controller to handle the redirection from
successful OIDC authentication.
For E2E case this requires callback controller to kick off onboard
process, which will be covered in subsequent commits.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-04-01 12:35:31 +08:00
Wenkai Yin
71b706e60a Update the replication API
1. Add getting execution by ID API
2. Return registry detail info in listing policies API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-03-30 20:46:02 +08:00
Wenkai Yin
8c7b63bac2
Merge pull request #7248 from ywk253100/190326_event
Add event based trigger and scheduled trigger
2019-03-29 14:58:09 +08:00
Wenkai Yin
4f8e283e8e Add event based trigger and scheduled trigger
This commit implements the event based trigger and scheduled trigger in replilcation

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-03-29 13:48:34 +08:00
Daniel Jiang
9ce98f4acd Add controller to handle oidc login
The controller will redirect user to the OIDC login page based on
configuration.
Additionally this commit add some basic code to wrap `oauth2` package
and `provider` in `go-oidc`, and fixed an issue in UT to make
InMemoryDriver for config management thread-safe.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-03-28 11:29:05 +08:00
Wenkai Yin
0e9bdbc09c
Merge pull request #7225 from ywk253100/190326_structure
Adjust the file structure of replication package
2019-03-27 13:08:05 +08:00
Yan
03709e4ec1
add authn proxy (#7199)
* add authn proxy docker login support

User could use the web hook token issued by k8s api server to login to harbor.
The username should add a specific prefix.

Signed-off-by: wang yan <wangyan@vmware.com>

* update code per review comments

Signed-off-by: wang yan <wangyan@vmware.com>

* Add UT for auth proxy modifier

Signed-off-by: wang yan <wangyan@vmware.com>
2019-03-27 12:37:54 +08:00
Wenkai Yin
017bba8dc1 Merge remote-tracking branch 'upstream/master' into 190327_sync 2019-03-27 11:43:51 +08:00