Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-11-29 13:45:20 +01:00
Code Issues Projects Releases Wiki Activity
12,228 Commits 53 Branches 317 Tags 335 MiB
6c394232b6
Commit Graph

1086 Commits

Author SHA1 Message Date
Wang Yan
bf0b5a3fd0
Merge pull request #8663 from wy65701436/fix-quota-api 
Fix quota switch fail to get project size
 2019-08-15 10:49:49 +08:00
wang yan
a947a4259d Fix quota switch fail to get project size 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-08-14 22:32:32 +08:00
He Weiwei
98e1f68468 feat(configuration,db): connection pool configs for db 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-08-14 14:30:34 +08:00
wang yan
9e0addee55 Enable usage sync when switch quota setting 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-08-14 12:47:12 +08:00
wang yan
76c52c2332 append commit to fix core compile error introduced by pr #8606 
Signed-off-by: wang yan <wangyan@vmware.com>

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-08-14 00:22:55 +08:00
Steven Zou
1adc3a9469
Merge pull request #8606 from ywk253100/190807_stuck 
Fix replication tasks stuck in "InProgress" issue
 2019-08-13 15:59:20 +08:00
stonezdj(Daojun Zhang)
3e0191be5a
Merge pull request #8621 from stonezdj/project_sort 
Sort project by name
 2019-08-13 14:13:29 +08:00
He Weiwei
c1cea42089 feat(quota,middleware): enable or disable quota per project by config 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-08-12 00:02:26 +00:00
peimingming
222c47142a Add chart and scanning event for webhook 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2019-08-11 18:01:07 +08:00
stonezdj
65dc665717 Sort project by name 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-08-09 16:22:55 +08:00
Wang Yan
54a39c7159
Merge pull request #8597 from heww/size-quota 
refactor(quota,middleware): implement size quota by quota interceptor
 2019-08-09 15:44:33 +08:00
He Weiwei
e62c29123d refactor(quota,middleware): implement size quota by quota interceptor 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-08-08 23:55:54 +00:00
Wang Yan
9cbcc93e8a
Merge pull request #8602 from goharbor/webhook-dev-20190807 
Add feature webhook implementation
 2019-08-08 16:01:39 +08:00
Wenkai Yin
8777c07d47 Fix replication tasks stuck in "InProgress" issue 
Fix replication tasks stuck in "InProgress" issue

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-08-08 15:42:42 +08:00
Yann David
6435f32bc5
Prevent duplicated entries 
Signed-off-by: Yann David <davidyann88@gmail.com>
 2019-08-07 13:16:43 -04:00
guanxiatao
e7fafd1941 webhook policy, job, event support 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2019-08-07 20:30:26 +08:00
cd1989
870d7115c4 Refactor code to extract a common task runner 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-08-07 17:14:10 +08:00
cd1989
e2e540233b Use context for concurrency control 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-08-07 17:14:10 +08:00
cd1989
1f541c890c Improve performance for other registry adapters 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-08-07 17:14:10 +08:00
Wenkai Yin(尹文开)
6c0c75743e
Merge pull request #8571 from ywk253100/190806_retention_time 
Populate pull/push time properties to the returning data when listing tags
 2019-08-07 12:41:23 +08:00
Wang Yan
305242e993
Merge pull request #8573 from stonezdj/change_trace_level 
Change trace level of missing configure metadata
 2019-08-07 12:41:00 +08:00
Wenkai Yin
216ef269b3 Populate pull/push time properties to the returning data when listing tags 
Populate pull/push time properties to the returning data when listing tags

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-08-07 11:47:05 +08:00
stonezdj
05f9920e62 Change trace level of missing metadata 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-08-06 14:09:54 +08:00
Daniel Jiang
eec4fc2798 Remove clair notifier 
The way Harbor handles notification is problematic.
It currently triggers rescan, which will cause problem when there are
lot of images in the registry.
Such as #7316
This commit removes the notifier and we need to revisit the notification
to figure out how to map the notification to a particular image if need
the notification mechanism in future.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-08-06 01:58:15 +08:00
stonezdj(Daojun Zhang)
12fb643f0a
Merge pull request #8557 from stonezdj/merge_user_group_roles 
Merge user roles and group roles
 2019-08-05 17:07:35 +08:00
stonezdj
35a49568ce Merge user roles and group roles 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-08-05 15:10:06 +08:00
Steven Zou
97c812a1e8
Merge pull request #8359 from nlowe/bugfix/logging-line-call-outside-repo-root 
Fix logger line() call if built outside of the repo root
 2019-08-05 14:49:06 +08:00
He Weiwei
9778954852 feat(quota,middleware): image count quota support 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-08-01 14:48:59 +08:00
He Weiwei
8cc9314984
feat(helm-chart,quota): count quota support for helm chart (#8439) 
* feat(helm-chart,quota): count quota support for helm chart

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-07-31 16:48:40 +08:00
Wang Yan
0a92e61d97
Merge pull request #8485 from wy65701436/internal-reg-quota 
add internal reg request handler chain
 2019-07-30 20:47:21 +08:00
wang yan
4410cc93f9 add internal reg request handler chain 
this is for internal registry api call, the request should be intercpeted by quota middlerwares, like retag and delete.
Note: The api developer has to know that if the internal registry call in your api, please consider to use
NewRepositoryClientForLocal() to init the repository client, which can handle quota change.

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-07-30 19:39:56 +08:00
Wenkai Yin(尹文开)
9e6b022ce1
Merge pull request #8425 from ywk253100/190726_acr 
Fix #8319, got error when replicating image with Azure container registry
 2019-07-30 15:19:12 +08:00
Wenkai Yin
4dac036013 Fix #8319, got error when replicating image with Azure container registry 
Fix #8319, got error when replicating image with Azure container registry

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-30 12:58:22 +08:00
Daniel Jiang
2211be7a80
Merge pull request #8446 from reasonerjt/group-perm-merge 
Update GetRolesByGroupID
 2019-07-29 19:11:51 +08:00
Daniel Jiang
37b7ab6174 Update GetRolesByGroupID 
This commit fixes #8432
When querying the role of group ID, all matched roles should be returned
instead of the minimal role ID.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-07-29 11:24:35 +08:00
wang yan
a23ff4e448 Update pull time in artifact table for docker image pull 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-07-28 12:30:20 +08:00
Wang Yan
b9ea3731f7
Merge pull request #8350 from wy65701436/blob-flow-dev 
Add size middleware to support quota
 2019-07-26 01:25:40 +08:00
Wang Yan
1dfc47d24e Add size middleware to support quota 
[Add]:
1, size middleware for quota size
2, count middleware for quota artifact count

[Support]:
1, put, patch, mount blob
2, put manifest

[Refactor]:
1, Add handle response for middlerware
2, Remove the modifyResponse for registry proxy
3, Use the custom response writer to recored status

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-07-26 00:28:36 +08:00
He Weiwei
f3a2280033
Merge pull request #8384 from heww/quota-apis 
feat(quota,api): APIs for quotas
 2019-07-25 15:19:46 +08:00
He Weiwei
e625f2aa11 feat(quota,api): APIs for quotas 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-07-25 13:40:26 +08:00
wang yan
4763864dae merge with latest master code with quota feature branch 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-07-24 08:47:05 -07:00
Steven Zou
c44747fd3c merge code from master and fix conflicts 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-07-24 17:27:37 +08:00
Ziming
43c2af9857 map retention with policy (#8313) 
Signed-off-by: Ziming Zhang <zziming@vmware.com>

Implement the API and controller of tag retention
 - API handler
 - retention controller
 - dao
 2019-07-24 17:22:26 +08:00
He Weiwei
ce58c58c01 feat(quota,api): quota support for create project API 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-07-24 01:02:51 +08:00
Yann David
51eb8bc60f
Search for LDAP_MATCHING_RULE_IN_CHAIN groups 
Signed-off-by: Yann David <davidyann88@gmail.com>
 2019-07-23 12:19:56 -04:00
Wenkai Yin
7362fae7cc Implement a common scheduler 
Implement a common scheduler that can be used globally

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-23 17:20:31 +08:00
wang yan
2292954a31 Merge branch 'master' of https://github.com/goharbor/harbor into project-quota-dev 2019-07-22 15:46:09 +08:00
Wang Yan
e8565a4539
Merge pull request #8335 from reasonerjt/add-oidc-ping-api 
Add API to ping OIDC endpoint
 2019-07-22 14:30:24 +08:00
Wang Yan
834e604ec0
Merge pull request #8246 from ninjadq/fix_chart_museum_500_error 
Fix: Internal server error with messy code when chartmuseum not work
 2019-07-22 11:07:55 +08:00
Nathan Lowe
b4e169db26
Fix logger line() call if built outside of the repo root 
If harbor is built (or `go test`'d) in a different folder than the repo
root, the call to common/utils/log/line(...) will panic with an index
out of range runtime error because the separator can't find `harbor/src`
in the path.

Signed-off-by: Nathan Lowe <public@nlowe.me>
 2019-07-21 22:30:17 -04:00
Wenkai Yin
5f1d2bd644 Fix package import cycle issue 
Fix package import cycle issue

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-19 13:50:55 +08:00
He Weiwei
9c9b8d3a6d Merge branch 'master' into project-quota-dev 2019-07-19 10:02:51 +08:00
Daniel Jiang
96e2e0b145 Add API to ping OIDC endpoint 
This commit adds an API to help admin verify the OIDC endpoint is a
valid one.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-07-18 19:32:12 +08:00
stonezdj
13772b859e Fix OnBoardGroup issue 
Signed-off-by: stonezdj <stonezdj@gmail.com>

Fix issue when adding a HTTP user group to a project member, returns HTTP 500 error.
 2019-07-18 19:19:09 +08:00
Steven Zou
746d082e2e Merge branch 'master' into feature/tag_retention 2019-07-18 10:40:49 +08:00
Wenkai Yin(尹文开)
a64e089773
Merge pull request #8210 from stonezdj/http_group_dao2 
Add HTTP group support
 2019-07-17 15:22:36 +08:00
DQ
af58195a29 Fix: Internal server error with messy code when chartmuseum not work 
log err when doesn't get data from chart museum

Signed-off-by: DQ <dengq@vmware.com>
 2019-07-17 15:14:50 +08:00
Ziming Zhang
815901ea33 fix 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
Change-Id: I3f2d3c7f1e32b4983c31c23d9753f04239e3c82f
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-07-16 19:24:40 +08:00
stonezdj
bb2ae7c093 Add HTTP group feature 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-07-16 15:38:46 +08:00
Ziming Zhang
c22c38994a retention api 
Change-Id: I70f2c34d6bb96ecf4cb5359e2b1ab2dbb99fdbf9
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-07-16 15:06:37 +08:00
Wang Yan
8ac6bdbbb0 Add quota workflow for quota 
1, apply count for manifest if it's a new image
2, insert data for artifact and artifact_blob

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-07-16 14:48:05 +08:00
wang yan
f066d986b9 merge with latest master code 2019-07-11 20:21:15 +08:00
Wang Yan
b98ca7bf0b
Merge pull request #8237 from wy65701436/redis-locker 
add redis lock
 2019-07-11 20:10:16 +08:00
wang yan
ef14f0cf35 add redis lock, it will be used to lock digest in the quota scenario 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-07-11 19:24:24 +08:00
Wenkai Yin(尹文开)
3bebf7bc64
Merge pull request #8238 from reasonerjt/project-cve-whitelist 
Enable project level CVE whitelist
 2019-07-10 14:41:01 +08:00
Wang Yan
155b0b0acd
Merge pull request #8175 from heww/quota-manager 
Add manager for quota
 2019-07-10 11:03:57 +08:00
wang yan
6d0271ee5c Merge branch 'master' of https://github.com/goharbor/harbor into project-quota-dev 2019-07-10 10:57:10 +08:00
He Weiwei
41ba410bb2 Manager for quota 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-07-09 13:59:48 +08:00
wang yan
24c3753581 add dao of artifact 
Signed-off-by: wang yan <wangyan@vmware.com>

Add dao for quota

Signed-off-by: He Weiwei <hweiwei@vmware.com>

fix govet

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-07-08 23:42:50 +08:00
Daniel Jiang
8f5f0031c7 Enable project level CVE whitelist 
This commit update the project API to support "reuse_sys_cve_whitelist"
setting in project metadata and "cve_whitelist" in project request.
Also modify the interceptor to support project level CVE whitelist if
the reuse flag is false.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-07-08 18:55:54 +08:00
Daniel Jiang
c296f0ddfb
Merge pull request #8176 from stonezdj/http_group 
Refactor LDAP usergroup
 2019-07-08 09:54:31 +08:00
stonezdj
c0ed55445d Refactor LDAP group 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-07-05 14:44:18 +08:00
Daniel Jiang
88a5572f8e Reload OIDC provider older than 3 seconds 
This commit make sure the OIDC is more actively recreated, to mitigate
the problem in #8177

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-07-04 14:55:34 +08:00
He Weiwei
4fedfa6580 Add dao for quota 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-07-04 11:53:26 +08:00
Daniel Jiang
5d887ad0d8
Merge pull request #8179 from reasonerjt/interceptor-use-whitelist 
Apply CVE white list in interceptor
 2019-07-03 15:12:33 +08:00
Daniel Jiang
bba4b2a6a4 Apply CVE white list in interceptor 
Interceptor will filter the vulnerability in whitelist while calculating
the serverity of an image and determine whether or not to block client
form pulling it.

It will use the system level whitelist in this commit, another commit
will switch to project level whitelist based on setting in a project.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-07-03 14:13:00 +08:00
He Weiwei
720dcc72bd Fix read permission of project member read api 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-07-02 14:40:46 +08:00
Ziming
af548e915e
Merge branch 'master' into replication_gcr_1.9 2019-06-27 11:27:33 +08:00
Steven Zou
5521b7b7ad
Merge pull request #7915 from bitsf/replication_ecr_1.9 
aws driver for replication
 2019-06-27 11:24:54 +08:00
Daniel Jiang
4aca812ff2 API for system level vulnerability whitelist 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-06-26 23:35:40 +08:00
Ziming Zhang
072bdd101b aws driver for replication 
Change-Id: I8792ffce2eaa5975359bb6159a1ba7b85926a925
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-06-25 19:11:27 +08:00
Ziming Zhang
e387c63242 gcr driver for replication 
Change-Id: I5a6626950d3878bfa9726b332e68bee59159269f
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-06-25 18:08:10 +08:00
wang yan
a4b202d656 remove the id in the post body when to create a robot account 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-06-11 10:47:56 +08:00
wang yan
056cfc7e31 Return account id when to issue a robot 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-05-22 10:39:26 +08:00
wang yan
2068732eef add validation for robot account registration 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-05-15 15:03:35 +08:00
Wang Yan
774a9f8d75
Remove unused configure item cfg_expiration (#7744) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-05-09 22:07:18 +08:00
Daniel Jiang
cbbf2ea973 Redirect regular user to OIDC login page (#7717) 
When the auth mode is OIDC, when a user login via Harbor's login form.
If the user does not exist or the user is onboarded via OIDC, he will be
redirected to the OIDC login page.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-05-09 10:53:40 +08:00
Wang Yan
095f7b2ff7
add scan all and gc schedule migration (#7628) 
* add scan all and gc schedule migration

Signed-off-by: wang yan <wangyan@vmware.com>

* Fix gofmt errors

Signed-off-by: wang yan <wangyan@vmware.com>

* Update code according to review comments

Signed-off-by: wang yan <wangyan@vmware.com>

* remove convertschedule return name just return value

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-05-08 19:11:33 +08:00
Daniel Jiang
4118769088 print more sectors of file path in logger 
This would help as we have more and more source files having duplicated
names.
Fixes #7202

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-05-08 15:49:19 +08:00
Daniel Jiang
c16b44d30b Make sure panic is not thrown when refresh token 
Fixes #7695

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-05-07 20:30:07 +08:00
Wenkai Yin
d74624d306 Iterate all paginations when listing projects and repositories (#7660) 
Iterate all paginations when listing projects and repositories

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-05-07 13:34:48 +08:00
Wenkai Yin
e64a71d809
Merge pull request #7594 from wy65701436/fix-gc-log 
Fix get log issue of Periodic job
 2019-04-30 10:19:17 +08:00
He Weiwei
37a4f1c982 Remove push+pull action (#7571) 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-04-29 15:37:10 +08:00
wang yan
02c7cbeec2 Fix get log issue of Periodic job 
Use the latest error or success execution as the periodic job log

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-29 15:30:05 +08:00
Wenkai Yin
c53d73775a
Merge pull request #7590 from reasonerjt/oidc-wrong-secret-err 
Return more details for error in exchange token
 2019-04-29 14:22:37 +08:00
Wang Yan
c26f655bce
add periodic job UUID to upstream job id and use execution log as the… (#7530) 
* add periodic job UUID to upstream job id and use execution log as the periodic log

Signed-off-by: wang yan <wangyan@vmware.com>

* add comments to fix codacy

Signed-off-by: wang yan <wangyan@vmware.com>

* Update code per comments

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-28 15:09:56 +08:00
Wenkai Yin
7af679af7e
Merge pull request #7567 from reasonerjt/oidc-google-refresh-token 
Persist the new token in DB after login
 2019-04-28 14:12:25 +08:00
Daniel Jiang
15626fcae0 Return more details for error in exchange token 
This commit update the response off OIDC callback when there's error in exchange token.
Additionally add comments to clarify that by default 500 error will not
contain any details.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-28 13:41:53 +08:00
Wenkai Yin
2a463016a9 Upgrade the distribution and notary library (#7516) 
* Return 404 when the log of task doesn't exist

Return 404 when the log of task doesn't exist

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Upgrade the distribution and notary library

Upgrade the distribution library to 2.7.1, the notary library to 0.6.1

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-28 12:00:26 +08:00
Daniel Jiang
473fed5689 Persist the new token in DB after login 
This commit make sure the token is persist to DB after every time after
a user logs in via OIDC provider, to make sure the secret is usable for
the OIDC providers that don't provide refresh token.

It also updates the authorize URL for google to make sure the refresh
token will be returned.

Also some misc refinement included, including add comment to the
OIDC onboarded user, preset the username in onboard dialog.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-27 23:03:59 +08:00
Wenkai Yin
6511417ba6
Merge pull request #7495 from stonezdj/const_debts 
Replace string with const in metadatalist.go
 2019-04-25 17:41:04 +08:00
stonezdj
504eab56c3 Replace string with const in metadatalist.go 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-04-25 17:01:43 +08:00
Wenkai Yin
7160e411cc
Merge pull request #7498 from ywk253100/190423_docker_hub 
Support replicate public repositories from Docker Hub
 2019-04-24 17:17:23 +08:00
Wenkai Yin
66087aac82
Merge pull request #7493 from stonezdj/tech_debts 
Remove adminserver in sourcecode
 2019-04-24 16:24:59 +08:00
Steven Zou
9bd2de3e35
Merge pull request #7452 from steven-zou/fix_issues_for_jobservice 
Fix issues for jobservice
 2019-04-24 16:15:43 +08:00
Wenkai Yin
5629bf8546 Support replicate public repositories from Docker Hub 
Support replicate the public repositories from Docker Hub without providing the credential

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-24 16:15:31 +08:00
Steven Zou
9bcbe2907b fix go vet issues in the code 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-24 07:31:37 +08:00
stonezdj
d7798a12d2 Remove adminserver in sourcecode 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-04-23 15:05:29 +08:00
wang yan
1b4c75af25 Add event into upload ctx 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-23 10:57:31 +08:00
wang yan
df6e0600c9 Fix chart upload issue on event based 
Use chart API to load the uploaded chart file to get the name and version

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-23 10:57:31 +08:00
Steven Zou
3937c8b0dc Merge branch 'master' into fix_issues_for_jobservice 2019-04-22 19:26:51 +08:00
Daniel Jiang
1fdc2e6ba9 Provide API to generate CLI secret 
This commit provide an API to allow a user that is onboarded via OIDC
authn update his CLI secret.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-22 13:34:12 +08:00
Steven Zou
8e734407c0 Merge branch 'master' into fix_issues_for_jobservice 2019-04-19 21:15:21 +08:00
stonezdj(Daojun Zhang)
36d13e8243
Merge pull request #7328 from stonezdj/debts 
Fix issue 6450 Test LDAP server error without save configuration
 2019-04-19 16:51:57 +08:00
Steven Zou
f8feaa192e add get scheduled and periodic executions APIs 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-19 13:54:23 +08:00
Wenkai Yin
059b75e97c
Merge pull request #7392 from reasonerjt/oidc-logout 
Handle OIDC user invalidation from OIDC provider.
 2019-04-19 12:46:36 +08:00
Daniel Jiang
239b33c5fb Handle OIDC user invalidation from OIDC provider. 
Ths commmit ensures that when user's token is invalidated OIDC provider, he
cannot access protected resource in Harbor with the user info in his session.
We share the code path with secret verification b/c the refresh token
can be used only once, so it has to be stored in one place.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-19 01:27:31 +08:00
Steven Zou
16f97326ad
Merge pull request #7433 from goharbor/replication_ng 
Merge the replication ng branch to master
 2019-04-18 16:35:45 +08:00
Steven Zou
1f481e492c Refactor job servcie primary logic to fix related bugs 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-18 16:02:49 +08:00
stonezdj
41a574e55c Fix issue 6450 Test LDAP server error without save configuration 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-04-18 14:24:21 +08:00
wang yan
ba76550d14 Disable throw internal error to UI 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-18 00:04:19 +08:00
wang yan
e017294f71 merge with master latest 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-17 17:52:39 +08:00
Wang Yan
a6af9e9972
Support well-formatted error returned from the REST APIs. (#6957) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-17 16:43:06 +08:00
wang yan
7a373c2eed Add event trigger to helm upload/deletion replication 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-15 19:02:33 +08:00
Wenkai Yin
c222f18fa7 Update replication 
1. Refine the health check of docker hub
2. Remove the GetNamespace method from adapter interface

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-13 15:20:06 +08:00
Daniel Jiang
f92bc8076d "Skip verify cert" to "verify cert" 
This commit tweaks the attribute for auth proxy mode and OIDC auth mode.
To change it from "Skip verify cert" to "verify cert" so they are more
consistent with other modes.
Additionally it removes a workaround in `SearchUser` in auth proxy
authenticator.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-12 23:25:54 +08:00
Daniel Jiang
763c5df010 Add UT 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-11 15:30:19 +08:00
Wenkai Yin
b73acde051 Support the migration for scheduled replication rule from previous version of Harbor 
Support the migration for scheduled replication rule from previous version of Harbor

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-11 13:14:32 +08:00
Daniel Jiang
0d18e6c82f Update according to comments 
For more context see PR #7335

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-10 19:38:12 +08:00
Daniel Jiang
0a2343f542 Support secret for docker CLI 
As CLI does not support oauth flow, we'll use secret for help OIDC user
to authenticate via CLI.
Add column to store secret and token, and add code to support
verify/refresh token associates with secret.  Such that when the user is
removed from OIDC provider the secret will no longer work.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-10 19:38:11 +08:00
Daniel Jiang
08e00744be Fix misc bugs for e2e OIDC user onboard process 
This commit adjust the code and fix some bugs to make onboard process
work.
Only thing missed is that the UI will need to initiate the redirection,
because the request of onboarding a user was sent via ajax call and didn't
handle the 302.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-10 19:38:11 +08:00
Wenkai Yin
580674f3da Merge remote-tracking branch 'upstream/master' into 190409_sync 2019-04-09 17:01:09 +08:00
Wenkai Yin
855c0a2a6e
Merge pull request #7194 from stonezdj/remove_error_msg 
Remove error message of saving system setting to db
 2019-04-09 12:02:17 +08:00
Wenkai Yin
d72a53aa0c
Merge pull request #7318 from ywk253100/190408_upgrade 
Upgrade the replication_job table
 2019-04-08 22:43:40 +08:00
Wenkai Yin
4ffa0c3da0 Upgrade the replication_job table 
This commit migrates the replication_job table, add one execution record and one task record for each job

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-08 22:23:53 +08:00
stonezdj
e8ab7156bc Remove error message of saving system setting to db 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-04-08 18:16:18 +08:00
cd1989
5a2d03593f Add helth check method to registry adapter 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-04-08 10:03:28 +08:00
Wenkai Yin
e8fe2aa60c Upgrade the registry and replication policy tables 
Upgrade the registry and replication tables in database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-05 13:25:00 +08:00
Wenkai Yin
4116433de8
Merge pull request #7306 from ywk253100/190404_cleanup 
Remove the useless replication code
 2019-04-04 21:18:04 +08:00
Wenkai Yin
c2f702be2a Remove the useless replication code 
This commit removes the useless replication code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-04 20:56:25 +08:00
Wenkai Yin
b66b1f341e Merge remote-tracking branch 'upstream/master' into 190404_sync 2019-04-04 14:55:09 +08:00
Yan
da0e20ec60
Add controller to onboard oidc user (#7286) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-03 20:47:22 +08:00
wang yan
dcf1d704e6 fix dao UT issue and refine the error of onboard OIDC user 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-03 14:05:18 +08:00
wang yan
41018041f7 remove oidc controller and add more UTs 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-03 09:54:21 +08:00
Yan
0de5999f52 add the controller for ocdi onboard user 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-04-03 09:52:22 +08:00
Wenkai Yin
74efee569e Update the registry client to support pulling public images from docker hub without login 
Only add the authentication info when the username is provided to support pulling public images from docker hub without login

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-01 19:15:07 +08:00
Daniel Jiang
587acd33ad Add callback controller for OIDC 
This commit add callback controller to handle the redirection from
successful OIDC authentication.
For E2E case this requires callback controller to kick off onboard
process, which will be covered in subsequent commits.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-04-01 12:35:31 +08:00
Wenkai Yin
8c7b63bac2
Merge pull request #7248 from ywk253100/190326_event 
Add event based trigger and scheduled trigger
 2019-03-29 14:58:09 +08:00
Wenkai Yin
4f8e283e8e Add event based trigger and scheduled trigger 
This commit implements the event based trigger and scheduled trigger in replilcation

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-29 13:48:34 +08:00
Daniel Jiang
9ce98f4acd Add controller to handle oidc login 
The controller will redirect user to the OIDC login page based on
configuration.
Additionally this commit add some basic code to wrap `oauth2` package
and `provider` in `go-oidc`, and fixed an issue in UT to make
InMemoryDriver for config management thread-safe.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-03-28 11:29:05 +08:00
Yan
03709e4ec1
add authn proxy (#7199) 
* add authn proxy docker login support

User could use the web hook token issued by k8s api server to login to harbor.
The username should add a specific prefix.

Signed-off-by: wang yan <wangyan@vmware.com>

* update code per review comments

Signed-off-by: wang yan <wangyan@vmware.com>

* Add UT for auth proxy modifier

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-03-27 12:37:54 +08:00
Wenkai Yin
017bba8dc1 Merge remote-tracking branch 'upstream/master' into 190327_sync 2019-03-27 11:43:51 +08:00
Daniel Jiang
49aae76205 Onbard settings for OIDC provider (#7204) 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-03-25 12:24:39 +08:00
Yan
8d3946a0e2
Refactor scan all api (#7120) 
* Refactor scan all api

This commit is to let scan all api using admin job to handle schedule
management. After the PR, GC and scan all share unified code path.

Signed-off-by: wang yan <wangyan@vmware.com>

* update admin job api code according to review comments

Signed-off-by: wang yan <wangyan@vmware.com>

* Update test code and comments per review

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-03-22 17:52:21 +08:00
Wenkai Yin
49cf50adb1 Merge remote-tracking branch 'upstream/master' into 190324_sync 
Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-22 15:55:52 +08:00
Meina Zhou
130e132f86 Merge branch 'master' into replication_ng 
Signed-off-by: Meina Zhou <meinaz@vmware.com>
 2019-03-21 14:16:33 +08:00
He Weiwei
79235fffd1 Fix pagination for users and users search apis 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-03-21 13:03:30 +08:00
Wenkai Yin
fb394c2c7a Replicate helm charts 
This commit provides the capability for Harbor to replicate helm charts

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-20 00:35:15 +08:00
stonezdj(Daojun Zhang)
7060747d5b ldap_url and ldap_base_dn not exist in user config (#7115) 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-03-15 15:20:12 +08:00
wang yan
73d68903d6 update robot account return attribute 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-03-14 13:57:50 +08:00
Wenkai Yin
258b22a9a5 Fix bug in replication 
This commit fixes bugs found in the implement of replciation NG

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-10 20:57:59 +08:00
Wenkai Yin
cabef73980 Add Harbor adapter for replication 
Implement the replication adapter for Harbor registry

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-13 21:39:39 +08:00
peimingming
4efad287ce Add execution and hooks 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2019-03-13 09:35:01 +08:00
Wenkai Yin
772367498f Merge remote-tracking branch 'upstream/master' into 190311_sync 2019-03-11 20:34:49 +08:00
Frank Kung
5bd5d59a4f 1. Define ng persist replication policy model. 
2. Add ng replication policy CURD methods.
3. Implement ng policy manger.

Signed-off-by: Frank Kung <kfanjian@gmail.com>
Signed-off-by: 慕薇疯魔 <kfanjian@gmail.com>
 2019-03-11 11:13:10 +08:00
Wenkai Yin
ec2a7f9239 Implement replication operation API 
This commit implements the replication operation related APIs

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-08 10:06:33 +08:00
stonezdj(Daojun Zhang)
f7745baf30
Merge pull request #6599 from stonezdj/pr6161 
Add new parameter ldap_group_membership_attribute (PR#6161)
 2019-03-07 13:26:26 +08:00
De Chen
2bc2a44db8
Merge branch 'replication_ng' into registries-management 2019-03-05 16:22:34 +08:00
cd1989
b00098d492 Add unit tests and fix CI 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-03-05 15:37:36 +08:00
stonezdj
4dfee0c1f0 Remove verify_remote_cert 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-03-05 14:04:10 +08:00
stonezdj
cf134bc80e Add new parameter ldap_group_membership_attribute 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-03-03 10:03:22 +08:00
Daniel Jiang
321874c815 Move Settings of HTTP auth proxy (#7047) 
Previously the settings of HTTP authproxy were set in environment
variable.
This commit move them to the configuration API

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-03-01 14:11:14 +08:00
cd1989
8732a20709 Rewrite registry manager with new interface 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-02-27 11:54:04 +08:00
cd1989
6bdf3053a7 Implement registries manager 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-02-27 11:54:04 +08:00
Wenkai Yin
95888b3dc2
Merge branch 'replication_ng' into 190130_transfer_repo 2019-02-27 11:00:42 +08:00
wang yan
91aa67a541 Update expiration variable name to expiresat/tokenduration 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-02-25 11:55:42 +08:00
wang yan
36a778b482 Update expiration schema to bigint and default unit to minute 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-02-22 18:42:43 +08:00
wang yan
47a09b5891 add expiration of robot account 
This commit is to make the expiration of robot account configurable

1, The expiration could be set by system admin in the configuation page or
by /api/config with robot_token_expiration=60, the default value is 30 days.
2, The expiration could be shown in the robot account infor both on UI and API.

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-02-22 18:42:34 +08:00
stonezdj(Daojun Zhang)
4cb49e5388
Merge pull request #6963 from stonezdj/remove_container 
Remove everything of adminserver container
 2019-02-22 18:27:43 +08:00
stonezdj
0cba36d79f Remove everything of adminserver 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-02-22 16:34:39 +08:00
Nguyen Quang Huy
eda6c47b3e add signoff for DCO gate (#6981) 
Some variable name, function name is colliding with builtin function.

Signed-off-by: Nguyen Quang Huy <huynq0911@gmail.com>
 2019-02-22 15:00:18 +08:00
Daniel Jiang
321adc8362
Merge pull request #6941 from ywk253100/190213_replication_policy 
Fix #6698: cannot create a same name replication policy after deleting it
 2019-02-21 16:03:55 +08:00
stonezdj
7a5fbf718f Revise code with review comments 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-02-18 15:20:54 +08:00
stonezdj
36e1c13a43 fix ut error in systeminfo_test.go 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-02-18 14:06:19 +08:00
stonezdj
1ae5126bb4 Refactor adminserver stage 3: replace config api and change ut settings 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-02-18 14:06:19 +08:00
Wenkai Yin
f0f2e77fb4 Implement the repository transfer 
This commit implements the Transfer interface for resource repository

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-02-15 17:49:35 +08:00
Wenkai Yin
530ba1d27b Fix #6698 
This commit fixes the issue #6698: cannot create a same name replication policy after deleting it

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-02-15 15:17:48 +08:00
Daniel Jiang
81639e2110
Merge pull request #6865 from wy65701436/remove-token 
Remove the token attribute from robot table
 2019-02-13 19:23:06 +08:00
Yan
e9556a4cec
Add post response for robot account API (#6906) 
This commit is to do:
1, Add post response on creating robot account
2, Lower-case the attribute of response

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-02-13 14:40:04 +08:00
He Weiwei
1c4b9aa346 Protect API using rbac 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-02-01 18:55:06 +08:00
wang yan
5d6a28d73e Remove the token attribute for robot table 
This commit is to remove the token attribute as harbor doesn't store the token in DB.

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-30 23:56:23 +08:00
wang yan
f4f4535304 Fix action and resouce of RBAC change 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-29 17:05:15 +08:00
Daniel Jiang
bf663df0e7
Merge pull request #6820 from wy65701436/robot-service 
Add robot account authn & authz implementation
 2019-01-29 16:08:25 +08:00
He Weiwei
6e95b98108 Standard actions for rbac 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-29 11:59:11 +08:00
He Weiwei
1da0a66fe5
Merge pull request #6781 from heww/user-permissions-api 
Implement api for get current user permissions
 2019-01-29 01:58:51 +08:00
He Weiwei
0ab7c93e16 Replace casbin builtin keyMatch2 with custom match func 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-29 01:26:38 +08:00
wang yan
2d7ea9c383 update codes per review comments 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-28 21:26:06 +08:00
He Weiwei
8b5e68073d Implement api for get current user permissions 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-28 18:06:52 +08:00
Yan
71f37fb820 * Add robot account authn & authz implementation. 
This commit is to add the jwt token service, and do the authn & authz for robot account.

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-28 17:39:57 +08:00
Daniel Jiang
20db0e737b Provide HTTP authenticator 
An HTTP authenticator verifies the credentials by sending a POST request
to an HTTP endpoint.  After successful authentication he will be
onboarded to Harbor's local DB and assigned a role in a project.

This commit provides the initial implementation.
Currently one limitation is that we don't have clear definition about
how we would "search" a user via this HTTP authenticator, a flag for
"alway onboard" is provided to skip the search, otherwise, a user has
to login first before he can be assigned a role in Harbor.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-01-28 15:43:44 +08:00
He Weiwei
3f8e06a8bc Support master role for project member create and update apis (#6780) 
* Support master role for project member create and update apis

Signed-off-by: He Weiwei <hweiwei@vmware.com>

* Fix description for role_id in swagger.yaml

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-23 14:56:23 +08:00
He Weiwei
ae061482ae Add Can method to securty.Context interface (#6779) 
* Add Can method to securty.Context interface

Signed-off-by: He Weiwei <hweiwei@vmware.com>

* Improve mockSecurityContext Can method

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-23 14:32:37 +08:00
wang yan
903e15235e Update validation and error message per comments 2019-01-17 15:33:05 +08:00
wang yan
4cde11892a update the conflict check with DB unique constrain error message 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-17 13:13:55 +08:00
Yan
1af0f3c3b9 Add API implementation of robot account 
Add API implementation of robot account

1. POST /api/project/pid/robots
2, GET /api/project/pid/robots/id?
3, PUT /api/project/pid/robots/id
4, DELETE /api/project/pid/robots/id

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-17 13:13:55 +08:00
He Weiwei
8dab10bbed
Merge pull request #6765 from heww/rename-ram 
Rename ram to rbac
 2019-01-17 11:50:14 +08:00
He Weiwei
bacfe64979 Rename ram to rbac 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-16 18:20:30 +08:00
Wenkai Yin
f8d9653419
Merge pull request #6737 from ywk253100/190109_health_check 
Implement the unified health check API
 2019-01-16 18:14:14 +08:00
He Weiwei
76bee7a9fc
Merge pull request #6710 from heww/security-by-ram 
Implement current security interfaces using ram
 2019-01-16 17:47:13 +08:00
Wenkai Yin
be4455ec1b Implement the unified health check API 
The commit implements an unified health check API for all Harbor services

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-01-16 17:21:04 +08:00
He Weiwei
ebd26c0105 Implement current security interfaces using ram 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-16 16:08:17 +08:00
Steven Zou
464bdf71cd
Merge pull request #6727 from wy65701436/robot-dao 
Add dao of robot account
 2019-01-14 19:34:23 +08:00
wang yan
d349c256e8 add support for query nil 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-11 16:19:42 +08:00
Daniel Jiang
a1d4bfd332
Merge pull request #6344 from reasonerjt/bump-up-golang 
Bump up golang to 1.11.2
 2019-01-11 16:15:59 +08:00
Daniel Jiang
5d59d6fab8 Bump up golang to 1.11.2 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-01-11 14:44:32 +08:00
wang yan
6bd6fbd4ad Add fuzzy match and delete funt per review comments 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-11 14:26:49 +08:00
wang yan
c6ae1388ec Add dao of robot account 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-10 14:51:33 +08:00
Daniel Jiang
80af81154c
Merge pull request #6702 from wy65701436/robot-db-scheme 
Add DB table for robot account
 2019-01-10 14:25:58 +08:00
wang yan
db09f9f101 Update token length and upper case the sql key words 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-09 10:00:54 +08:00
wang yan
362a0638d0 Add DB table for robot account 
This commit is to add DB scheme for robot account and update the db orm releated.

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-01-08 18:46:16 +08:00
Daniel Jiang
b5788f0695
Merge pull request #6671 from heww/ram 
Add ram pkg
 2019-01-08 15:39:36 +08:00
He Weiwei
79f786ecbe Add ram pkg 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-01-04 13:17:13 +08:00
Wenkai Yin
75d45ebd9d
Merge pull request #6547 from cd1989/retag-input-validation 
Validate repo and tag names in retag
 2019-01-03 17:45:44 +08:00
cd1989
c117a23133 Validate repo and tag names in retag 
Signed-off-by: cd1989 <chende@caicloud.io>
 2018-12-24 16:49:39 +08:00
Daniel Jiang
93c0a18b06
Merge pull request #6537 from stonezdj/ref_admin_driver 
Refactor config settings stage2
 2018-12-21 15:12:56 +08:00
stonezdj
2446878f6b Refactor config settings stage2 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2018-12-21 10:59:11 +08:00
Wenkai Yin
b28bca7af4
Merge pull request #6541 from salkin/proxy-transport 
Add support for http proxy in transport
 2018-12-18 15:46:29 +08:00
Niklas Wik
138bc69f0f Add support for http proxy in transport 
Signed-off-by: Niklas Wik <niklas.wik@nokia.com>
 2018-12-17 10:35:27 +02:00
stonezdj(Daojun Zhang)
13511d74ed Refactor config settings encrypt + metadata (#6387) 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2018-12-12 12:14:33 +08:00
Wenkai Yin
f7a28ee2a2 Remove the duplicate http error struct (#6516) 
There are two different types to represent http error in the current code. This commit updates the codes to keep only one.

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2018-12-12 11:51:19 +08:00
cd1989
caf07a96fe Give meaningful messages when retag forbided 
Signed-off-by: cd1989 <chende@caicloud.io>
 2018-12-06 16:25:21 +08:00
Daniel Jiang
ae240df031 Remove the Scan all in-memory marker (#6399) 
Previously there was a in-memory marker to prevent user from frequently
calling the "scan all" API.  This has become problematic in HA
deployment, and is no longer needed after enhancement in jobservice.

This commit removes the marker for "scan all" api, however, we need to
review the mechanism and rework to make it stateless.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-12-02 15:40:50 +08:00
Steven Zou
ec2ad4d0b8
Merge pull request #6093 from cd1989/replication-record-id 
Add op uuid to image replication
 2018-11-30 14:54:43 +08:00
Wenkai Yin
9d5cf57373 Check the existence of name when creating replication rule and fix bugs in testing library (#6381) 
1. Fix #5102 by checking the existence of name when creating/editing replication rule
2. Add unique constraint to the name of replication policy and target
3. Fix bugs of testing library

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2018-11-30 13:32:20 +08:00
peimingming
238dbc0347 Add UT and review comments and issue fix (#6144) 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2018-11-28 17:43:14 +08:00
peimingming
c67fdc40f5 Support store job log in DB (#6144) 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2018-11-28 15:09:29 +08:00
Steven Zou
e6d4c024ee Update README of job service to reflect latest updates 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-08 10:35:12 +08:00
Steven Zou
7b106d06c5 Build logger framework to support configurable loggers/sweepers/getters 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-06 09:31:31 +08:00
Daniel Jiang
39b4d011c7 Not submit scan all job when core container starts 
Fixes #6115

As for the change in migration sql file, in 1.7 we'll switch to
jobservice for scheduling "scan all" job.  To avoid inconsistency,
this item will be reset and user will need to configure the policy again.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-10-25 19:01:52 +08:00
Steven Zou
3b76a960e1
Merge pull request #6039 from stonezdj/refact_5996 
Refactor capacity
 2018-10-24 10:50:11 +08:00
陈德
1ffd9d8fba Add op uuid to image replication 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-10-21 23:55:57 +08:00
Steven Zou
db24cbe25a
Merge pull request #5779 from cd1989/images-retag 
Merge Images retag
 2018-10-19 11:04:48 +08:00
Daniel Jiang
fe2e58e1a0 Ignore duplication error when inserting config 
This commit mitigates the situation when more then one adminserver is
deployed and there may be duplication error when they try to initialize
the configuration to DB.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-10-19 10:23:00 +08:00
陈德
a1b4729aa7 Add more unit tests 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-10-18 00:26:25 +08:00
stonezdj
0278981523 Change admin server to core in jobservice 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2018-10-16 19:23:12 +08:00
stonezdj(Daojun Zhang)
b764033fc9
Merge pull request #6007 from stonezdj/refact_5998 
Change admin server to core in jobservice
 2018-10-15 17:52:24 +08:00
stonezdj
79bac7a64e Change admin server to core in jobservice 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2018-10-15 14:56:18 +08:00
Daniel Jiang
00c8344c13 Remove the local scheduler 
This is no longer needed after moving the "scan all" to job-service.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-10-15 14:14:11 +08:00
Wenkai Yin
83147b1982
Merge pull request #6003 from wy65701436/fix-gc-bug 
Fix gc api issues
 2018-10-11 10:26:38 +08:00
Daniel Jiang
1188bd89b9 Use secure transport to access HTTP endpoint 
In various parts of the code, we used insecure transport in http Client
when we assume the endpoint is http.  This causes complaints form
security scanner.  We should use secure transport in such cases.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-10-10 17:51:02 +08:00
wang yan
a4ad4c7282 Fix gc api issues 
1, filter out the scan all jobs in the gc list.
2, make it able to delete unexecuted scheduler.

Signed-off-by: wang yan <wangyan@vmware.com>
 2018-10-10 15:45:03 +08:00
陈德
b648084d95 Improve code styles and fix after Harbor refactoring 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-10-09 10:49:03 +08:00
陈德
03d5157eaf Updae retag api spec 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-10-08 19:07:23 +08:00
陈德
75f1cdb449 Update swagger file to add retag API 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-10-08 19:07:22 +08:00
陈德
48d2435146 Fix notification event filtered because of user agent 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-10-08 19:07:22 +08:00
陈德
03af3c5936 Add image retag API 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-10-08 19:07:21 +08:00
James Zabala
e09a157dce
Merge pull request #5896 from erks/normalize_ldap_group_dn 
Normalize (make lowercase) ldap_group_dn during onboarding
 2018-10-02 16:03:03 -04:00
Daniel Jiang
b12dc3b5d8 Schedule "scan all" via jobservice 
This commit leverage the jobservice to trigger "scan all" and
gets rid of the local scheduler to make the harbor-core container
stateless.
It keeps using the notifer mechanism to handle the configuration change.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-09-28 15:42:37 +08:00
Steven Zou
8b538cbc0a Return the total count of charts under the project in project API 
- add new interface method to get total count of charts under namespaces by calling get index
- add new field 'chart_count' in project model
- append chart count to the project model in project API

Signed-off-by: Steven Zou <szou@vmware.com>
 2018-09-25 17:56:11 +08:00
Daniel Jiang
0699980924 Add Scan All job to job service (#5934) 
This commit adds the job to scan all images on registry.
It also makes necessary change to Secret based security context, to
job service has higher permission to call the API of core service.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-09-22 13:07:32 +08:00
clouderati
587459df15 Replacing copyright notices with "Copyright Project Harbor Authors". 
Signed-off-by: clouderati <35942204+clouderati@users.noreply.github.com>
 2018-09-19 16:59:36 +00:00
Qian Deng
7873a0312a Rename harbor-ui to harbor-core 
1. Update the nginx.conf
2. Update Makefile
3. Update docker-compose
4. Update image name
5. Rename folder ui to core
6. Change the harbor-ui's package name to core
7. Remove unused static file on harbor-core
8. Remove unused code for harbor-portal

Signed-off-by: Qian Deng <dengq@vmware.com>
 2018-09-19 16:35:13 +08:00
Yan
29ca31cf6c
Update gc api to fix issues found by UI implemention (#5920) 
This commit is to update gc api to fix issues found by UI implemention:
1, Return json format of gc schedule
2, Unify capital and small letter
3，Return gc records by desc

Signed-off-by: wang yan <wangyan@vmware.com>
 2018-09-19 14:36:47 +08:00
Touch Ungboriboonpisal
e256547411 Normalize (make lowercase) the ldap group dn when onboarding 
Fixes #5895

Signed-off-by: Touch Ungboriboonpisal <tungbori@zynga.com>
 2018-09-18 13:37:35 -07:00
Wenkai Yin
dfcd6f044d
Merge pull request #5888 from steven-zou/mark_labels_to_chart 
Add API to support marking labels to charts
 2018-09-14 15:09:46 +08:00
Steven Zou
7b8fe27c22 Add API to support marking labels to charts 
- add related chart label API entries
- extract label related functionalities to a separate manager interface
- add a base controller for label related actions
- add related UT cases

Signed-off-by: Steven Zou <szou@vmware.com>
 2018-09-14 13:27:50 +08:00
Wenkai Yin
89893779fb Support configuring sslmode for the connection of database (#5861) 
The sslmode of the connection with postgresql is hardcoded as "disable" currently, this commit expose it as an environment variable so that users can configure it

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2018-09-14 13:05:05 +08:00
Wenkai Yin
1f195c2b5f
Merge pull request #5840 from Colstuwjx/fix-tcp-probe 
Fix `TestTCPConn` break issue.
 2018-09-10 14:46:04 +08:00
Daniel Jiang
cd31cbf892
Merge pull request #5828 from stonezdj/ldap_caseinsense 
LDAP group DN should be case insensitively
 2018-09-07 10:48:31 +08:00
Colstuwjx
e49a9de2f4 Fix TestTCPConn break issue. 
Signed-off-by: Colstuwjx <Colstuwjx@gmail.com>
 2018-09-06 14:58:04 +08:00
stonezdj
9dca49ba6e LDAP group DN should be case insensitive 
Fix issue #5776, LDAP servers are case insensitive. because only LDAP
group DN is used to compare/equal operation, lowercase all LDAP group DN
when retrieves it from LDAP server, and lowercase them before save in DB

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2018-09-06 11:33:05 +08:00
Wenkai Yin
5427c0064c
Merge pull request #5731 from Colstuwjx/fix-log-test 
Fix logger test case, add SetSkipLine func.
 2018-09-06 08:17:17 +08:00
Colstuwjx
bab203c0f4 Fix logger test case. 
Signed-off-by: Colstuwjx <Colstuwjx@gmail.com>
 2018-09-05 19:25:17 +08:00
陈德
0582db9a82 Apply consistent format for comments 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-09-05 16:16:31 +08:00
Wenkai Yin
49bb5cfafb Test TCP connection before upgrading database schema 
This commit moves the database schema upgrading after database initialization. The init will test TCP connection.

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2018-09-05 02:07:47 +08:00
陈德
6eb972c383 Add pull scope to post/put/patch method 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-09-03 11:12:11 +08:00
陈德
a59af8ce82 fix gofmt 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-08-30 14:11:18 +08:00
陈德
666bd692fe Support repo list sorting 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-08-30 10:56:50 +08:00
wang yan
aab761ac8a Fix gofmt check results 
Signed-off-by: wang yan <wangyan@vmware.com>
 2018-08-29 11:50:00 +08:00
Yan
fca2bb3a6b
Fix misspell checking results (#5749) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2018-08-29 10:25:42 +08:00
Daniel Jiang
dcf4e2ee78 Update import path in go code 
vmware -> goharbor

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-08-23 17:50:53 +08:00
wangyan
9a95f14918 Cherry-pick -- Fix security issue found by gas 2018-08-03 01:16:53 -07:00
Daniel Jiang
6062bf279b Set default creation_time and update_time at model 
This commit set the default value of creation_time and update_time to
data objects by adding `orm:add_now` annotations.
 2018-07-31 12:56:14 +08:00
stonezdj
9b209858f4 Ldap_group_admin_dn can not updated via rest 2018-07-24 17:47:57 +08:00
silenceshell
7745b79b2e var name should not be error (#5332) 
Rename the variable names from "error" to "err"
 2018-07-24 11:33:21 +08:00
Steven Zou
bb380e6dbc
Merge pull request #5314 from steven-zou/chart_repo_supporting 
Refactor chart API endpoints
 2018-07-20 20:43:55 +08:00
Steven Zou
0227a1315a Keep the chart server related configurations in adminserver 
append chart server related config options to the supporting list of adminserver
provide chart server related config access method in the API layer
update prepare script and ui env template file to enable cache driver config for chart server API
append flag info in the systeminfo API to indicate if chart server is deployed with Harbor
refactor the response rewriting logic to return structual error object
add api init method to initilizing objects required in API handlers
chage owner of the storage folder
update offline/online package scripts in Harbor-Util.robot
 2018-07-20 19:40:33 +08:00
Yan
efdb57548f
add admin job api (#5344) 
It supports Harbor admin to trigger job either manual or
schedule. The job will be populated to job service to execute. 
The api includes:
1. POST /api/system/gc
2, GET /api/system/gc/:id 
3, GET /api/system/gc/:id/log
4, PUT/GET/POST /api/system/gc/schedule
 2018-07-20 19:22:37 +08:00
stonezdj
f5e82f75a7 Add SafeCast function and set default value for some sytem configure 
Add SafeCastString, SafeCastInt, SafeCastFloat64, SafeCastBool function to check
the type matched and avoid panic in runtime

Add default value to configure settings to avoid cannot save configure
issue
 2018-07-17 17:00:06 +08:00
Yan
9e65499c10
Add garbage collection job implemention, this job could (#5268) 
be triggered by manual and schedule. It calls registrtctl
to do the GC job, and log the output.
 2018-07-16 18:08:40 +08:00
Yan
d5b85a6748
Add the registry controller httpserver, it's responsible for controlling (#5265) 
docker regsitry. This version has the API to call regsitry GC with jobservice
secret. Seprates it into a standalone container as do not want to invoke two
processes in one container.

It needs to mount the registry storage into this container in order to do GC,
and needs to copy the registry binary into it.
 2018-07-16 16:50:28 +08:00
Daniel Jiang
0d6ea995e1 Let adminserver initialise the DB schema. 
This commit make update to remove the code from ui container to init the
DB schema.  As UI has dependency on admin server, so it's safe to assume
adminserver has to be ready first.  Regardless the setting of the config
store of admin server, it will try to access and intialize the schema of
database.
 2018-07-13 17:32:17 +08:00
stonezdj
62acdb14f3 Add settings to define admin with LDAP group DN 2018-07-05 14:46:44 +08:00
Deng, Qian
edbe2fe620 Update migrator to 1 6 0 
1. Add new alembic_pg folder for postgres
2. Add migration file for 1.6.0
3. Update version to 1.6.0
4. update migrator dockerfile
 2018-07-02 21:23:47 +08:00
Daniel Jiang
1afb09b6cf
Merge pull request #5194 from ywk253100/180626_search 
Fix bug in search API
 2018-06-29 21:13:09 +08:00
Daniel Jiang
aef3213dfa
Merge pull request #5190 from stonezdj/reload_config 
Fix issue that harbor tile can not save customized settings
 2018-06-29 13:04:36 +08:00
Daniel Jiang
c9b1962b1e Initialise Harbor DB schema in Harbor UI/adminserver container 
This commit fixes #5040, the harbor-db image will only contain empty
databases, and harbor ui container will use migrate tool to run initial
SQL scripts to do initialization.  This is helpful for the case to
configure Harbor against external DB or DBaaS like RDS for HA deployment
However, this change will results some confusion as there are two tables
to track schema versions have been using alembic for migration, for this
release we'll try to use alembic to mock a `migration` table during
upgrade so the migrator will be bypassed, in future we'll consider to
consolidate to the golang based migrator.
Another issue is that the UI and adminserver containers will access DB
after start up in different congurations, can't ensure the sequence, so
both of them will try to update the schema when started up.
 2018-06-28 16:22:53 +08:00
stonezdj
72e9b22e10 Fix issue that harbor tile can not save customized settings 2018-06-28 16:20:10 +08:00
Wenkai Yin
982760a132 Fix bug in search API 
Refactor the logic of search API to fix bug mentioned in issue #5156 and #3838
 2018-06-27 12:51:08 +08:00
stonezdj
d6a4d79a03 Handle Invalid syntax and not found error 2018-06-20 14:27:29 +08:00
Daniel Jiang
255a6d6f95
Merge pull request #5070 from ywk253100/180601_label_fuzzy_match 
Fix #4742: fuzzy match label name
 2018-06-12 14:39:35 +08:00
Wenkai Yin 79628
0c56493fb6 Soft delete label 
Modify the deletion of label to soft deletion, in this way the names of deleted labels referenced by replication rules can be shown to users
 2018-06-07 17:14:12 +08:00
Wenkai Yin 79628
07c092c9be Fix #4742 
This commit provides the support of fuzzy matching for label name when listing labels
 2018-06-01 07:38:21 +08:00
Daniel Jiang
2f4950b80c
Merge pull request #5034 from ywk253100/180524_relativeurl 
Fix replication issue when the remote registry enables relativeurls
 2018-05-25 19:33:29 +08:00
Wenkai Yin 79628
29a4a3335e Fix replication issue when the remote registry enables relativeurls 
The location header returned by the remote registry contains no scheme and host parts if "relativeurls" is enabled,
this commit fix it by adding them at the beginning of location.
 2018-05-24 15:34:31 +08:00
Wenkai Yin 79628
76274dbf84 Update change password API 
Modify the changing password API to support that admin user can change the password of normal users without old password
 2018-05-22 19:02:20 +08:00
stone
d3930ae17c Put user info into session (#4885) 
Fix the following issues.
1) GroupList is not found in SecurityContext user info
2) Retrieve multiple memberof information from LDAP.
3) If user is in two groups with guest, administrator role separately, display the max privilege role.
 2018-05-17 16:23:51 +08:00
stonezdj
b8a48d0326 Update security context for assign role to project group member 
The project list will contain all public projects, user is a member of this project, or user is in the group which is a member of this projects.
Change the behaviour of user roles, if the user is not a member of this project, then return the user's groups role of current project
 2018-05-03 16:49:16 +08:00
Yan
ae257433cc
Fully migrate harbor db to postgresql (#4689) 
* Merge harbor db to postgres
 2018-04-27 02:27:12 -07:00
Wenkai Yin
f77e4167ac
Merge pull request #4802 from ywk253100/180427_label_db 
Modify unique constraint of table harbor_label
 2018-04-27 17:10:05 +08:00
Wenkai Yin
73babbf1ab Modify unique constraint of table harbor_label 
Add unique constraint to column name, scope and project_id  of table harbor_label to make creating same name labels under different projects valid
 2017-12-19 22:15:56 +08:00
Tan Jiang
93c448d91b Fix mis-interpretation of severity in Clair 
Currently "Critical" vulnerablity is treated as "Unknown" in Harbor.
This commit provides a quickfix that it will be interpret as "High".  In
future, we should consider introduce "Critical" and enable UI to handle
it to be more consistent with CVSS spec.
 2018-04-25 10:31:12 +08:00
Tan Jiang
1fc4142e1a Do not call chown to config files 
This commit fixes a recently discovered issue on Kubernetes #4496
It make necessary to avoid calling `chown` to config files during the
bootstrap of the containers.
 2018-04-20 13:44:21 +08:00
Daniel Jiang
f8d577994f
Merge pull request #4719 from reasonerjt/fix-db-broken-pipe 
Fix intermittent `broken pipe` issue in log
 2018-04-19 10:23:59 +08:00
Tan Jiang
7fa8261661 Fix intermittent broken pipe issue in log 
This commit fixes #4713, by adopting the suggested fix in:
https://github.com/go-sql-driver/mysql/issues/529
When creating the DB instance in orm, call `SetConnMaxLifetime()`
 2018-04-18 17:39:13 +08:00
Wenkai Yin 79628
f6bd2f245d Fix bug #4688 
Fix bug: the user can push images although he have no permisson by checking empty value before assign permissions.
 2018-04-18 12:03:06 +08:00
stonezdj
7e57c685ac Add project member search by name 
Previous implementation contains the search user by name feature. This implementation can search user and user group by name.
 2018-04-16 18:38:10 +08:00
stonezdj
de49165427 Refactor project member API 
1) Remove the previous /api/projects/?:project_id/members/:userid
    2) Move the /api/projects/:project_id/projectmembers/?:pmid to
        /api/projects/:project_id/members/?:pmid
    3) Change the project member maintain ui to call new REST API
 2018-04-11 17:49:33 +08:00
Tan Jiang
ff06ec05c3 Store secret in header instead of cookie 2018-04-07 22:02:06 +08:00
Daniel Jiang
b5f52bc3fb
Merge pull request #4586 from reasonerjt/job-context-props 
Read the system properties from scan job context
 2018-04-07 10:23:30 +08:00
Tan Jiang
15580a5e8c Read the system properties from scan job context 2018-04-04 19:58:54 +08:00
stone
df63a73fd4
Merge pull request #4483 from stonezdj/api4assign_role_to_group4 
Add REST API for assign role to group
 2018-04-04 16:19:37 +08:00
stonezdj
9bcfaedc0e Add REST API for assign role to group 2018-04-04 13:39:42 +08:00
Wenkai Yin
79f0ca96bc
Merge pull request #4572 from ywk253100/180402_job 
Hide schedule job when listing replication jobs
 2018-04-03 16:12:10 +08:00
Wenkai Yin
500651a5a1 Hide schedule job when listing replication jobs 2018-04-03 01:11:55 +08:00
Tan Jiang
7aec4d9f21 Add UT and remove useless code. 2018-04-02 21:28:46 +08:00
Wenkai Yin
3436729d52
Merge pull request #4547 from ywk253100/180328_schedule_replication_job 
Move schedule replication job to new jobservice
 2018-03-30 21:03:45 +08:00
Steven Zou
ba91fc2861 Merge master into job_service and fix conflicts 2018-03-30 19:26:04 +08:00
Wenkai Yin
dd40f187ec Move schedule replication job to new jobservice 2018-03-30 17:44:05 +08:00
Steven Zou
d1899c840d Merge branch 'master' into job_service 2018-03-29 23:25:20 +08:00
Tan Jiang
5dd75bb0b0 Trust Root CA of VIC appliance when accessing Admiral 2018-03-29 19:53:41 +08:00
Daniel Jiang
2c2cbd9c52
Merge pull request #4510 from reasonerjt/master 
Add indexes to job tables and bump up schema version.
 2018-03-28 16:58:16 +08:00
Tan Jiang
b6df6cf169 Add indexes to job tables and bump up schema version. 2018-03-28 16:15:54 +08:00
stone
203b1b52bb
Merge pull request #4415 from stonezdj/user_group_and_project_member 
Add DAO for user group and project member
 2018-03-26 15:21:20 +08:00
stonezdj
49d960b060 Add DAO for project member and user group 2018-03-26 14:38:32 +08:00
yixingj
cb64ad96ff Make endpoint configurable 
Move all the endpoint to harbor.cfg
 2018-03-26 10:50:18 +08:00
Wenkai Yin
dd156ca243 Handle replication job status hook 2018-03-24 21:18:58 +08:00
Tan Jiang
be97a91650 Integrate with jobservice webhook 2018-03-27 21:27:52 +08:00
Tan Jiang
c859616e25 fix golint and go vet issue 2018-03-26 22:10:01 +08:00
Tan Jiang
582deea9e7 resolve conflict 2018-03-26 18:11:39 +08:00
Tan Jiang
41ce0891ab Trigger scan job from UI. 2018-03-26 18:07:21 +08:00
Wenkai Yin
38568a1d2c
Merge pull request #4485 from ywk253100/180326_period_job 
Create a job to call UI's replication API to do the period replication job
 2018-03-26 17:31:10 +08:00
Tan Jiang
745b21abbc Merge remote-tracking branch 'upstream/master' into scan-job-migrate 2018-03-26 15:39:42 +08:00
Tan Jiang
381ecc3521 Merge with master 2018-03-26 10:37:17 +08:00
Wenkai Yin
85f357ec6b Delete the mapping relationship between resources and labels when the label is deleted 2018-03-24 02:22:51 +08:00
Wenkai Yin
e63d5a1c06 Create a job to call UI's replication API to do the period replication job 2018-03-23 23:53:15 +08:00
Wenkai Yin
ed08a42e4b Migrate replication job to the new jobservice 2018-03-23 18:36:37 +08:00
Yan
cbcca015b0
add read only mode to stop docker push (#4433) 2018-03-23 03:16:08 -07:00
Wenkai Yin
c6e65d2ded Fuzzy matching repository name in GET repositories API 2018-03-22 13:38:26 +08:00
Tan Jiang
b8ec419c8e Add missed package to fix compilation issue 2018-03-21 16:59:32 +08:00
Tan Jiang
613464bc16 Migrate scan job to job service V1 phase1 2018-03-21 16:25:32 +08:00
Wenkai Yin
838b439560 Implement filter repository and tags by label API 2018-03-21 10:51:06 +08:00
stonezdj
44fc373c6d Add LDAP Group Search Configure Param 2018-03-15 06:16:47 +08:00
Wenkai Yin
36b9c4e458 Implement adding/removing labels to/from repositories and images API 2018-03-12 19:30:05 +08:00
Wenkai Yin
379f113452 Implement label management API 2018-03-09 12:17:27 +08:00
Wenkai Yin
0a8929b85e Do the authentication with CRAM-MD5 when the connection is insecure 2018-03-08 14:21:44 +08:00
stonezdj
4c6d1488bd Add UT 2018-02-09 15:29:08 +08:00
stonezdj
f138067242 Refactor project member 2018-02-09 10:38:51 +08:00
Wenkai Yin
10f56d26fe Change codes to make everything OK after upgrading to beego 1.9.0 2018-02-05 13:07:52 +08:00
Wenkai Yin
9022abfc13 Fix code issues found by Gas 2018-01-29 15:17:03 +08:00
Wenkai Yin
515cac010a
Merge pull request #4071 from ywk253100/180117_policy_pagination 
Add pagination support in listing replication policy API
 2018-01-19 15:19:45 +08:00
Wenkai Yin
611709a7be Add pagination support in listing replication policy API 2018-01-18 15:54:12 +08:00
stone
c815dc01dd
Merge pull request #4043 from reasonerjt/uaa-bugfix 
Read Email from UAA while onboarding user.
 2018-01-18 14:04:35 +08:00
Tan Jiang
d5d913f51d Read Email from UAA while onboarding user. 
Will call the userinfo API of UAA to get user info and generage user
model based on the response.  Also this commit include a change that
whenever the UAA Client is to be used it will update the configuraiton,
this is needed as we enable user to update the configuration of UAA via
UI.
 2018-01-17 10:28:49 +08:00
Wenkai Yin
8cda2d8d65
Merge pull request #4036 from ywk253100/180116_s3 
Propagate registry storage driver name to adminserver and return it in /api/systeminfo
 2018-01-16 18:41:08 +08:00
Qian Deng
5017670d00
Merge pull request #4005 from ninjadq/db_migrate_from_1_3_to_1_4 
Update migration tool for v1.4
 2018-01-16 17:04:54 +08:00
Wenkai Yin
53d5a2256a Propagate registry storage driver name to adminserver and return it in /api/systeminfo 2018-01-16 16:57:28 +08:00
Deng, Qian
b3e65ed71e Update migration tool for v1.4 
1. Update database meta file
2. Add migration file for 1.4
 2018-01-16 15:38:51 +08:00
Wenkai Yin
a1dd8c3bff
Merge pull request #4004 from ywk253100/180111_jobservice 
Provide a mechanism to stop pending and retrying jobs
 2018-01-15 12:55:44 +08:00
Tan Jiang
d6bf0ea11d Remove data generated by dao_test after the test. 2018-01-12 15:56:30 +08:00
Wenkai Yin
4070ed5152 Provide a mechanism to stop pending and retrying jobs 2018-01-12 15:29:20 +08:00
Daniel Jiang
43afd426bb
Merge pull request #3995 from reasonerjt/admin-rename 
Provide API to rename admin
 2018-01-12 13:59:13 +08:00
stone
ec173305a3
Merge pull request #3974 from stonezdj/ldap_ping_timeout 
Setting timeout for ldap ping
 2018-01-12 11:22:27 +08:00
Tan Jiang
a392a8dc29 Provide API to rename admin 
This is to provide a workaround for very corner case that in user's
authentication backend (LDAP, UAA) has a user called "admin" and because
Harbor's super user is hard coded to "admin" it's not possible to login
the "admin" with credentials in LDAP or UAA.

To minimize the impact, we'll provide an internal API for user to update
the super user's username from "admin" to "admin@harbor.local", this API
can be called by "admin" only, and is not reversible.
 2018-01-11 23:01:06 +08:00
stonezdj
c48c7f7b6a Setting timeout for ldap ping 2018-01-10 15:14:30 +08:00
Wenkai Yin
e26b442c9c
Merge pull request #3951 from ywk253100/180104_replicate_interval 
Manual starting replication will be rejected if there are pending/running jobs
 2018-01-10 10:56:45 +08:00
Daniel Jiang
f8af1f275e
Merge pull request #3911 from stonezdj/ldap_search_level 
Ambiguous UI and internal values ldap_scope
 2018-01-08 14:53:55 +08:00
Wenkai Yin
87ce1c84d5 Manual starting replication will be rejected if there are pending/running jobs 2018-01-05 17:05:57 +08:00
stonezdj
26b86984d2 Ambiguous UI and internal values ldap_scope #3764 2018-01-05 15:51:37 +08:00
pfh
13308ce9d8 Merge remote-tracking branch 'upstream/master' into repEnhance 2018-01-05 14:09:03 +08:00
Wenkai Yin
51297cdfd7
Merge pull request #3887 from ywk253100/171227_ssrf 
Fix SSRF security issue #3755 in ping target, email server and LDAP server APIs
 2018-01-04 18:11:47 +08:00
Daniel Jiang
8e5115c832
Merge pull request #3870 from stonezdj/ldap_syncuser2 
Sync user email in ldap #3663
 2018-01-04 13:28:51 +08:00
Wenkai Yin
3448fd9a2d Fix SSRF security issue #3755 in ping target, email server and LDAP server APIs 2018-01-04 12:26:17 +08:00
Tan Jiang
e02de2068a Enable configuring the CA Certificate for UAA 
Enable configuring the path of root cert of UAA in harbor.cfg.  It only
takes effects if the verify_cert is set to "true" If the file does not
exist, the configuration is skipped.
The intention for this commit is to support integration with nested UAA
in PAS or PKS, we don't expect user to manually configure this value,
though he can do it if he wants.
 2018-01-03 16:21:29 +08:00
Wenkai Yin
96a63c56b1 Merge remote-tracking branch 'upstream/master' into 180103_merge 2018-01-03 10:32:03 +08:00
Wenkai Yin
a9d7403bee Update project ID property if needed when updating replication policy 2017-12-27 15:04:26 +08:00
stonezdj
35716dedd3 Sync user email in ldap #3663 2017-12-26 18:53:32 +08:00
stonezdj
9f99d0400c Call EscapeFilter for filter to avoid security issue 2017-12-26 15:34:14 +08:00
Daniel Jiang
94c78b3bee
Merge pull request #3858 from xuri/master 
Simple code and typo fixed.
 2017-12-26 12:06:27 +08:00
Tan Jiang
da20e4f11c Search UAA when adding member to a project. 
1)Enable UAA client to search UAA by calling '/Users' API.
2)Implement 'SearchUser' in UAA auth helper, register it to auth
package.
 2017-12-26 00:25:32 +08:00
Ri Xu
9adccd3723
Simple code and typo fixed. 
Signed-off-by: Ri Xu <xuri.me@gmail.com>
 2017-12-23 20:55:07 +08:00
yixingjia
fa67e11680
Merge pull request #3831 from yixingjia/HA_Clair 
Make Clair DB configurable
 2017-12-21 11:31:26 +08:00
Tan Jiang
12cd733678 Remove useless code from UI router and API 
Some URLs are not used on UI, so they are removed.  And the validation
code of API is removed as we use the security context approach.

fix test issue
 2017-12-20 23:10:38 +08:00
yixingj
f63588855f Make Clair DB configurable 
Make the HOST,PORT,USERNAME,DB configurable for
Clair
 2017-12-20 18:29:50 +08:00
Wenkai Yin
8d62d989a5 Fix bug #4791 
Remove the table join when querying repositories with project name
 2017-12-19 21:47:39 +08:00
Tan Jiang
2ffc58a5d4 Refactor the configuraiton of UAA 
Remove the attribute "uaa_ca_root" from harbor.cfg and introduce
"uaa_verify_cert".  Similar to LDAP settings, this allow user to
explicitly turn of the cert verification against UAA server, such that
the code will work with self-signed certificate.
 2017-12-19 14:42:07 +08:00
Daniel Jiang
62cebbdb5d
Merge pull request #3797 from reasonerjt/uaa-restriction 
Disable user management features when auth mode is UAA.
 2017-12-18 22:47:08 +08:00
Daniel Jiang
cdadc94d0f
Merge pull request #3804 from ywk253100/171215_jobservice 
Print stack trace when recover from panic and print warning message rather than returning an error when updating 0 records
 2017-12-18 16:36:20 +08:00
Tan Jiang
224f75b9a6 Refactor /users API, add more restircation in password reset 
Simplified the code when checking if a user is modiable in different
auth modes.
Also add restriction in password, such that when the auth mode is not DB
auth, only the super user can choose to reset his password.
 2017-12-18 14:32:29 +08:00
Wenkai Yin
260ef561c4 Update the HTTP client for easy use by add more util functions 2017-12-16 06:45:59 +08:00
stonezdj
9393d26fdc Fix ldap ping issue #3653 2017-12-15 14:47:54 +08:00
Wenkai Yin
a736cb7b09 Update the HTTP client according to the comments 2017-12-15 09:40:31 +08:00
Wenkai Yin
b5e7de331e Delete enabled and start_time properties of replication rule 2017-12-15 09:40:31 +08:00
Wenkai Yin
fe10c2e7f5 Create replicator to submit replication job to jobservice 2017-12-15 09:40:31 +08:00
Wenkai Yin
8b4fdfc2cc Add unit tests for replication related methods 2017-12-15 09:40:31 +08:00
Wenkai Yin
a54b7dd4c0 Merge remote-tracking branch 'upstream/master' into 171219_merge 2017-12-15 08:48:57 +08:00
Wenkai Yin
43489c2b67 Print stack trace when recover from panic and print warning message rather than returning an error when updating 0 records 2017-12-14 13:48:45 +08:00
stone
cbd1431333
Merge pull request #3726 from stonezdj/ldap_refactor2 
Refactor LDAP code

Changes include:
1. Use session to manage the lifecycle of LDAP connections
2. Abstract common AuthenticateHelper interface for db_auth, ldap_auth, uaa_auth
 2017-12-13 16:21:20 +08:00
stonezdj
ec67974104 Refactor ldap 
Changes include:

1. Use Session to manage the lifecycle of ldap connections
2. Abstract common AuthenticateHelper interface for db_auth, ldap_auth,
uaa_auth mode
 2017-12-13 14:57:04 +08:00
Wenkai Yin
665a54edc3 Merge remote-tracking branch 'upstream/master' into 171213_merge 2017-12-13 13:40:24 +08:00
yixingj
9b03c93afd Add database driver for Harbor configurations 
1>Add a new database driver for configurations
2> change the current default driver from json
to database
 2017-12-06 13:06:54 +08:00
Wenkai Yin
594d213630 Publish replication notification for manual, scheduel and immediate trigger 2017-12-04 15:07:30 +08:00
Daniel Jiang
d13321f2b5
Support getting user info via token in UAA Client (#3686) 2017-11-27 18:13:36 +08:00
Wenkai Yin
6b0ee138e5 Implement immediate trigger and the methods of WatchList 2017-11-27 14:23:21 +08:00
stonezdj
16243cfbbc Add LDAP remote certifcate validation 
push test

Add unit test for ldap verify cert

remove common.VerifyRemoteCert

Update code with PR review comments

Add change ldaps config and add UT testcase for TLS feature

add ldap verfiy cert checkbox about #3513

Draft harbor ova install guide

Search and import ldap user when add project members

Add unit test case for SearchAndImportUser

ova guide

Add ova install guide

Add ova install guide 2

Add ova install guide 3

Call ValidateLdapConf before search ldap

trim space in username

Remove leading space in openLdap username

Remove doc change in this branch

Update unit test for ldap search and import user

Add test case about ldap verify cert checkbox

Modify ldap testcase
 2017-11-24 12:41:51 +08:00
Wenkai Yin
31cf6c078e Implement replication policy manager 2017-11-16 10:55:03 +08:00
Steven Zou
c2e0c8d1f2 Define the related interfaces for triggers and core controllers of replication service 2017-11-10 15:06:24 +08:00
reasonerjt
19a13e8575 Deprivilege harbor-ui harbor-jobservice harbor-adminserver 
Use non-root user to run the service within these docker images, and provide HEALTHCHECK
mechanism.
 2017-11-09 03:09:09 -08:00
Wenkai Yin
149b628292 update 2017-11-09 16:20:56 +08:00
Wenkai Yin
5cef58baa1 update according to the comments 2017-11-08 17:53:41 +08:00
Daniel Jiang
8dfe5f0bfc
Merge pull request #3536 from ywk253100/171102_fail_earlier 
Fail earlier when found database schema dismatch
 2017-11-07 15:01:14 +08:00
Wenkai Yin
5293a9287b Fail earlier when found database schema dismatch 2017-11-07 13:07:56 +08:00
Tan Jiang
512384722a Make the internal URL of UI and JobService configurable 2017-11-03 20:43:25 +08:00
Wenkai Yin
51d5df0849 Update replication policy API to support trigger and filter 2017-11-02 14:59:26 +08:00
Steven Zou
87d966e369
Merge pull request #3510 from steven-zou/master 
Update the alternate policy and corresponding task to support byweekly
 2017-11-01 21:51:04 -05:00
Steven Zou
cee0bcec22 Update the alternate policy and corresponding task to support by weekly besides daily 2017-11-01 13:55:56 +08:00
Wenkai Yin
0ddca31355 Add column id to table project_metadagta as the primary key 2017-10-30 17:37:25 +08:00
Wenkai Yin
5b2ececae8 Merge pull request #3436 from ywk253100/171020_meta_api 
Add project metadata API
 2017-10-27 05:16:50 -05:00
Wenkai Yin
c355034c14 Add project metadata API 
Project metadata API can be used to integrated with project management
service which can not provide all metadatas needed by Harbor.
 2017-10-27 17:05:15 +08:00
Daniel Jiang
d8634290e8 Merge pull request #3420 from reasonerjt/master 
Add Unit test cases for Clair Client.
 2017-10-23 12:18:05 +08:00
Tan Jiang
b925569767 Add Unit test cases for Clair Client. 2017-10-22 21:54:04 +08:00
Wenkai Yin
2156750b04 Move certificate verification to target level 
The certificate verification is on system level before this commit. Moving it
to target level makes the configuration more flexible for different targets.
 2017-10-20 15:36:56 +08:00
Wenkai Yin
66b2d0d3f3 Apply project level policies to standalone Harbor 
The following features are only enabled in integration mode, this commit moves
these to standalone Harbor:
 - Content trust policy: only signed images can be pulled
 - Vulnerability policy: only images whose severity is below the threshold can be pulled
 - Automatic scan policy: automatic scan pushed images
 2017-10-19 17:33:28 +08:00
Tan Jiang
eab6b43d99 Make the root CA certificate of UAA should be configurable 2017-10-16 17:40:29 +08:00
Tan Jiang
51286d9baa Provide UAA authenticator for password based authentication. 2017-10-07 00:16:53 +08:00
Wenkai Yin
e495357d98 implement the default project metadata manager 2017-09-28 16:17:51 +08:00
Wenkai Yin
e79334a445 Add interfaces to implement project level policy (#3271) 
* add interfaces to implement project level policy
 2017-09-26 16:41:08 +08:00
Wenkai Yin
dc4f2ece72 readjust package structure 2017-09-20 15:24:19 +08:00
Wenkai Yin
f0946b63cf fix code style issues reported by golint 2017-09-19 17:16:54 +08:00
Wenkai Yin
8d7644b8b5 Merge pull request #3151 from ywk253100/170830_email_insecure 
Expose the insecure flag for email configuration
 2017-09-15 15:01:30 +08:00
weibaohui
84d66d85fa Correct spelling 
Correct spelling
 2017-09-11 15:13:24 +08:00
Wenkai Yin
923a8d65b1 expose insecure flag in api 2017-09-04 15:10:07 +08:00
Daniel Jiang
f41d2ff436 Merge pull request #3101 from ywk253100/170822_replica 
Convert 500 error returned by Admiral to duplicate project error when creating duplicate project
 2017-08-22 15:59:19 +08:00
Wenkai Yin
599d94be0c update 2017-08-22 15:22:25 +08:00
Wenkai Yin
ffb2f4201b update 2017-08-22 14:28:45 +08:00
Wenkai Yin
bb958a7f4b convert 500 error returned by Admiral to duplicate project error when creating duplicate project 2017-08-22 13:34:06 +08:00
Tan Jiang
c1bbcb5bab update the interval of clair updater to 12 hours, and update the interval for scan all to 2 hours 2017-08-21 13:45:23 +08:00
Wenkai Yin
7296bdc131 increase length of username in database to 256 2017-08-17 15:24:34 +08:00
Tan Jiang
2ffcf10eaa restart scan jobs when jobservice is started 2017-08-16 17:24:41 +08:00
Daniel Jiang
1403fe09ff Merge pull request #3030 from reasonerjt/fix-jobservice-update-vuln-bug 
Do not throw error if the scan result is unchanged
 2017-08-11 13:26:15 +08:00