Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-10-25 05:40:05 +02:00
Code Issues Projects Releases Wiki Activity
10,312 Commits 45 Branches 311 Tags 346 MiB
ec83f49a1a
Commit Graph

4789 Commits

Author SHA1 Message Date
Wang Yan
ad47d2f444
fix upgrade issue (#12857) 
fixes #12849

1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.

2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-24 16:08:15 +08:00
He Weiwei
356be80f56 fix(quota): change log to debug level when warning not found 
Closes #12861

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-08-24 07:55:46 +00:00
Daniel Jiang
c0602b5fb3
Merge pull request #12832 from ywk253100/200820_data 
Add id column to data_migration table
 2020-08-21 19:30:05 +08:00
Ted Guan
645dea36a6
Fix for duplicate webhook policy name (#12729) 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-08-20 18:02:13 +08:00
Wenkai Yin
975ef193dd Add id column to data_migration table 
Add id column to data_migration table and add logic to make sure there is only one data version record

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-20 17:43:15 +08:00
AllForNothing
b472907362 Change storage of csrf token from cookie to localstorage 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-20 16:22:25 +08:00
Daniel Jiang
05afb94b9b Store csrf token in the header of response instead of cookie 
The current approach will prevent the effectiveness of `Cache-Control`
header and gorilla's library add `Vary:Cookie` header in all responses.

We will set the token in a header of response so the response can be
cached when needed.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-08-20 15:46:02 +08:00
Will Sun
446ae4c173
Merge pull request #12817 from AllForNothing/test-2 
Fix issues with label target 2.1 and RC
 2020-08-20 15:13:27 +08:00
Wenkai Yin(尹文开)
d4f18139ef
Merge pull request #12618 from ywk253100/200729_tk_mgr 
Refresh the status of execution for every status changing of task
 2020-08-20 14:38:09 +08:00
AllForNothing
56083cd5c9 Fix issues with label target 2.1 and RC 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-19 16:46:28 +08:00
wang yan
a7b9495ba0 Merge branch 'master' of https://github.com/goharbor/harbor into fix-resource-order 2020-08-19 15:30:48 +08:00
He Weiwei
d894d4c8bc
Merge pull request #12799 from wy65701436/fixes-12795 
fix gc job submit issue
 2020-08-19 15:27:02 +08:00
Will Sun
c35a39d432
Merge pull request #12814 from AllForNothing/ui-ut 
Improve UI UT for artifact list page
 2020-08-19 15:08:19 +08:00
AllForNothing
2b441a7ee5 Improve UI UT for artifact list page 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-19 12:45:39 +08:00
wang yan
1cc73bd92a Merge branch 'master' of https://github.com/goharbor/harbor into fix-resource-order 2020-08-19 12:21:45 +08:00
wang yan
648b80bc34 udpate resource list order 
1, order label by creation time.
2, order webhook policy by creation time.
3, order replication policy by creation time.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-19 11:20:31 +08:00
Steven Zou
c1b6be6ac9
Merge pull request #12801 from heww/fix-issue-9471 
refactor(scan): remove duplicate CVESet types
 2020-08-19 10:37:10 +08:00
Ted Guan
eb317fb8cb
tag retention webhook support (#12749) 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-08-19 00:07:45 +08:00
stonezdj(Daojun Zhang)
f7e1d186e0
Merge pull request #12792 from ywk253100/200814_proxy_cache 
Return 403 when trying to push artifacts into the proxy cache project…
 2020-08-18 16:22:51 +08:00
stonezdj(Daojun Zhang)
716625a769
Merge pull request #12793 from ywk253100/200816_disable_tag 
Disable the tag creation for the artifact under a proxy cache project
 2020-08-18 16:15:33 +08:00
He Weiwei
ef37bd1afb refactor(scan): remove duplicate CVESet types 
Closes #9471

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-08-18 06:33:17 +00:00
wang yan
44f7bf6739 fix gc job submit issue 
fixes #12795
Return 409 on triggering a GC job when there is a job ongoing.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-18 14:31:54 +08:00
Wang Yan
da52e677e5
remove robot accounts when to delete a project (#12789) 
The robots associate with the project should be removed after the project is deleted.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-18 10:38:45 +08:00
Daniel Jiang
77281ca68b
Token service handling invalid resource (#12790) 
This commit updates the way token service handles invalid resource, for
example a resource without projectname.
It will clear the requested access instead of returning 500 error.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-08-18 10:37:20 +08:00
Will Sun
89b597a811
Merge pull request #12783 from AllForNothing/p2p-4 
Fix UI bugs found on testing day for target 2.1
 2020-08-18 10:29:52 +08:00
Will Sun
9671ff1f10
Merge pull request #12701 from AllForNothing/artifact-icon 
get artifact icon from backend API
 2020-08-18 10:29:15 +08:00
fanjiankong
09ba463cc7 Fix. 
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-08-17 21:40:37 +08:00
He Weiwei
16d1613b10
Merge pull request #12786 from heww/fix-artifact-scan-permission 
fix(scan): fix the permission checking for artifact scanning
 2020-08-17 18:30:18 +08:00
He Weiwei
f659523f50 fix(scan): fix the permission checking for artifact scanning 
Closes #12778

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-08-17 09:55:38 +00:00
Wenkai Yin
0fd230c2d6 Refresh the status of execution for every status changing of task 
Refresh the status of execution for every status changing of task to support filtering executions by status directly

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-17 17:38:55 +08:00
AllForNothing
b664c3c235 Fix UI bugs found on testing day for target 2.1 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-17 15:22:18 +08:00
He Weiwei
929fc8b3f7
Merge pull request #12787 from chlins/fix/preheat-get-log-api 
fix(preheat): fix the swagger definition of preheat get log api
 2020-08-17 14:54:12 +08:00
Wenkai Yin(尹文开)
2de10700d8
Merge pull request #12762 from monofone/10712-remove-forgot-password 
remove reset password dialog from sign-in component
 2020-08-17 14:24:36 +08:00
chlins
4f95c4d067 fix(preheat): fix the swagger definition of preheat get log api 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-17 12:41:02 +08:00
AllForNothing
e6541672d5 get artifact icon from backend API 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-17 10:21:47 +08:00
Wenkai Yin
e9ce631aa3 Disable the tag creation for the artifact under a proxy cache project 
Disable the tag creation for the artifact under a proxy cache project
Fixes #12713

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-16 22:02:30 +08:00
Wenkai Yin
a73742c0a7 Return 403 when trying to push artifacts into the proxy cache project to avoid the retrying in the docker client 
Return 403 when trying to push artifacts into the proxy cache project to avoid the retrying in the docker client
fixes #12731

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-16 21:22:22 +08:00
He Weiwei
f309896f2f refactor(api): generate project apis by go-swagger 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-08-15 16:10:57 +00:00
Wenkai Yin
b1ddb5e2cc Implement the icon API to get the icon of artifact 
Implement the icon API to get the icon of artifact

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-15 08:40:38 +08:00
stonezdj(Daojun Zhang)
205f4f6695
Merge pull request #12768 from stonezdj/200813_bug12741 
Use orm.Context instead request context in background go routing
 2020-08-14 21:15:15 +08:00
stonezdj
ff937e90d3 Use orm.Context instead request context in background go routing 
Fixes #12741,
Avoid use ctx info in event handler because it is background go routine

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-08-14 15:25:06 +08:00
Wenkai Yin
cca1dcca51 Use a separated database table to store the data version 
Use a separated database table to store the data version.
Fixes #12747

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-14 11:38:13 +08:00
Sören Rohweder
8e876cb946 remove reset password dialog from sign-in component 
As mentioned in #10712 forgot password should be removed. The
dialog was left in place.

Signed-off-by: Sören Rohweder <soeren.rohweder@fastleansmart.com>
 2020-08-13 08:27:07 +02:00
Wang Yan
01e4aa61f7
fix gc remove manifest issue (#12748) 
fixes #12720

The GC job doesn't remove the manifest of scheme1.MediaTypeSignedManifest as it's recognized by GC job.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-13 11:25:06 +08:00
He Weiwei
e476286fdc
Merge pull request #12755 from heww/expired-blob-size 
fix(blob): make blob size in redis expired in 24 hours
 2020-08-13 11:23:27 +08:00
chlins
b3012ec08d fix(replication): fix quay adapter authorization 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-12 22:12:08 +08:00
He Weiwei
e3fd83aad1 fix(blob): make blob size in redis expired in 24 hours 
Closes #11873

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-08-12 11:45:50 +00:00
Wang Yan
980c6291df
fix gc foreign layer issue (#12736) 
fixes #12717

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-12 16:43:26 +08:00
Yiyang Huang
b98dc97fbd feat: enhanced default processor 
Signed-off-by: Yiyang Huang <huangyiyang.huangyy@bytedance.com>
 2020-08-11 01:31:02 +08:00
Yiyang Huang
b98b8b9159 Unify parameters for functions in Processor interface 
Signed-off-by: Yiyang Huang <huangyiyang@caicloud.io>
 2020-08-11 01:30:00 +08:00
Chlins Zhang
ad158964bb
Merge pull request #12690 from chlins/fix/preheat-disable-instance-precheck 
fix(preheat): add precheck before disable instance
 2020-08-10 17:36:29 +08:00
stonezdj(Daojun Zhang)
49f4559608
Merge pull request #12518 from stonezdj/202020717_duplicate_op 
Fix #12487: Proxy cache create duplicated operation log
 2020-08-10 17:01:07 +08:00
Wenkai Yin(尹文开)
d599cd98bf
Merge pull request #10455 from chlins/fix/quay-replication-adapter-refactor 
fix(replication): refactor quay adapter to fix authorization and supp…
 2020-08-10 16:37:19 +08:00
stonezdj
aa8b3a1343 Fix #12487: Proxy cache create duplicated operation log 
Change method UseLocalManifest to avoid pull manifest frequently

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-08-10 16:27:59 +08:00
chlins
3340086af5 fix(preheat): add precheck before disable instance 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-10 15:35:44 +08:00
Wenkai Yin(尹文开)
e8f9fb63c0
Merge pull request #12674 from reasonerjt/standalone-db-migrator 
Provide a standalone migrator to migrate DB schema.
 2020-08-10 15:11:52 +08:00
Will Sun
f31b29a01c
Merge pull request #12675 from AllForNothing/p2p-3 
Add name and endpoint check for p2p-preheat
 2020-08-10 10:16:03 +08:00
Chlins Zhang
0cd820ae6c
Merge pull request #12669 from steven-zou/fix/reuse_pro_config 
fix(preheat):leverage project settings
 2020-08-08 13:23:37 +08:00
chlins
b765cfe0ce fix(replication): refactor quay adapter to fix authorization and support quay.io and enterprise quay (#10317) 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-08 13:17:01 +08:00
Chlins Zhang
6b9929d848
Merge pull request #12678 from chlins/fix/preheat-delete-instance-precheck 
fix(preheat): add precheck when delete instance
 2020-08-08 12:54:00 +08:00
Wenkai Yin
219b9910eb Show the detail error message when failed to fetch the artifacts during replication 
Show the detail error message when failed to fetch the artifacts during replication

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-07 18:31:02 +08:00
AllForNothing
041a489fad Add name and endpoint check for p2p-preheat 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-07 16:44:15 +08:00
chlins
5c433c0875 fix(preheat): add precheck when delete instance 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-07 14:06:35 +08:00
疯魔慕薇
19234cdb46
Merge pull request #12673 from chlins/fix/preheat-execution-and-task-pagination 
fix(preheat): add pagination for execution and task list api
 2020-08-06 23:08:21 +08:00
Daniel Jiang
4f94f59d2a Provide a standalone migrator to migrate DB schema. 
Fixes #11885
This part will not by default be packaged into release.
A README.md will be added in another commit.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-08-06 18:57:55 +08:00
chlins
302e4c6659 fix(preheat): add pagination for execution and task list api 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-06 16:59:38 +08:00
Steven Zou
f7bb6dd233
Merge pull request #12665 from chlins/fix/preheat-list-providers-under-project 
fix(preheat): provider more info about p2p provider instance
 2020-08-06 16:57:21 +08:00
Steven Zou
4f90ebce9f
Merge pull request #12668 from chlins/fix/preheat-delete-policy 
fix(preheat): check running executions when delete policy
 2020-08-06 16:47:53 +08:00
Steven Zou
d7bb6dac17 fix(preheat):leverage project settings 
- use content trust settings of project configurations
- use vulnerability severity settings of project configurations

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-08-06 14:15:21 +08:00
chlins
5a2d4dd552 fix(preheat): check running executions when delete policy 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-05 17:54:20 +08:00
Wenkai Yin
d6288a43e8 Do some refine for the scheduler 
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-05 17:43:18 +08:00
chlins
89972f766f fix(preheat): provider more info about p2p provider instance 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-05 17:23:43 +08:00
fanjiankong
4570a46823 Fix bug of update preheat instance default. 
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-08-04 18:54:03 +08:00
Will Sun
5b4573c3ac
Merge pull request #12628 from AllForNothing/pull-by-tag 
Add pull command by tag for artifact detail page
 2020-08-04 09:59:54 +08:00
Will Sun
1006386572
Merge pull request #12641 from AllForNothing/improve-p2p-2 
Improve p2p UI
 2020-08-04 09:59:23 +08:00
Will Sun
18196edaa8
Merge pull request #12642 from AllForNothing/cron 
Improve cron component and create project component
 2020-08-04 09:58:52 +08:00
Chlins Zhang
dc58271d3e
Merge pull request #12649 from chlins/fix/prheat-log-display 
fix(preheat): fix preheat task log display
 2020-08-03 23:01:28 +08:00
Chlins Zhang
6f6742894c
Merge pull request #12647 from chlins/refactor/preheat-policy-serialize 
refactor(preheat): refactor policy schema serialize funcs
 2020-08-03 23:01:01 +08:00
AllForNothing
7065a7a0a7 Improve p2p UI 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-03 18:56:28 +08:00
Daniel Jiang
53577ceb13
Merge pull request #12650 from wy65701436/fix-gc-art-trash 
update GC job
 2020-08-03 18:54:22 +08:00
Wenkai Yin(尹文开)
bfb8f4aca0
Merge pull request #12622 from bitsf/fix_aliacr_pull 
fix[replicatioin] aliyun acr pull with repo filter
 2020-08-03 18:27:23 +08:00
Steven Zou
a6e85c1f8a
Merge pull request #12608 from bitsf/fix_gitlab_pull 
fix[replication] gitlab pull-based multi namespace
 2020-08-03 18:24:45 +08:00
AllForNothing
dff472c497 Improve cron component and create project component 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-08-03 17:36:54 +08:00
wang yan
e7c7c03b8c update GC job 
Remove the artifact trash record after manifest delete success.

Don't use the flush method is because that when all of records are removed, only GC job knows these informations, and they are in the memory. Once the jobservice is crashed and restarted at GC job execution phase, the trash records are lost, then these manifest are become orphan manifests, cannot be removed any more.
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-03 12:08:00 +08:00
chlins
b008dffe70 fix(preheat): fix preheat task log display 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-03 09:45:18 +08:00
chlins
4446302330 refactor(preheat): refactor policy schema serialize funcs 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-02 14:43:30 +08:00
He Weiwei
df1bdc1020 refactor(project): add more methods to project controller and manager 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-31 17:55:35 +00:00
Wang Yan
9ce29c9dc8
update time format of audit log (#12629) 
fixes #11522

use the format: date-time as the format of audit op_time, then it could be rendered by FF and Chrome.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-30 17:29:25 +08:00
AllForNothing
58b8183503 Add pull command by tag for artifact detail page 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-30 16:38:45 +08:00
Wang Yan
e14e6938da
add gc read only job (#12591) 
Add the read only job as a back up plan, user still can use it but just with API, and specify the parameter read_only:true

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-30 15:30:52 +08:00
stonezdj(Daojun Zhang)
518a1721a7
Merge pull request #12571 from ywk253100/200723_proxy_cache_secret 
Limit the permission of secret used by proxy cache service
 2020-07-30 14:04:54 +08:00
Daniel Jiang
0a9219dcd6
Merge pull request #12615 from wy65701436/update-logs 
revise the blob logs
 2020-07-30 09:10:07 +08:00
疯魔慕薇
4dbbf79265
Merge pull request #12617 from steven-zou/fix/preheat_npe_issues 
fix(preheat): fix npe issues
 2020-07-30 08:14:28 +08:00
Steven Zou
507d792655 fix(preheat): fix npe issues 
- fix npe issue in create/update policy
- fix issue of missing schedule job id in the preheat policy

Signed-off-by: Steven Zou <szou@vmware.com>

- increase the client timeout
 2020-07-30 00:29:26 +08:00
Daniel Jiang
1ee4b3dc82
Refine request checking for OIDC CLI secret (#12596) 
This commit makes OIDC CLI secret filter allow more URLs so that the
OIDC CLI secret can be used for replication

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-07-30 00:21:27 +08:00
Ziming Zhang
d16753ae96 fix[replicatioin] aliyun acr pull with repo filter 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-07-30 00:07:54 +08:00
stonezdj(Daojun Zhang)
5b1b94f25c
Merge pull request #12512 from ywk253100/200717_summary 
Include the registry info in the project summary API for proxy cache project
 2020-07-29 14:39:30 +08:00
Steven Zou
5d76a1bd51
Merge pull request #12567 from kofj/schedule-preheat 
Schedule preheat policy.
 2020-07-29 13:56:08 +08:00
wang yan
20df844d5a revise the blob logs 
1, update typo in the update blob status sql, the typo will not impact the sql result.
2, correct blob status in the middleware & GC job log.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-29 12:45:30 +08:00
Wang Yan
cf46775fd7
remove limitation of artifact in the GC job (#12595) 
To handle the orphan blobs that created in the quota exceeding case deletion, remove the limition of artifact remove.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-29 12:04:30 +08:00
Will Sun
b2593589ec
Merge pull request #12602 from AllForNothing/task-col 
Modify column name for preheat task list UI
 2020-07-29 10:45:37 +08:00
fanjiankong
3653d3cdef Schedule preheat policy. 
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-29 00:36:34 +08:00
Steven Zou
d392e27ef9 fix(preheat):enhance preheat job 
- add job stop check points in preheat job
- add missing digest property for the preheat request sent to the provider

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-28 22:19:00 +08:00
Steven Zou
be5858b1ed fix(preheat):fix issues of event-based preheat 
- fix issue of missing handling error in the preheat event handler
- change preheat artifact logic to reduce health check times
- publish pushed events only for the tagged artifacts

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-28 18:32:26 +08:00
Ziming Zhang
e7f8869ff7 fix[replication] gitlab pull-based multi namespace 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-07-28 15:51:49 +08:00
Steven Zou
1d361a89a0
Merge pull request #12599 from steven-zou/fix/p2p_job_log 
fix(preheat):improve preheat job logs
 2020-07-28 13:10:37 +08:00
Daniel Jiang
8fae685708
Merge pull request #11060 from stonezdj/20200311_ldap_group_name 
Set LDAP groupname when PopulateGroup
 2020-07-28 12:47:45 +08:00
AllForNothing
71fa6e9c13 Modify column name for preheat task list UI 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-28 12:18:39 +08:00
Wang Yan
588bf475ae
fix update pull time issue (#12601) 
fixes #12574

let the tag ID as 0 when tags array is empty when to update pull time

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-28 12:15:46 +08:00
stonezdj
7d97ae6ea2 Set LDAP groupname when PopulateGroup 
Search ldap group name with default ldap group attribute name
fixes #10940

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-07-28 10:27:15 +08:00
Steven Zou
8f6fb7db5b
Merge branch 'master' into fix/p2p_job_log 2020-07-27 22:47:15 +08:00
Steven Zou
1adaf58ab1 fix(preheat):improve preheat job logs 
Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-27 22:43:18 +08:00
Steven Zou
0bd321edf7
Merge pull request #12586 from ninjadq/extending_chart_client_timeout 
Enhance: Prolong the timeout of chartclient
 2020-07-27 16:05:29 +08:00
Will Sun
7367227462
Merge pull request #12589 from AllForNothing/fc-bug-css 
Fix css bugs with label target 2.1.0
 2020-07-27 15:43:23 +08:00
Will Sun
0598b3f712
Merge pull request #12592 from AllForNothing/improve-p2p 
Improve p2p-preheat ui
 2020-07-27 15:33:06 +08:00
Will Sun
fab3cfab92
Merge pull request #12585 from AllForNothing/fc-bug 
Fix ui bugs with label target2.1.0
 2020-07-27 15:07:24 +08:00
AllForNothing
71852f6200 Fix css bugs with label target 2.1.0 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-27 15:06:42 +08:00
AllForNothing
7e7663648c Improve p2p-preheat ui 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-27 14:36:50 +08:00
DQ
7bf8a2d044 Enhance: Prolong the timeout of chartclient 
In some perspectives to reduce the too many charts performence issue

Signed-off-by: DQ <dengq@vmware.com>
 2020-07-27 12:06:24 +08:00
Wenkai Yin
ced7b73322 Limit the permission of secret used by proxy cache service 
Limit the permission of secret used by proxy cache service, fixes #12257

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-27 10:15:00 +08:00
Daniel Jiang
3445608e62
Merge pull request #12416 from jonasrosland/move-docs-to-website 
Moving docs from the harbor to the website repo
 2020-07-27 09:45:37 +08:00
Steven Zou
522bd7a8ee
Merge pull request #12582 from steven-zou/fix/nothing_filtered_out_issue 
fix(p2p-preheat):fix issues of triggering preheat
 2020-07-26 22:05:22 +08:00
Steven Zou
716da7f3ff fix(p2p-preheat):fix issues of triggering preheat 
- fix invalid data type of vulnerability filter param
- add more debug logs
- add more logs in the preheat job
- fix issue of getting empty list when doing querying artifacts

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-26 13:47:58 +08:00
Wenkai Yin(尹文开)
6df1f8ae5c
Merge pull request #12549 from chlins/fix/jfrog-replication 
fix(replication): fix jfrog replication when filter includes multi im…
 2020-07-24 16:50:36 +08:00
Steven Zou
cafb2e819c
Merge pull request #12547 from kofj/fix-transaction 
Fix: preheat install update transaction.
 2020-07-24 16:33:29 +08:00
AllForNothing
15d813d479 Add p2p-preheat policy UI 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-23 18:06:16 +08:00
AllForNothing
cc05a60ec5 Fix ui bugs with label target2.1.0 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-23 18:04:03 +08:00
Daniel Jiang
a6c7e15d7e
Merge pull request #12556 from wy65701436/rm-mf-404 
handle path not found when to delete manifest
 2020-07-23 11:22:21 +08:00
stonezdj(Daojun Zhang)
8bbfb811a4
Merge pull request #12543 from stonezdj/200721_release_conn_ldap 
Release connection after search ldap user
 2020-07-23 11:06:14 +08:00
wang yan
53fb5c4230 handle path not found when to delete manifest 
Return not found error if gets the driver PathNotFound.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-23 03:37:11 +08:00
Daniel Jiang
2041fd337b
Merge pull request #12448 from ninjadq/update_chart_client 
Fix Chart V2 Can't parse
 2020-07-22 13:31:05 +08:00
chlins
f70b674b3f fix(replication): fix jfrog replication when filter includes multi images 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-22 12:30:26 +08:00
He Weiwei
06dc5bf10a
Merge pull request #12548 from heww/gen-and-check-mocks 
chore(mocks): add make targets to generate and check mocks
 2020-07-22 11:28:13 +08:00
He Weiwei
e3b1ec775f
refactor(quota): align Get and List methods of quota controller (#12434) 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-22 11:18:05 +08:00
Wang Yan
eeb8fca255
add debugging env for GC time window (#12528) 
* add debugging env for GC time window

For debugging, the tester/users wants to run GC to delete the removed artifact immediately instead of waitting for two hours, add the env(GC_BLOB_TIME_WINDOW) to meet this.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-22 11:09:01 +08:00
Will Sun
46fa64462a
Merge pull request #12532 from AllForNothing/proxy-sum 
Get registrry from project summary api
 2020-07-22 11:01:39 +08:00
Will Sun
f9a66ca666
Merge pull request #12531 from AllForNothing/gc-dry-run 
Add dry run to gc page
 2020-07-22 11:01:10 +08:00
He Weiwei
aa55fcfce7 chore(mocks): add make targets to generate and check mocks 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-22 01:59:32 +00:00
Daniel Jiang
4170b267a1
Merge pull request #12496 from reasonerjt/secret-mgr-proxycache 
Provide secret manager for proxy cache project
 2020-07-22 09:48:24 +08:00
fanjiankong
adbdaaffe6 Fix: preheat install update transaction. 
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-22 08:15:18 +08:00
jonasrosland
e7d1563c6b Moving docs from the harbor to the website repo 
Signed-off-by: jonasrosland <jrosland@vmware.com>
 2020-07-21 12:41:24 -04:00
Wenkai Yin(尹文开)
112e38a080
Merge pull request #12541 from stonezdj/20200720_tag_ret_proxy 
Add default retention policy
 2020-07-21 19:31:12 +08:00
He Weiwei
f18ff08008
Merge pull request #12538 from heww/db-tx-skippers 
perf(db): skip tx for get blob, patch/put blob upload apis
 2020-07-21 19:04:18 +08:00
stonezdj
07694db90d Release connection after search ldap user 
Fixes: 12162

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-07-21 18:00:39 +08:00
stonezdj
2f0f3c45d2 Add default retention policy 
Remove images that are not visited in last 7 days
Add a build method in policy models.go

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-07-21 16:49:35 +08:00
DQ
b62bebc005 Update vendor file 
Signed-off-by: DQ <dengq@vmware.com>
 2020-07-21 10:16:22 +08:00
AllForNothing
2396410cce get registrry from project summary api 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-21 10:10:09 +08:00
DQ
f957acf763 Fix Chart V2 Can't parse 
Use helm v3 libary to replace v2

Signed-off-by: DQ <dengq@vmware.com>
 2020-07-21 10:07:08 +08:00
AllForNothing
8d9b1e18d3 Add dry run to gc page 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-21 09:51:31 +08:00
He Weiwei
6db1a1cb91 perf(db): skip tx for get blob, patch/put blob upload apis 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-20 16:42:16 +00:00
Steven Zou
cae087ba49
Merge pull request #12511 from steven-zou/fix/p2p_provider_health_check 
fix(preheat):remove health property check when saving instance
 2020-07-20 21:52:06 +08:00
Steven Zou
ee35e1ecc6
Merge pull request #12507 from chlins/fix/preheat-update-instance 
fix(preheat): fix preheat handler PingInstance and UpdateInstance
 2020-07-20 17:45:24 +08:00
jwangyangls
25e0f161ea
Merge pull request #12508 from jwangyangls/unified-processing-session-expired 
[fix][ui] Operation unified processing session expired
 2020-07-20 17:21:54 +08:00
Steven Zou
144edcc0c4 fix(preheat):fix ctl ut failure case 
Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-20 17:14:17 +08:00
Steven Zou
6b9e0e66c4 Merge branch 'master' into fix/p2p_provider_health_check 2020-07-20 16:02:40 +08:00
Daniel Jiang
14203169bf Add GC mechanism to secret manager 
When Generate is called and the size is larger than cap, GC will be
triggered.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-07-20 15:53:36 +08:00
Steven Zou
46d7434d0b
Merge pull request #12473 from ywk253100/200706_scheduler 
Refactor the scheduler with the task manager mechanism
 2020-07-20 15:53:14 +08:00
chlins
78927af032 fix(preheat): fix preheat handler PingInstance and UpdateInstance 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-20 15:42:24 +08:00
Wenkai Yin
4dc4b6728c Refactor the scheduler with the task manager mechanism 
Refactor the scheduler with the task manager mechanism, this will reduce the duplicate code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-20 14:03:15 +08:00
Steven Zou
3c02b5da03 fix(preheat):remove health property check when saving instance 
Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-20 13:53:06 +08:00
Steven Zou
a11c2f32da
Merge pull request #12502 from bitsf/fix_replication_swr_pull 
fix SWR replication adapter pull-based
 2020-07-20 13:25:47 +08:00
Wang Yan
24ed52112e
fix blob deleting status issue (#12481) 
1, The update blob status method should udpate the blob version of the blob object as well, otherwise the GC job cannot handle the blob status transform(none - delete - deleting - deletefailed)
as the method is using version equals as the query condition.
2, For the deleting blob which marked for more than 2 hours, it should be set to delete failed in head blob & put manifest request

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-20 11:44:29 +08:00
Wenkai Yin(尹文开)
5a898c1661
Merge pull request #12510 from ywk253100/200717_copy_proxy 
Prevent copying artifact to a proxy cache project
 2020-07-20 11:10:46 +08:00
Wenkai Yin(尹文开)
c654b08346
Merge pull request #12506 from ywk253100/200717_save_registry 
Don't return the error detail back to the client when adding registry
 2020-07-20 11:09:36 +08:00
Ziming Zhang
8857e89e40 feature(redis) support redis sentinel 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-07-19 21:19:03 +08:00
Daniel Jiang
d891e023db
Merge pull request #9311 from airadier/autoonboard-and-custom-user-claim 
Add options for automatic onboarding and username claim
 2020-07-19 19:15:34 +08:00
Steven Zou
d9ca9bbc69
fix(jobservice):add job id in log getting error (#12513) 
Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-17 22:33:41 +08:00
Wenkai Yin
d7327d8d8e Include the registry info in the project summary API for proxy cache project 
Include the registry info in the project summary API for proxy cache project

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-17 16:14:45 +08:00
wang yan
3345b8aae2 fix get manifest return code 
When to call,
~~~ REQUEST ~~~
GET  /v2/conformance/testrepo/manifests/.INVALID_MANIFEST_NAME

Per OCI distribution spec, it has to return 404, instead of 400 (project name required)

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-17 15:40:52 +08:00
Yogi_Wang
7b22b38c5c [fix][ui] Operation unified processing session expired 
fix #11268  includes the following functions:
delete group  delete chart version  delete project   delete member  add group    delete repo  delete artifact   delete tag   delete robot  delete user   delete registry   delete label  delete replication list     replication
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-07-17 15:33:39 +08:00
Wenkai Yin
54a1155140 Prevent copying artifact to a proxy cache project 
Prevent copying artifact to a proxy cache project

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-17 15:24:18 +08:00
Wenkai Yin
9493611666 Don't return the error detail back to the client when adding registry 
Don't return the error detail back to the client when adding registry to avoid security issue

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-17 11:58:08 +08:00
Ziming Zhang
e7c89ce1d9 fix SWR replication adapter pull-based 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-07-16 19:19:18 +08:00
Daniel Jiang
840aa86dfa Provide secret manager for proxy cache project 
This commit provides the secret manager for proxy cache.
The secret is used for pushing blobs to local when it's proxied from
remote registry.
Each secret can be used only once and has a relatively short expiration
time.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-07-16 19:12:52 +08:00
Alvaro Iradier
81a7239c66 Better error handling 
* Raise an internal error if username claim is not found, instead of just logging a warning
* Don't remove userInfoKey for session on error when it is not required
* Rename "OIDC Username Claim" to just "Username claim"

Signed-off-by: Alvaro Iradier <airadier@gmail.com>
 2020-07-16 12:12:08 +02:00
Alvaro Iradier
6f88ff7429 Fix test suite and add test for userClaim 
Signed-off-by: Alvaro Iradier <airadier@gmail.com>
 2020-07-16 12:12:08 +02:00
Alvaro Iradier
714f989759 Add options for automatic onboarding and username claim 
- Add an option in the UI to enable or disable the automatic user onboarding
- Add an option to specify the claim name where the username is retrieved from.

Signed-off-by: Alvaro Iradier <airadier@gmail.com>
 2020-07-16 12:12:08 +02:00
Wang Yan
bad8f026fc
upgrade golang to v1.14.5 (#12489) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-16 16:20:54 +08:00
Wang Yan
d73265d10d
revise gc job to align non blocking gc (#12439) 
two phases:
1, mark, select the gc candidates bases on the DB and mark them as status delete.
2, sweep, select the candidate and mark it as status deleting and remove it from backend and database.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-16 14:35:54 +08:00
Steven Zou
4d4a04fad4
Merge pull request #12478 from steven-zou/feat/read_pro_config 
feat(p2p):enhance policy enforcer
 2020-07-16 11:40:29 +08:00
Ted Guan
9e7edb7a6e
Fix for project metadata (#12410) 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-07-15 18:46:45 +08:00
Steven Zou
fcfde5a588 feat(p2p):enhance policy enforcer 
Read security settings from the project configurations and override the preheat policy settings if necessary.

Check the project security settings and override the related settings in the policy if necessary.
NOTES: if the security settings (relevant with signature and vulnerability) are set at the project configuration,
they will have the highest priority and override the related settings of the preheat policy.
 e.g (use signature as an example, similar case to vulnerability severity part):
   if policy.signature = false and project.config.signature = true; then policy.signature = true
   if policy.signature = true and project.config.signature = true; then policy.signature = true
   if policy.signature = true and project.config.signature = false; then policy.signature = true
   if policy.signature = false and project.config.signature = false; then policy.signature = false

Signed-off-by: Steven Zou <szou@vmware.com>

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-15 14:18:34 +08:00
He Weiwei
cadcd4b877
Merge pull request #12480 from heww/move-pkg-types 
refactor(quota): move pkg/types to pkg/quota/types
 2020-07-15 11:32:27 +08:00
Daniel Jiang
947eadaa72
Merge pull request #12440 from heww/remove-init-clair-db 
refactor: remove initialization of clair db
 2020-07-15 00:38:12 +08:00
He Weiwei
a22d803a95 refactor(quota): move pkg/types to pkg/quota/types 
Closes #9664

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-14 14:28:53 +00:00
stonezdj
b9c861f3f1 Add disable push for proxy project 
Add middleware for blob and manifest push operation

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-07-14 10:13:38 +08:00
疯魔慕薇
f187509a90
Merge pull request #12454 from chlins/feat/list-providers-at-project-level 
feat(preheat): add list providers under project level handler
 2020-07-14 08:17:53 +08:00
chlins
7322d0ac7c feat(preheat): add list providers under project level handler 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-13 21:53:50 +08:00
stonezdj
3abe77d6cb Add proxy cache feature 
Update route to add proxy related middleware
Add proxy controller

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-07-13 21:18:43 +08:00
chlins
08bd46e125 feat: add preheat execution api handler 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-13 13:14:08 +08:00
chlins
2863e68718 feat: add task controller 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-13 13:02:24 +08:00
fanjiankong
a99aa21c8a Enable RBAC control in the preheat API 
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-13 11:06:25 +08:00
AllForNothing
4ec919dfe6 Query preheat instance by name 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-11 17:41:56 +08:00
Julien Vey
84c34afa22 Fix typo in sql log 
Signed-off-by: Julien Vey <vey.julien@gmail.com>
 2020-07-10 14:29:04 +02:00
Will Sun
ec1ac6dbc8
Merge pull request #12344 from AllForNothing/p2p 
Add P2p preheat distribution instance UI
 2020-07-10 16:01:04 +08:00
Steven Zou
1dfc93c3f6
Merge pull request #12430 from kofj/preheat_and_healthcheck 
Preheat and healthcheck
 2020-07-10 15:17:34 +08:00
Steven Zou
fdff077ff0
Merge pull request #12445 from chlins/fix/preheat-instance-and-policy-name-validation 
fix(preheat): validate instance/policy name and set unique name
 2020-07-10 15:14:44 +08:00
chlins
38d14dff30 fix(preheat): validate instance/policy name, set unique filed and policy 
manager adds parsePolicy

Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-10 10:29:47 +08:00
fanjiankong
080afbfe1b Add preheat APIs, handlers. 
1. Manual preheat.
2. Instance health check.

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-10 09:48:35 +08:00
He Weiwei
039aef5356 refactor: remove initialization of clair db 
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-09 15:26:14 +00:00
He Weiwei
9483559d18
Merge pull request #12433 from heww/fix-db-max-open-conns 
fix(db): set max open conns of sql.DB manually
 2020-07-09 17:58:14 +08:00
疯魔慕薇
5d7f757b7b
Merge pull request #12428 from mmpei/official-master-p2p-200708 
Add P2P trigger event and handler
 2020-07-09 15:54:24 +08:00
He Weiwei
e095958a27 fix(db): set max open conns of sql.DB manually 
Due to the issues of beego v1.12.1 and v1.12.2, we set the max open conns
ourselves.

Closes #12403

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-09 03:35:41 +00:00
peimingming
65c5561032 Add P2P trigger event and handler 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2020-07-09 11:20:22 +08:00
fanjiankong
8a44ee400d Instance handler. 
- Add logic of preheat instance methods without RBAC.

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-09 00:01:14 +08:00
Steven Zou
3b2934bf48
Merge pull request #12419 from chlins/feat/p2p-preheat-healthcheck-controller 
feat(preheat): add healthcheck methods for p2p preheat controller
 2020-07-08 23:49:26 +08:00
Will Sun
71e50bd364
Merge pull request #12420 from AllForNothing/proxy-from-api 
Add query string for getting registry
 2020-07-08 14:20:38 +08:00
Will Sun
96b2326e24
Merge pull request #12418 from goharbor/rename-master-role 
Rename master role to maintainer
 2020-07-08 11:02:24 +08:00
AllForNothing
2c422435d9 Add query string for getting registry 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-08 10:42:55 +08:00
chlins
b6cab91bfa feat(preheat): add healthcheck methods for p2p preheat controller 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-08 09:58:01 +08:00
Steven Zou
b87aa3b9d5
chore(preheat):add mock policy enforcer (#12414) 
Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-08 09:45:14 +08:00
Will Sun
cd66899abb
Merge pull request #12417 from AllForNothing/rename-role 
Rename project role for UI
 2020-07-08 09:21:57 +08:00
Daniel Jiang
1637e6a588 Rename master role to maintainer 
This commit rename the var name, text appearance, and swagger of "master" role
to "maintainer" role.
It only covers backend code.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-07-08 09:20:07 +08:00
AllForNothing
706c6bdb2b Rename project role 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-08 09:15:01 +08:00
Steven Zou
b56a49efe2
Merge pull request #12411 from kofj/instance_by_name 
Get instance by name.
 2020-07-07 23:20:01 +08:00
fanjiankong
3c1c799f0d Get instance by name. 
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-07 22:06:38 +08:00
Wang Yan
c3baeac5ae
add time windows support in artifact trash (#12400) 
support with time window to filter and delete item in artifact trash

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-07 20:13:08 +08:00
AllForNothing
d01ff31dc8 Add P2p preheat distribution instance UI 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-07 19:55:56 +08:00
stonezdj(Daojun Zhang)
6f4e8150d5
Merge pull request #12383 from ywk253100/200702_registry_api 
Suport filtering registries by type in listing registry API
 2020-07-07 14:21:54 +08:00
Wenkai Yin
02690d1d04 Suport filtering registries by type in listing registry API 
Suport filtering registries by type in listing registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-07 10:30:46 +08:00
Steven Zou
642953b9ef
Merge pull request #12395 from chlins/feat/preheat-controller-policy-part 
Feat/preheat controller policy part
 2020-07-06 22:21:34 +08:00
chlins
37a00912b7 feat: add p2p preheat swagger yaml and implement preheat api policy handler 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-06 21:05:22 +08:00
jwangyangls
e498893777
Merge pull request #12391 from jwangyangls/fix-replication-repo 
[fix] [replication] Success rate should be displayed as 0 when status is inprogress
 2020-07-06 10:09:20 +08:00
chlins
254ea193fa feat: add p2p preheat controller policy releated logic 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-04 12:23:04 +08:00
Steven Zou
2f2c545a0f
Merge pull request #12392 from steven-zou/fix/remove_inst_provider_todo 
fix(p2p):remove the provider manager related to
 2020-07-03 23:59:55 +08:00
Will Sun
7dfab5858c
Merge pull request #12374 from AllForNothing/proxy-cache 
Add proxy cache ui
 2020-07-03 18:14:25 +08:00
Steven Zou
1ee3f00709 fix(p2p):remove the provider manager related to 
- use real provider instance manager
- move mock insatnce manager to testing/pkg
- modify kraken deriver implementation to remove digest fetcher
- update related UT cases

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-03 17:30:11 +08:00
Yogi_Wang
de4793c52d [fix] [replication] Success rate should be displayed as 0 when status is inprogress 
1.[replication] sccess rate should be displayed as 0 when status is inprogress
2.[repository] fix get repo name error when the repo contains the project name field
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-07-03 16:19:18 +08:00
Steven Zou
c7a050b629
Merge pull request #12389 from chlins/fix/p2p-preheat-policy-count-total 
fix: add count method of policy manager to replace list method return…
 2020-07-03 13:50:34 +08:00
Steven Zou
f3fcb96570
Merge pull request #12335 from kofj/p2p_preheat_api 
feat(preheat):add preheat api, controller and manager
 2020-07-03 13:47:04 +08:00
chlins
ace21240a4 fix: add count method of policy manager to replace list method return wrong counts 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-03 11:59:32 +08:00
Wang Yan
e8784de5fe
support list blobs by update time (#12385) 
Add support list blob with update time.
As introduces the time window in GC, it wants to list the blobs less than specific time.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-03 11:50:53 +08:00
Daniel Jiang
cf07ff0052
Merge pull request #12386 from wy65701436/delete-blob-not-found 
fix return value for blob & manifest not found
 2020-07-03 11:37:31 +08:00
Daniel Jiang
8252930083
Merge pull request #12356 from wy65701436/put-mf-mw 
add middlware for put manifest
 2020-07-03 11:30:25 +08:00
Daniel Jiang
650142e536
Merge pull request #12381 from wy65701436/fix-delete-manifest 
fix delete manifest route issue
 2020-07-03 11:29:06 +08:00
fanjiankong
a0c2d0ac9e feat(preheat):add preheat api, controller and manager 
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-03 11:25:42 +08:00
AllForNothing
a13642c2af Add proxy cache ui 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-07-03 10:20:55 +08:00
wang yan
95bee9a0cc fix return value for blob & manifest not found 
When to delete an non exist blob/manifest, the API has to return a 404 instead of 500

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-02 19:18:57 +08:00
wang yan
7d1507feaa update code arrording to the review comments 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-02 14:57:04 +08:00
wang yan
67be511a85 add middlware for put manifest 
The middleware is to handle manifest(blob) status in different push manifest situation, similar with blob

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-02 14:57:04 +08:00
Steven Zou
264bd02892
Merge pull request #12378 from steven-zou/feat/preheat_policy_enforcement 
feat(preheat):implement policy enforcer
 2020-07-02 14:30:25 +08:00
wang yan
12c92dbfce fix delete manifest route issue 
The repository name contains blackslash, the mux router has to use the * to match the blackslash. Otherwise the caller(gc job) will get a 404.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-02 13:42:48 +08:00
Steven Zou
18137a5c55 feat(preheat):implement policy enforcer 
- define policy enforcer interface
- implement the default enforcer
- registrer P2P preheat job to JS
- add the missing mock manager&controller in the src/testing pkg
- Add UT cases for enforcer
- fix #12285
- left one TODO: query provider instance by instance Manager

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-07-02 11:33:11 +08:00
Wang Yan
57c72b7952
add get GC candidate (#12314) 
* add get GC candidate

select non referenced blobs from table blob and exclude the ones in the time windows.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-30 20:41:17 +08:00
Chlins Zhang
5bfe82612a
Merge pull request #12342 from chlins/feat/p2p-preheat-policy-dao-and-manager 
feat: add p2p preheat policy dao and manager(#12286)
 2020-06-30 16:35:42 +08:00
chlins
15e4361d6e feat: add p2p preheat policy dao and manager(#12286) 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-06-30 15:56:50 +08:00
Steven Zou
a06c03625d
Merge pull request #12349 from steven-zou/feat/p2p_policy_evaluator 
feat(preheat):add artifact filters for preheat policy
 2020-06-30 14:56:58 +08:00
Steven Zou
d8e88ef5bc feat(preheat):add artifact filters for preheat policy 
- add new selector based on vulnerability severity criteria
- add new selector based on signature(signed) criteria
- do change to the select factory method definition
- do changes to selector.Candidate model
- add preheat policy filter interface and default implementation
- add UT cases to cover new code

Signed-off-by: Steven Zou <szou@vmware.com>

misspelling
 2020-06-30 10:48:21 +08:00
Wang Yan
468ba50a7e
handle blob status chanage in put blob middlware (#12315) 
* handle blob status chanage in put blob middlware

After blob is uploaded success, the middleware will update the blob status accordingly.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-29 16:38:47 +08:00
Daniel Jiang
7d50a6aab6
Merge pull request #12281 from pcfens/upgrade_aws_module 
Upgrade aws-sdk-go to 1.32.5 for OIDC support
 2020-06-29 16:35:11 +08:00
Steven Zou
8e2c334b43
Merge pull request #12338 from steven-zou/feat/p2p_preheat_job 
feat(preheat):add preheat job
 2020-06-29 12:27:05 +08:00
Steven Zou
6424480f37 feat(preheat):add preheat job 
- implement the preheat job
- add relevant UT case
- fix issue #12285

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-06-28 23:08:02 +08:00
chlins
dd9e97f755 feat: add p2p preheat policy model 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-06-28 10:46:50 +08:00
Steven Zou
df86ae1ad0 feat(preheat):add preheat drivers 
- define preheat driver interface
- implement dragonfly driver
- implememt kraken driver
- add related UT cases with testify framework
- fix #10870 #10871
- some code are picked up from the original P2P feat branch

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-06-25 23:39:34 +08:00
AllForNothing
fff6f7529a Replace all whitelist with allowlist 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-06-24 16:17:17 +08:00
Phil Fenstermacher
33069e0f98
Upgrade aws-sdk-go 
Versions of the Go AWS SDK newer than 1.23.13 support OIDC in EKS.
Running Harbor on EKS doesn't require keys in a configmap for the
registry to authenticate to S3 when using the newer library.

Signed-off-by: Phil Fenstermacher <pcfens@wm.edu>
 2020-06-23 09:04:12 -04:00
Wenkai Yin(尹文开)
202916e396
Merge pull request #12280 from ywk253100/200616_task_manager 
Implement task and execution manager
 2020-06-23 18:44:44 +08:00
Wenkai Yin
ea20690264 Implement task and execution manager 
Implement task and execution manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-06-23 17:10:58 +08:00
wang yan
0e175017aa continue updating code per review comments 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-23 14:42:28 +08:00
wang yan
446739f967 rebase with latest source code 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-23 13:10:57 +08:00
wang yan
c10467eb36 continue refactor 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-23 13:10:57 +08:00
Wang Yan
de504993ad update blob controller & manager 
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-23 13:10:57 +08:00
Wang Yan
58b7242a25
move send error to source lib (#12175) 
* move send error to source lib

Move the sendError into library in case the cycle dependency as regsitry and core are now the consumers.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-19 01:04:50 +08:00
Steven Zou
ffd889f82a
Merge pull request #12239 from ywk253100/200615_task_manager_dao 
Implement execution/task DAO for task manager
 2020-06-17 17:56:00 +08:00
stonezdj(Daojun Zhang)
91bff55b66
Merge pull request #12214 from stonezdj/20200611_add_proxyservice_secret 
Add temporary secret for harbor proxy service
 2020-06-17 10:46:13 +08:00
Wenkai Yin
5a1827768a Implement execution/task DAO for task manager 
Implement execution/task DAO for task manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-06-16 10:24:58 +08:00
Wenkai Yin
127988b70c Define the task manager interface and data model 
Define the task manager interface and data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-06-15 18:42:09 +08:00
stonezdj(Daojun Zhang)
56ca3dddda
Merge pull request #12203 from ywk253100/200610_proxy_cache_api 
Update creating project API to support proxy cache project
 2020-06-15 15:24:08 +08:00
stonezdj
82f59cb760 Add temporary secret for harbor proxy service 
Use GenerateRandomStringWithLength function to generate secret for harbor_proxyservice
Add harbor-proxyservice secret used by proxy service

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-06-15 14:43:43 +08:00
Daniel Jiang
373ac25b9b
Merge pull request #12195 from reasonerjt/fix-post-user-403 
Makes api/users return 401 for request not authenticated
 2020-06-11 15:51:52 +08:00
Steven Zou
64f03cc7fe
Merge pull request #12079 from ninjadq/chartrepo_enumeration_master 
Fix: chart repo response code inconsistent
 2020-06-11 15:08:14 +08:00
Wenkai Yin(尹文开)
b08dfd57b6
Merge pull request #12127 from ywk253100/200603_label 
Remove the label from artifacts when deleting a label
 2020-06-11 14:49:14 +08:00
Wenkai Yin
06f7b7f763 Remove the label from artifacts when deleting a label 
Fixes #12112, remove the label from artifacts when deleting a label

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-06-11 14:02:46 +08:00
DQ
e1313da2f0 Fix: chart repo response code inconsistent 
Response code should consistent whether namespace is existed or not

Signed-off-by: DQ <dengq@vmware.com>
 2020-06-11 13:55:12 +08:00
Will Sun
3f4c8cc983
Merge pull request #12191 from AllForNothing/new-2.0.1 
Fix UI issues with label target 2.0.1
 2020-06-11 12:09:56 +08:00
AllForNothing
a773115cc3 Fix UI issues with label target 2.0.1 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-06-11 11:29:18 +08:00
jwangyangls
4b496438b1
Merge pull request #12204 from jwangyangls/fix-harbor-icon-safari-break 
[fix][ui] about dialog Icon cutoff in safari
 2020-06-11 10:15:09 +08:00
Daniel Jiang
9b4f2cb0bc
Merge pull request #12194 from reasonerjt/unknown-serverity-lower 
Lower the severity of a "Unknown" vulnerability
 2020-06-10 20:31:26 +08:00
Yogi_Wang
622cb4da2b [fix][ui] about dialog Icon cutoff in safari 
fix #12144
env 10.78.96.167
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-06-10 17:28:34 +08:00
Wenkai Yin
a79bb127b3 Update creating project API to support proxy cache project 
Update creating project API to support proxy cache project

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-06-10 17:14:12 +08:00
Daniel Jiang
3b776d1a47 Makes api/users return 401 for request not authenticated 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-06-10 16:37:49 +08:00
Daniel Jiang
091dbc3454 Lower the severity of a "Unknown" vulnerability 
This commit lower the actual severity of "Unknown" vulnerability to the
same level of "None"

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-06-10 15:16:25 +08:00
Daniel Jiang
0ac5568619
Merge pull request #12183 from reasonerjt/project-id-401 
Change status code for projects API
 2020-06-09 23:09:14 +08:00
jwangyangls
9e1778b32a
Merge pull request #12165 from jwangyangls/popup-unauthorized 
[fix][unauthorized] Support show relogin when unauthorized  in popup modal
 2020-06-09 13:05:45 +08:00
Daniel Jiang
f73aa3ce80 Change status code for projects API 
Update to projects API so it will not differentiate if a project
does not exist or the user doesn't have permission to access it.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-06-09 08:24:06 +08:00
Wang Yan
c993103e01
enable storage driver of gcs and oss (#12180) 
Add go build tags for gcs and oss, otherwise these drivers cannot be registered and the error "StorageDriver is not regsited: GCS" will raise on registryctl launch under the setting of GCS storage.

These build tags are designed in the distribution, just refer to https://github.com/docker/distribution/blob/release/2.7/registry/storage/driver/gcs/gcs.go#L13

Pin the google cloud API to a old version is because distribution depends on it, otherwise go mode will use v0.17.0 that go-migrate is using as the dependency version, but this version will break the compile process with following error:
harbor/pkg/mod/google.golang.org/cloud@v0.0.0-20151119220103-975617b05ea8/storage/acl.go:65:16: invalid type assertion: v.(map[string]<inter>) (non-interface type *storage.ObjectAccessControl on left)
that's bacause another dependency google.golang.org/cloud requires the pinned version of google.golang.org/api.

The pinned package should be removed once https://github.com/docker/distribution/pull/3019 is merged, and distribution ships their v2.8.0

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-09 00:57:27 +08:00
Yogi_Wang
6f7aa68c89 [fix][unauthorized] Support show relogin when unauthorized in popup modal 
1.support show relogin when unauthorized  in popup modali
2.change change member role error handle
3.change helm chart service error handle
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-06-08 16:40:34 +08:00
Wang Yan
dec8397c21
Add api to delete blob and manifest (#12006) 
* Add api to delete blob and manifest

Enable the capability of registry controller to delete blob and manifest

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-06 01:34:23 +08:00
Daniel Jiang
9f159393df
Merge pull request #12163 from reasonerjt/disable-webhook-test 
Remove the API to test a webhook
 2020-06-05 18:14:39 +08:00
Daniel Jiang
39a5efd54c
Merge pull request #12160 from reasonerjt/bump-up-go-migrator 
Bump up golang-migrate
 2020-06-05 18:13:40 +08:00
Daniel Jiang
9a9e7d61fc Remove the API to test a webhook 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-06-05 13:16:18 +08:00
Daniel Jiang
bc7ede21c7 Bump up golang-migrate 
This commit bumps it up to 4.11.0
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-06-05 10:29:02 +08:00
Wenkai Yin(尹文开)
b7b1ce457b
Merge pull request #10389 from lxShaDoWxl/fix/gitlab 
Fixed search images in registry gitlab
 2020-06-04 14:49:40 +08:00
Steven Zou
6db856c3e2
Merge pull request #12121 from steven-zou/fix/job_stop_status_issue 
fix(jobservice):mismatch status issue when stopping job
 2020-06-03 18:13:23 +08:00
Steven Zou
3cd47af9a5 fix(jobservice):mismatch status issue when stopping job 
- returnning nil instead of error when trying to stop a job that has been in the final status(Error/Success/Stopped)
- do enhancements to the periodic job unschedule func
- fix a UT nil ptr issue

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-06-03 16:18:42 +08:00
AllForNothing
8bf77d01f4 Fix bugs with label 'target 2.0.1' 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-06-03 11:25:22 +08:00
jwangyangls
6939446c3c
Merge pull request #12115 from jwangyangls/fix-issue-add-label 
[fix] Fix  issue in front ui 2.0.1
 2020-06-02 15:52:37 +08:00
Yogi_Wang
459314308b [fix] issue in front ui 2.0.1 
1.fix add label issue
2.fix unauthorized user can contrl page
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-06-02 13:54:22 +08:00
Steven Zou
c7c1742b88
Merge pull request #12106 from heww/clean-clair-url 
refactor(configuration): cleanup unneeded CLAIR_URL configuration in core
 2020-06-01 19:24:19 +08:00
Steven Zou
a032c546f8
Merge pull request #12096 from steven-zou/fix/remove_checkin_data 
fix(jobservice):fix issues of jobservice
 2020-05-29 16:49:33 +08:00
He Weiwei
d97be71234 refactor(configuration): cleanup unneeded CLAIR_URL configuration in core 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-05-29 07:27:50 +00:00
Steven Zou
51f1b2e590 fix(jobservice):fix issues of jobservice 
- never expire the jobs that are not entering the final status (Error,Success or Stopped)
- set different expireation time to the jobs with different status
- never store the `check_in` data in the redis db to save space

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-05-29 11:36:08 +08:00
Chlins Zhang
f862805244
Merge pull request #11998 from chlins/fix/replication-pull-blob-content-length 
fix(replication): ignore the problem of not getting Content-Length fr…
 2020-05-29 10:06:44 +08:00
mmpei
7c2bfb1378
Fix helmhub insecure issue when using with a proxy (#12014) 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2020-05-28 10:57:51 +08:00
He Weiwei
9f5f8e4681
Merge pull request #12077 from heww/fix-issue-11951 
fix(cve-whitelist): fix panic in `Get` method of whitelist manager
 2020-05-27 19:16:11 +08:00
He Weiwei
0d1b9c96ff fix(cve-whitelist): fix panic in Get method of whitelist manager 
Closes #11951

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-05-27 09:55:54 +00:00
chlins
54895def90 fix(replication): ignore the problem of not getting Content-Length from the pull blob api 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-05-27 13:00:54 +08:00
Will Sun
31a943bb42
Merge pull request #12043 from AllForNothing/disable-test 
Remove test button for webhook
 2020-05-27 11:46:58 +08:00
AllForNothing
ea753bec97 Remove test button for webhook 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-05-27 10:26:41 +08:00
Will Sun
da0e80b712
Merge pull request #11452 from jd615645/master 
Chinese(Traditional) translation #402
 2020-05-27 10:01:25 +08:00
lxShaDoWxl
681e97482e fix(Registries/Gitlab): upgrade testing 
Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>
 2020-05-26 20:22:26 +06:00
lxShaDoWxl
05f78a561a Merge remote-tracking branch 'origin/master' into fix/gitlab 2020-05-24 16:05:32 +06:00
Will Sun
2ff0638f8b
Merge pull request #10576 from hobti01/repo-slash 
fix: Allow forward slash in destination namespace
 2020-05-21 10:50:55 +08:00
yuzhiquan
6332daef31 struct instead of int within signal chan, and trans Printf to Println 
Signed-off-by: yuzhiquan <yuzhiquanlong@gmail.com>
 2020-05-21 00:47:55 +08:00
Wenkai Yin
a31315aa36 Fix replication bugs #11974, #11939 
Fix replication bugs: fix #11974, fix #11939

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-05-21 00:25:49 +08:00
He Weiwei
69fe9e9bf7
Merge pull request #11106 from qinshaoxuan/fix_11051 
Fix bug when scanner is unhealthy
 2020-05-20 21:06:22 +08:00
ctu
7ea8ed2588 🐈 Add zh-tw translate 
Signed-off-by: ctu <ctu@cs.nctu.edu.tw>
 2020-05-20 10:56:29 +00:00
Tim Hobbs
36747087ce
fix: Allow forward slash in destination namespace 
Signed-off-by: Tim Hobbs <timothy.hobbs@ic-consult.com>
 2020-05-20 11:27:50 +02:00
Daniel Jiang
b803975b36
Merge pull request #11971 from wy65701436/fixes-11949 
fix conformance issue on deleting manifest
 2020-05-20 13:19:47 +08:00
AllForNothing
2d141a919d Add more UT for create project component 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-05-19 17:39:42 +08:00
wang yan
f0fabb9ef8 fix conformance issue on deleting manifest 
fixes #11949
When to call delete manifest API with a tag as the reference, Harbor should give a unsupported error code.

Reference: Note that a manifest can only be deleted by digest. https://github.com/opencontainers/distribution-spec/blob/master/spec.md#delete-manifest

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-05-19 15:58:33 +08:00
Wang Yan
b1793e795c
fix conformance test ErrorsCodes failure (#11961) 
fixes #11945
When client calls PUT with an invalid digtest, Harbor has to give a 400 instead of 500.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-05-18 17:20:23 +08:00
guanxiatao
0aefd8f2e2 Add UT for webhook when replicating with docker registry 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-05-13 19:09:33 +08:00
AllForNothing
1c670329dc Fix duration and log bugs for replication task UI 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-05-13 12:38:23 +08:00
Will Sun
5289b3519e
Merge pull request #11860 from AllForNothing/docker-logo 
Modify doc and fix some ui bugs
 2020-05-11 11:00:36 +08:00
guanxiatao
e8655c667c Core panic fix when triggering a webhook of docker-registry replication 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-05-11 10:33:23 +08:00
AllForNothing
c50cbbf3cc Modify doc and fix some ui bugs 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-05-11 09:39:33 +08:00
Will Sun
119751e0fd
Merge pull request #11835 from AllForNothing/ui-bugs 
Fix some UI bugs
 2020-05-07 11:45:57 +08:00
AllForNothing
bdf6c5b8d9 Fix some UI bugs 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-05-06 16:41:35 +08:00
lxShaDoWxl
47a57b80d7 Merge remote-tracking branch 'origin/master' into fix/gitlab 2020-05-06 13:26:18 +06:00
AllForNothing
90e34e0104 Improve i18n service 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-05-06 14:45:56 +08:00
Ziming
56609a8026
fix(retention) set pushtime and pulltime for untagged (#11786) 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-04-28 19:20:46 +08:00
Will Sun
4324f0d2c8
Merge pull request #11780 from AllForNothing/tag-re 
Replace images with artifacts for tag-retention
 2020-04-28 16:53:47 +08:00
danfengliu
16f6ad3688
Merge pull request #11768 from AllForNothing/nightly-login 
Add new nightly case  admin add new users
 2020-04-28 14:57:37 +08:00
AllForNothing
b5617c0868 Replace images with artifacts for tag-retention 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-28 14:04:09 +08:00
AllForNothing
8ff4003438 Add new nightly case admin add new users 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-28 11:10:06 +08:00
He Weiwei
5c5ba0b764
Merge pull request #11765 from heww/quota-webhook-enhancement 
feat(quota,webhook): send quota webhook for put and mount blob
 2020-04-28 09:35:57 +08:00
He Weiwei
b1c9d452ce feat(quota,webhook): send quota webhook for put and mount blob 
Closes #11712

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-28 00:59:16 +00:00
Daniel Jiang
87f006d3a9
Merge pull request #11755 from reasonerjt/token-service-core-url 
v2 auth middleware handles the ping request from internal
 2020-04-27 15:04:10 +08:00
stonezdj(Daojun Zhang)
fb90bc23f2
Merge pull request #11757 from heww/fix-issue-11692 
feat(scan): support to scan artifact automatic after it pushed
 2020-04-27 12:08:06 +08:00
Daniel Jiang
ef008fd4cf
Merge pull request #11751 from wy65701436/fixes-11744 
fix return code on getting non exist manifest
 2020-04-27 12:06:37 +08:00
Daniel Jiang
fe587d0cc8 v2 auth middleware handles the ping request from internal 
When scanner like trivy handles the auth flow to pull image, it pings
the /v2 and access the token service url in response body, by default it
will be external endpoint of Harbor.
There will be problem when Harbor is deployed on a single node with hairpinning not
supported.

This commit makes sure the address of token service in the challenge is
internal url of core component when the request is from internal.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-04-27 00:54:49 +08:00
He Weiwei
bc1f7b8079 feat(scan): support to scan artifact automatic after it pushed 
Closes #11692

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-26 15:56:58 +00:00
wang yan
c4c279089a fix return code on getting non exist manifest 
It's found by conformance test, it should be 404 instead of 500 when to get a non exist manifest

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-26 23:31:11 +08:00
AllForNothing
fd65520fa0 dESTInation namespace should support slash 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-26 17:24:15 +08:00
Steven Zou
06d955c6e3
Merge pull request #11737 from steven-zou/fix/issue_#11720 
fix(jobservice):fix job stats NOT_FOUND issue
 2020-04-26 16:56:25 +08:00
Wang Yan
8d5fc4c752
Fix delete repository event issue (#11722) 
In the delete repository scenario, the repository has already been removed from the DB, the repository cannot be retrived.
Just let the creation time as empty.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-26 16:51:13 +08:00
Steven Zou
0ccea49c18 fix(jobservice):fix job stats NOT_FOUND issue 
Signed-off-by: Steven Zou <szou@vmware.com>
 2020-04-26 16:23:10 +08:00
Daniel Jiang
c28f6bd6ff
Merge pull request #11738 from bitsf/retention_not_retry 
fix(retention) not retry retention job
 2020-04-26 15:34:34 +08:00
Ziming Zhang
97a7a6dc35 fix(retention) fix retention repository with slash 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-04-26 15:07:25 +08:00
Ziming Zhang
601ce08660 fix(retention) not retry retention job 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-04-26 14:33:21 +08:00
Ziming Zhang
257924856c fix(retention) fix empty pull time log 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-04-26 11:56:26 +08:00
jwangyangls
0460e40a28
Merge pull request #11680 from jwangyangls/200421-fix-issue-2.0 
[Fixed] Fix issue #11334 #11252 #11191
 2020-04-24 17:46:22 +08:00
AllForNothing
432f5512d8 Fix bugs for testing kick 3 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-24 17:11:34 +08:00
Yogi_Wang
7bb7c2c868 [Fixed] Fix issue #11334 #11252 #11191 
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-04-24 16:38:59 +08:00
Daniel Jiang
447ec13527 Allow sys admin to call /c/UserExists 
The commit fix a regression introduced by #11672 which impacts admin
adding new users.
When admin is creating new users, /c/UserExists is called by UI. We must
allow it called by admin when self-registration is turned off.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-04-23 11:40:54 +08:00
Will Sun
7d784ef02f
Merge pull request #11674 from AllForNothing/more-ut 
Add more UT
 2020-04-22 17:13:20 +08:00
Daniel Jiang
4c41ac81d9
Merge pull request #11672 from reasonerjt/restrict-userexists 
Enable userExists only when self-registration is turned on.
 2020-04-21 13:19:20 +08:00
mmpei
4b6196a00d
fix slack rate limit issue (#11623) 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2020-04-21 11:44:58 +08:00
AllForNothing
0c5d3cc899 Add more UT 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-21 09:54:32 +08:00
Daniel Jiang
201955c2fb Enable userExists only when self-registration is turned on. 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-04-21 00:17:12 +08:00
Wang Yan
ff2a7e61c9
fix catalog api issue (#11666) 
The v2 catalog API needs to filter out the empty repository and the repository which artifacts are all with no tags.

1，In v2.0.0, Harbor does not delete repository even there is no artifact, it's different with v1.10.0
2, Compares with docker distribution, it doesn't return the respository with untagged images.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-20 23:37:16 +08:00
DQ
e0b98685f3 Add comments for new tls transport 
To explain why use this to avoid replication hang forever issue

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-20 19:19:15 +08:00
Yogi_Wang
cd98a7a9b6 [Fixed] Fix issue when delete signed tag 
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-04-17 16:04:16 +08:00
Steven Zou
3b0b9f3c7d
Merge pull request #11645 from ninjadq/avoid_default_transport_https_bug 
Fix: Default Transport HTTP2 related hang issue
 2020-04-17 15:44:52 +08:00
jwangyangls
981cae9f11
Merge pull request #11648 from jwangyangls/add-sign-pop-when-delete 
[feat] Add pop up command when delete signed tag
 2020-04-17 13:14:19 +08:00
Yogi_Wang
6f54262889 [feat] Add pop up command when delete signed tag 
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-04-17 12:45:57 +08:00
DQ
53111d1d16 Fix: Default Transport HTTP2 related hang issue 
Create a transport the same as default except forceattempthttp2

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-17 11:36:02 +08:00
Wenkai Yin(尹文开)
e3cbfac0cc
Fix bug for replicating chart triggered by event (#11578) 
Use the new event model to fix bug for replicating chart triggered by event

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-04-16 16:15:21 +08:00
Wang Yan
8a0e8627ff
replace pkg errors with lib errors (#11605) 
Fixes #9704

As we do want to unify error handling, so just decreprates pkg errors, use lib/errors instead for Harbor internal used errors model.

1, The lib/errors can cover all of funcs of pkg/errors, and also it has code attribute to define the http return value.
2, lib/errors can give a OCI standard error format, like {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized"}]}

If you'd like to use pkg/errors, use lib/errors instead. If it cannot meet your request, enhance it.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-15 22:41:45 +08:00
Will Sun
205e3a969f
Merge pull request #11621 from AllForNothing/gc-ui 
Modify gc ui
 2020-04-15 22:01:59 +08:00
He Weiwei
385aaac00d
Merge pull request #11620 from heww/fix-issue-11524 
feat(scanner): make Clair and Trivy as reserved name for scanners
 2020-04-15 15:21:35 +08:00
Wang Yan
9bc96dd97a
add MarshalJSON func for lib errs (#11614) 
Customize the json output with message with err.Error(). Otherwise, the wrappged message will be lost
in the final errors object.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-15 11:30:14 +08:00
AllForNothing
a337dcf517 Modify gc ui 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-15 11:06:17 +08:00
He Weiwei
f5487479dd feat(scanner): make Clair and Trivy as reserved name for scanners 
Closes #11524

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-15 02:26:02 +00:00
Wang Yan
c6860ac35f
handle stop signal in GC job (#11612) 
Stop GC job in the init if receives the stop signal

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-14 18:59:03 +08:00
Wenkai Yin(尹文开)
2d618370ae
Merge pull request #11588 from wy65701436/trace-error 
update trace level to error
 2020-04-14 18:30:21 +08:00
Qian Deng
95d7c9382b
Merge pull request #11592 from ninjadq/min_version_tls_to_12 
Min version tls to 12
 2020-04-14 18:12:55 +08:00
Will Sun
ad60bdddf8
Merge pull request #11591 from AllForNothing/nightly-untag 
Add new nightly case for GC untagged images
 2020-04-14 17:34:58 +08:00
Will Sun
faf554f642
Merge pull request #11587 from AllForNothing/sort-name 
Fix repo list sorting and filtering bug
 2020-04-14 17:34:39 +08:00
wang yan
ff2a6c7a01 add warning to registry binary name 
Fixes #11606

As we DO NOT want to user to execute GC in the container, rename it and append the warning message.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-14 15:16:50 +08:00
wang yan
7622c17817 update trace level to error 
Use level error instead to log trace

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-14 15:01:52 +08:00
Steven Zou
adb305e4e1
feat(job):enable priority option (#11608) 
- priority option is supported when doing job registration
- the priority is defined by a unique priority sampler
- the default priority is 1000 (max is 10000)

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-04-14 14:54:44 +08:00
jwangyangls
fb3da503ea
Merge pull request #11584 from jwangyangls/add-docker-pull-push-command 
[feat] Add pull/push command in repo
 2020-04-14 10:05:39 +08:00
AllForNothing
46320641b4 Add new nightly case for GC untagged images 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-14 09:25:06 +08:00
Wenkai Yin(尹文开)
f972f2989c
Close the reponse body after reading data (#11594) 
Close the reponse body after reading data

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-04-14 01:19:40 +08:00
Wenkai Yin(尹文开)
4b4091b217
Merge pull request #11599 from wy65701436/revise-base-error 
update internal error output format
 2020-04-13 20:07:05 +08:00
wang yan
269d0b9f9d update internal error output format 
1, remove the code from output
2, output format is same as pkg/errors, it's easy to migrate
3, add UT

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-13 19:29:55 +08:00
DQ
b3db293091 TLS update min version and cipher suits 
min version set to tls 1.2
suit only use ecdhe and strenth above 256

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-13 18:13:30 +08:00
Steven Zou
3ad5b2ba06
fix(job currency):introduce max corrency of job (#11589) 
- update Job interface to introdcue MaxCurrency method for declaring the max currency of the specified job
- change the downstream jobs to implement the new interface method
  - GC and sample jobs are set to 1
  - other jobs are set to 0 that means unlimited
- add max currency optiot when doing job registration
- resolve issue #11586
  - probably resolve issue #11281
  - resolve issue #11570

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-04-13 18:07:54 +08:00
Wenkai Yin
7553845b4d Remove the duplicated const definition 
Remove the duplicated const definition for artifact type

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-04-13 17:02:23 +08:00
He Weiwei
49c9e4f696
Merge pull request #11585 from heww/cleanup-quota 
refactor(quota): cleanup code for quota
 2020-04-13 15:11:17 +08:00
Yogi_Wang
9e1bdc88e6 [feat] Add pull/push command in repo 
1.add pull/push command in repo;
2.move annotations from artifact list to artifact summary

Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-04-13 13:11:33 +08:00
AllForNothing
e5e39c03c5 Fix repo list sorting and filtering bug 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-04-13 12:24:46 +08:00
Will Sun
009662a317
Merge pull request #11426 from AllForNothing/nightly-tag 
Add new nightly cases for tag CRUD
 2020-04-13 12:15:18 +08:00
He Weiwei
0b87eaf039
Merge pull request #11505 from heww/revert-registry-authorization-type-support 
feat(scan): revert bearer token support for scanner
 2020-04-13 11:19:02 +08:00
He Weiwei
c0349da812 refactor(quota): cleanup code for quota 
1. Remove `common/quota` package.
2. Remove functions about quota in `common/dao` package.
3. Move `Quota` and `QuotaUsage` models from `common/models` to
`pkg/quota/dao`.
4. Add `Count` and `List` methods to `quota.Controller`.
5. Use `quota.Controller` to implement quota APIs.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-12 16:16:06 +00:00
Daniel Jiang
93f316ccfe
Merge pull request #11582 from heww/fix-issue-11564 
fix(blob): delete project blob with project_id
 2020-04-12 18:00:54 +08:00
He Weiwei
c585e22d18 fix(blob): delete project blob with project_id 
Closes #11564

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-12 04:23:18 +00:00
Wang Yan
740b1e46b4
fix artifact trash filter issue (#11575) 
fixes #11533

GC jobs will use the filter results to call registry API to delete manifest.

In the current imple, the filter function in some case does not return the deleted artifact as it's using digest as the filter condition.

Like: If one artifact is deleted, but there is another project/repo has a image with same digest with the deleted one, filter func will
not mark the deleted artifact as candidate. It results in, GC job does not call API to remove the manifest.

To fix it, update the filter to use both digest and repository name to filter candidate.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-12 03:11:49 +08:00