Commit Graph

4317 Commits

Author SHA1 Message Date
Wenkai Yin
9493611666 Don't return the error detail back to the client when adding registry
Don't return the error detail back to the client when adding registry to avoid security issue

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-17 11:58:08 +08:00
Ziming Zhang
e7c89ce1d9 fix SWR replication adapter pull-based
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-07-16 19:19:18 +08:00
Daniel Jiang
840aa86dfa Provide secret manager for proxy cache project
This commit provides the secret manager for proxy cache.
The secret is used for pushing blobs to local when it's proxied from
remote registry.
Each secret can be used only once and has a relatively short expiration
time.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-07-16 19:12:52 +08:00
Alvaro Iradier
81a7239c66 Better error handling
* Raise an internal error if username claim is not found, instead of just logging a warning
* Don't remove userInfoKey for session on error when it is not required
* Rename "OIDC Username Claim" to just "Username claim"

Signed-off-by: Alvaro Iradier <airadier@gmail.com>
2020-07-16 12:12:08 +02:00
Alvaro Iradier
6f88ff7429 Fix test suite and add test for userClaim
Signed-off-by: Alvaro Iradier <airadier@gmail.com>
2020-07-16 12:12:08 +02:00
Alvaro Iradier
714f989759 Add options for automatic onboarding and username claim
- Add an option in the UI to enable or disable the automatic user onboarding
- Add an option to specify the claim name where the username is retrieved from.

Signed-off-by: Alvaro Iradier <airadier@gmail.com>
2020-07-16 12:12:08 +02:00
Wang Yan
bad8f026fc
upgrade golang to v1.14.5 (#12489)
Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-16 16:20:54 +08:00
Wang Yan
d73265d10d
revise gc job to align non blocking gc (#12439)
two phases:
1, mark, select the gc candidates bases on the DB and mark them as status delete.
2, sweep, select the candidate and mark it as status deleting and remove it from backend and database.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-16 14:35:54 +08:00
Steven Zou
4d4a04fad4
Merge pull request #12478 from steven-zou/feat/read_pro_config
feat(p2p):enhance policy enforcer
2020-07-16 11:40:29 +08:00
Ted Guan
9e7edb7a6e
Fix for project metadata (#12410)
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-07-15 18:46:45 +08:00
Steven Zou
fcfde5a588 feat(p2p):enhance policy enforcer
Read security settings from the project configurations and override the preheat policy settings if necessary.

Check the project security settings and override the related settings in the policy if necessary.
NOTES: if the security settings (relevant with signature and vulnerability) are set at the project configuration,
they will have the highest priority and override the related settings of the preheat policy.
 e.g (use signature as an example, similar case to vulnerability severity part):
   if policy.signature = false and project.config.signature = true; then policy.signature = true
   if policy.signature = true and project.config.signature = true; then policy.signature = true
   if policy.signature = true and project.config.signature = false; then policy.signature = true
   if policy.signature = false and project.config.signature = false; then policy.signature = false

Signed-off-by: Steven Zou <szou@vmware.com>

Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-15 14:18:34 +08:00
He Weiwei
cadcd4b877
Merge pull request #12480 from heww/move-pkg-types
refactor(quota): move pkg/types to pkg/quota/types
2020-07-15 11:32:27 +08:00
Daniel Jiang
947eadaa72
Merge pull request #12440 from heww/remove-init-clair-db
refactor: remove initialization of clair db
2020-07-15 00:38:12 +08:00
He Weiwei
a22d803a95 refactor(quota): move pkg/types to pkg/quota/types
Closes #9664

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-14 14:28:53 +00:00
stonezdj
b9c861f3f1 Add disable push for proxy project
Add middleware for blob and manifest push operation

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-07-14 10:13:38 +08:00
疯魔慕薇
f187509a90
Merge pull request #12454 from chlins/feat/list-providers-at-project-level
feat(preheat): add list providers under project level handler
2020-07-14 08:17:53 +08:00
chlins
7322d0ac7c feat(preheat): add list providers under project level handler
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-13 21:53:50 +08:00
stonezdj
3abe77d6cb Add proxy cache feature
Update route to add proxy related middleware
Add proxy controller

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-07-13 21:18:43 +08:00
chlins
08bd46e125 feat: add preheat execution api handler
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-13 13:14:08 +08:00
chlins
2863e68718 feat: add task controller
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-13 13:02:24 +08:00
fanjiankong
a99aa21c8a Enable RBAC control in the preheat API
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-13 11:06:25 +08:00
AllForNothing
4ec919dfe6 Query preheat instance by name
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-07-11 17:41:56 +08:00
Will Sun
ec1ac6dbc8
Merge pull request #12344 from AllForNothing/p2p
Add P2p preheat distribution instance UI
2020-07-10 16:01:04 +08:00
Steven Zou
1dfc93c3f6
Merge pull request #12430 from kofj/preheat_and_healthcheck
Preheat and healthcheck
2020-07-10 15:17:34 +08:00
Steven Zou
fdff077ff0
Merge pull request #12445 from chlins/fix/preheat-instance-and-policy-name-validation
fix(preheat): validate instance/policy name and set unique name
2020-07-10 15:14:44 +08:00
chlins
38d14dff30 fix(preheat): validate instance/policy name, set unique filed and policy
manager adds parsePolicy

Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-10 10:29:47 +08:00
fanjiankong
080afbfe1b Add preheat APIs, handlers.
1. Manual preheat.
2. Instance health check.

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-10 09:48:35 +08:00
He Weiwei
039aef5356 refactor: remove initialization of clair db
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-09 15:26:14 +00:00
He Weiwei
9483559d18
Merge pull request #12433 from heww/fix-db-max-open-conns
fix(db): set max open conns of sql.DB manually
2020-07-09 17:58:14 +08:00
疯魔慕薇
5d7f757b7b
Merge pull request #12428 from mmpei/official-master-p2p-200708
Add P2P trigger event and handler
2020-07-09 15:54:24 +08:00
He Weiwei
e095958a27 fix(db): set max open conns of sql.DB manually
Due to the issues of beego v1.12.1 and v1.12.2, we set the max open conns
ourselves.

Closes #12403

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-09 03:35:41 +00:00
peimingming
65c5561032 Add P2P trigger event and handler
Signed-off-by: peimingming <peimingming@corp.netease.com>
2020-07-09 11:20:22 +08:00
fanjiankong
8a44ee400d Instance handler.
- Add logic of preheat instance methods without RBAC.

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-09 00:01:14 +08:00
Steven Zou
3b2934bf48
Merge pull request #12419 from chlins/feat/p2p-preheat-healthcheck-controller
feat(preheat): add healthcheck methods for p2p preheat controller
2020-07-08 23:49:26 +08:00
Will Sun
71e50bd364
Merge pull request #12420 from AllForNothing/proxy-from-api
Add query string for getting registry
2020-07-08 14:20:38 +08:00
Will Sun
96b2326e24
Merge pull request #12418 from goharbor/rename-master-role
Rename master role to maintainer
2020-07-08 11:02:24 +08:00
AllForNothing
2c422435d9 Add query string for getting registry
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-07-08 10:42:55 +08:00
chlins
b6cab91bfa feat(preheat): add healthcheck methods for p2p preheat controller
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-08 09:58:01 +08:00
Steven Zou
b87aa3b9d5
chore(preheat):add mock policy enforcer (#12414)
Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-08 09:45:14 +08:00
Will Sun
cd66899abb
Merge pull request #12417 from AllForNothing/rename-role
Rename project role for UI
2020-07-08 09:21:57 +08:00
Daniel Jiang
1637e6a588 Rename master role to maintainer
This commit rename the var name, text appearance, and swagger of "master" role
to "maintainer" role.
It only covers backend code.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-07-08 09:20:07 +08:00
AllForNothing
706c6bdb2b Rename project role
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-07-08 09:15:01 +08:00
Steven Zou
b56a49efe2
Merge pull request #12411 from kofj/instance_by_name
Get instance by name.
2020-07-07 23:20:01 +08:00
fanjiankong
3c1c799f0d Get instance by name.
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-07 22:06:38 +08:00
Wang Yan
c3baeac5ae
add time windows support in artifact trash (#12400)
support with time window to filter and delete item in artifact trash

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-07 20:13:08 +08:00
AllForNothing
d01ff31dc8 Add P2p preheat distribution instance UI
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-07-07 19:55:56 +08:00
stonezdj(Daojun Zhang)
6f4e8150d5
Merge pull request #12383 from ywk253100/200702_registry_api
Suport filtering registries by type in listing registry API
2020-07-07 14:21:54 +08:00
Wenkai Yin
02690d1d04 Suport filtering registries by type in listing registry API
Suport filtering registries by type in listing registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-07 10:30:46 +08:00
Steven Zou
642953b9ef
Merge pull request #12395 from chlins/feat/preheat-controller-policy-part
Feat/preheat controller policy part
2020-07-06 22:21:34 +08:00
chlins
37a00912b7 feat: add p2p preheat swagger yaml and implement preheat api policy handler
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-06 21:05:22 +08:00
jwangyangls
e498893777
Merge pull request #12391 from jwangyangls/fix-replication-repo
[fix] [replication] Success rate should be displayed as 0 when status is inprogress
2020-07-06 10:09:20 +08:00
chlins
254ea193fa feat: add p2p preheat controller policy releated logic
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-04 12:23:04 +08:00
Steven Zou
2f2c545a0f
Merge pull request #12392 from steven-zou/fix/remove_inst_provider_todo
fix(p2p):remove the provider manager related to
2020-07-03 23:59:55 +08:00
Will Sun
7dfab5858c
Merge pull request #12374 from AllForNothing/proxy-cache
Add proxy cache ui
2020-07-03 18:14:25 +08:00
Steven Zou
1ee3f00709 fix(p2p):remove the provider manager related to
- use real provider instance manager
- move mock insatnce manager to testing/pkg
- modify kraken deriver implementation to remove digest fetcher
- update related UT cases

Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-03 17:30:11 +08:00
Yogi_Wang
de4793c52d [fix] [replication] Success rate should be displayed as 0 when status is inprogress
1.[replication] sccess rate should be displayed as 0 when status is inprogress
2.[repository] fix get repo name error when the repo contains the project name field
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-07-03 16:19:18 +08:00
Steven Zou
c7a050b629
Merge pull request #12389 from chlins/fix/p2p-preheat-policy-count-total
fix: add count method of policy manager to replace list method return…
2020-07-03 13:50:34 +08:00
Steven Zou
f3fcb96570
Merge pull request #12335 from kofj/p2p_preheat_api
feat(preheat):add preheat api, controller and manager
2020-07-03 13:47:04 +08:00
chlins
ace21240a4 fix: add count method of policy manager to replace list method return wrong counts
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-03 11:59:32 +08:00
Wang Yan
e8784de5fe
support list blobs by update time (#12385)
Add support list blob with update time.
As introduces the time window in GC, it wants to list the blobs less than specific time.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-03 11:50:53 +08:00
Daniel Jiang
cf07ff0052
Merge pull request #12386 from wy65701436/delete-blob-not-found
fix return value for blob & manifest not found
2020-07-03 11:37:31 +08:00
Daniel Jiang
8252930083
Merge pull request #12356 from wy65701436/put-mf-mw
add middlware for put manifest
2020-07-03 11:30:25 +08:00
Daniel Jiang
650142e536
Merge pull request #12381 from wy65701436/fix-delete-manifest
fix delete manifest route issue
2020-07-03 11:29:06 +08:00
fanjiankong
a0c2d0ac9e feat(preheat):add preheat api, controller and manager
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-03 11:25:42 +08:00
AllForNothing
a13642c2af Add proxy cache ui
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-07-03 10:20:55 +08:00
wang yan
95bee9a0cc fix return value for blob & manifest not found
When to delete an non exist blob/manifest, the API has to return a 404 instead of 500

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-02 19:18:57 +08:00
wang yan
7d1507feaa update code arrording to the review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-02 14:57:04 +08:00
wang yan
67be511a85 add middlware for put manifest
The middleware is to handle manifest(blob) status in different push manifest situation, similar with blob

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-02 14:57:04 +08:00
Steven Zou
264bd02892
Merge pull request #12378 from steven-zou/feat/preheat_policy_enforcement
feat(preheat):implement policy enforcer
2020-07-02 14:30:25 +08:00
wang yan
12c92dbfce fix delete manifest route issue
The repository name contains blackslash, the mux router has to use the * to match the blackslash. Otherwise the caller(gc job) will get a 404.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-02 13:42:48 +08:00
Steven Zou
18137a5c55 feat(preheat):implement policy enforcer
- define policy enforcer interface
- implement the default enforcer
- registrer P2P preheat job to JS
- add the missing mock manager&controller in the src/testing pkg
- Add UT cases for enforcer
- fix #12285
- left one TODO: query provider instance by instance Manager

Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-02 11:33:11 +08:00
Wang Yan
57c72b7952
add get GC candidate (#12314)
* add get GC candidate

select non referenced blobs from table blob and exclude the ones in the time windows.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-30 20:41:17 +08:00
Chlins Zhang
5bfe82612a
Merge pull request #12342 from chlins/feat/p2p-preheat-policy-dao-and-manager
feat: add p2p preheat policy dao and manager(#12286)
2020-06-30 16:35:42 +08:00
chlins
15e4361d6e feat: add p2p preheat policy dao and manager(#12286)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-06-30 15:56:50 +08:00
Steven Zou
a06c03625d
Merge pull request #12349 from steven-zou/feat/p2p_policy_evaluator
feat(preheat):add artifact filters for preheat policy
2020-06-30 14:56:58 +08:00
Steven Zou
d8e88ef5bc feat(preheat):add artifact filters for preheat policy
- add new selector based on vulnerability severity criteria
- add new selector based on signature(signed) criteria
- do change to the select factory method definition
- do changes to selector.Candidate model
- add preheat policy filter interface and default implementation
- add UT cases to cover new code

Signed-off-by: Steven Zou <szou@vmware.com>

misspelling
2020-06-30 10:48:21 +08:00
Wang Yan
468ba50a7e
handle blob status chanage in put blob middlware (#12315)
* handle blob status chanage in put blob middlware

After blob is uploaded success, the middleware will update the blob status accordingly.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-29 16:38:47 +08:00
Daniel Jiang
7d50a6aab6
Merge pull request #12281 from pcfens/upgrade_aws_module
Upgrade aws-sdk-go to 1.32.5 for OIDC support
2020-06-29 16:35:11 +08:00
Steven Zou
8e2c334b43
Merge pull request #12338 from steven-zou/feat/p2p_preheat_job
feat(preheat):add preheat job
2020-06-29 12:27:05 +08:00
Steven Zou
6424480f37 feat(preheat):add preheat job
- implement the preheat job
- add relevant UT case
- fix issue #12285

Signed-off-by: Steven Zou <szou@vmware.com>
2020-06-28 23:08:02 +08:00
chlins
dd9e97f755 feat: add p2p preheat policy model
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-06-28 10:46:50 +08:00
Steven Zou
df86ae1ad0 feat(preheat):add preheat drivers
- define preheat driver interface
- implement dragonfly driver
- implememt kraken driver
- add related UT cases with testify framework
- fix #10870 #10871
- some code are picked up from the original P2P feat branch

Signed-off-by: Steven Zou <szou@vmware.com>
2020-06-25 23:39:34 +08:00
AllForNothing
fff6f7529a Replace all whitelist with allowlist
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-06-24 16:17:17 +08:00
Phil Fenstermacher
33069e0f98
Upgrade aws-sdk-go
Versions of the Go AWS SDK newer than 1.23.13 support OIDC in EKS.
Running Harbor on EKS doesn't require keys in a configmap for the
registry to authenticate to S3 when using the newer library.

Signed-off-by: Phil Fenstermacher <pcfens@wm.edu>
2020-06-23 09:04:12 -04:00
Wenkai Yin(尹文开)
202916e396
Merge pull request #12280 from ywk253100/200616_task_manager
Implement task and execution manager
2020-06-23 18:44:44 +08:00
Wenkai Yin
ea20690264 Implement task and execution manager
Implement task and execution manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-23 17:10:58 +08:00
wang yan
0e175017aa continue updating code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 14:42:28 +08:00
wang yan
446739f967 rebase with latest source code
Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
wang yan
c10467eb36 continue refactor
Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
Wang Yan
de504993ad update blob controller & manager
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
Wang Yan
58b7242a25
move send error to source lib (#12175)
* move send error to source lib

Move the sendError into library in case the cycle dependency as regsitry and core are now the consumers.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-19 01:04:50 +08:00
Steven Zou
ffd889f82a
Merge pull request #12239 from ywk253100/200615_task_manager_dao
Implement execution/task DAO for task manager
2020-06-17 17:56:00 +08:00
stonezdj(Daojun Zhang)
91bff55b66
Merge pull request #12214 from stonezdj/20200611_add_proxyservice_secret
Add temporary secret for harbor proxy service
2020-06-17 10:46:13 +08:00
Wenkai Yin
5a1827768a Implement execution/task DAO for task manager
Implement execution/task DAO for task manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-16 10:24:58 +08:00
Wenkai Yin
127988b70c Define the task manager interface and data model
Define the task manager interface and data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-15 18:42:09 +08:00
stonezdj(Daojun Zhang)
56ca3dddda
Merge pull request #12203 from ywk253100/200610_proxy_cache_api
Update creating project API to support proxy cache project
2020-06-15 15:24:08 +08:00
stonezdj
82f59cb760 Add temporary secret for harbor proxy service
Use GenerateRandomStringWithLength function to generate secret for harbor_proxyservice
Add harbor-proxyservice secret used by proxy service

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-06-15 14:43:43 +08:00
Daniel Jiang
373ac25b9b
Merge pull request #12195 from reasonerjt/fix-post-user-403
Makes api/users return 401 for request not authenticated
2020-06-11 15:51:52 +08:00
Steven Zou
64f03cc7fe
Merge pull request #12079 from ninjadq/chartrepo_enumeration_master
Fix: chart repo response code inconsistent
2020-06-11 15:08:14 +08:00
Wenkai Yin(尹文开)
b08dfd57b6
Merge pull request #12127 from ywk253100/200603_label
Remove the label from artifacts when deleting a label
2020-06-11 14:49:14 +08:00