Commit Graph

10167 Commits

Author SHA1 Message Date
stonezdj
670a94835b Cache manifest list for proxy cache
Fixes #13566: Quota of dockerhub is still used in v2.1.1 after the image is cached
Cache manifest list in redis cache.
Trade off between efficiency and data integrity, it might cause the proxy cache return the full content of a manifest list instead of the actual manifest list saved in the Harbor storage, which is a part of the manifest list. but this change doesn't break any /v2/ API, just caches full manifest list.

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-12-15 11:30:30 +08:00
He Weiwei
10e4350360
test(cache): increase sleep time to fix ut (#13761)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 19:22:42 +08:00
Wenkai Yin(尹文开)
fd900889c1
Merge pull request #13729 from ywk253100/201210_fix
Tiny fixes for task manager
2020-12-14 18:41:46 +08:00
Wenkai Yin(尹文开)
c6bfa04661
Merge pull request #13759 from bitsf/fix_tencenttcr_getAdapterInfo
fix_tencenttcr_getAdapterInfo
2020-12-14 17:30:22 +08:00
Wenkai Yin
69808f033e Tiny fixes for task manager
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-12-14 17:26:32 +08:00
Will Sun
fff1bbe8d9
Merge pull request #13757 from AllForNothing/new-auth-mode
Add admin groups parameter to http auth
2020-12-14 17:02:24 +08:00
AllForNothing
11b706adeb Add admin groups parameter to http auth
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-12-14 16:02:34 +08:00
Ziming Zhang
d8b5746df1 fix_tencenttcr_getAdapterInfo
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-12-14 16:02:19 +08:00
He Weiwei
af24a073dc
feat(api): support project name in the path of apis (#13744)
Support project name in the path of projects and robotsV1 APIs.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 15:48:52 +08:00
Will Sun
949379f7bc
Merge pull request #13755 from AllForNothing/new-registry
Improve regitries UI to cover new registries
2020-12-14 14:45:23 +08:00
AllForNothing
8de8122119 Improve regitries UI to cover new registries
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-12-14 14:07:36 +08:00
He Weiwei
08580f9fec
refactor(scan): refactor scan/scan all job to task manager (#13684)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 13:34:35 +08:00
DQ
19e8527cc1 Fix log level issue in registry
1. fix level issue in registry.jinja
2. add log level to registryctl

Signed-off-by: DQ <dengq@vmware.com>
2020-12-14 11:52:42 +08:00
Will Sun
90f9bea965
Merge pull request #13752 from AllForNothing/improve-robot
Improve refresh secret page for robot account UI
2020-12-14 11:13:10 +08:00
Wenkai Yin(尹文开)
6569016d35
Merge pull request #13139 from wy65701436/migrate-gc
Migrate gc to task manager
2020-12-14 10:43:44 +08:00
Wang Yan
39e1a4f2b4 add extra attributes in the schedule table
Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-14 02:28:52 +08:00
danfengliu
0a38ae1843
Merge pull request #13260 from danfengliu/Migrate-gitlab-replication-test-to-nightly
Migrate gitlab replication test to nightly
2020-12-11 19:29:31 +08:00
danfengliu
3659abc62a Migrate gitlab replication test to nightly
Add 2 gitlab pull/push replication test scripts in replication pipeline.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-11 18:04:46 +08:00
Wenkai Yin(尹文开)
43104ab0b9
Merge pull request #13724 from reasonerjt/http-auth-admin-grp
Support admin group in http authproxy
2020-12-11 13:06:26 +08:00
Daniel Jiang
4ff1c99ea0
Merge pull request #13539 from ninjadq/add_exporter_for_harbor
Add exporter for harbor
2020-12-11 12:52:03 +08:00
Daniel Jiang
fc94da54c1
Merge pull request #13728 from heww/fix-ut-for-get-reserved-resources
test(quota): increase sleep time to fix fail of test
2020-12-11 12:49:01 +08:00
Wenkai Yin(尹文开)
a4f21b1d85
Merge pull request #13694 from kofj/ghcr-provider
GHCR Provider
2020-12-11 09:41:10 +08:00
He Weiwei
2e09e54b38 test(quota): increase sleep time to fix fail of test
Increase the sleep time to fix the fail in TestGetReservedResources of
quota

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-10 11:23:47 +00:00
wang yan
1bb79d402d update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 16:08:52 +08:00
Daniel Jiang
60e3668d43 Support admin group in http authproxy
This commit adds admin_groups into the configuration of http_auth
settings, it's a string in the form of "group1, group2".  If the token
review result shows the user is in one of the groups in the setting he
will have the administrator role in Harbor.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-10 15:57:15 +08:00
Wang Yan
dba5522d0b Migrate to task manager (#129)
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 14:00:33 +08:00
sluetze
5584e55070 fixed small typos in variable names
Signed-off-by: sluetze <13255307+sluetze@users.noreply.github.com>
2020-12-09 15:24:22 +01:00
DQ
85bf88c74e Add test case exporter
- cache ut
- cli ut
- project collector ut
- system collector ut

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 21:47:29 +08:00
DQ
d95f22448c Add cache for exporter
Add timed cache for exporter
default cache time is 30s, cleanup job run every 4 hours

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 21:22:40 +08:00
DQ
f0db193895 Add prepare file for exporter
prepare env for exporter

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 21:22:13 +08:00
DQ
dc0047c48c Add build script for exporter
- Add dockerfile
- update makefile

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 20:42:21 +08:00
DQ
ade69e20ef Fix typo
Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 20:42:21 +08:00
DQ
e6b53593ee Add exporter to harbor
- add main file
- add pkg file
- update go.mod
- update vendor

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 20:42:14 +08:00
Steven Zou
c12664122c fix(api):update stop job api
- update StopJob() of basic worker
- update UnSchedule() of basic scheduler
- update the policy store to get more data

fix #13599 , fix #13597

Signed-off-by: Steven Zou <szou@vmware.com>
2020-12-09 20:42:01 +08:00
Steven Zou
4acb708938
Merge pull request #13705 from steven-zou/fix/js_api
fix(api):update stop job api
2020-12-09 13:41:58 +08:00
Will Sun
13900df810
Merge pull request #13710 from AllForNothing/ut-case
Fix UI UT failure
2020-12-09 12:13:34 +08:00
danfengliu
ad7581d9b3 Migrate gitlab replication test to nightly
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-09 12:00:38 +08:00
AllForNothing
2f2e85577d Fix UI UT failure
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-12-09 11:33:55 +08:00
fanjiankong
2898117411 GHCR Provider
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-12-09 09:54:04 +08:00
Steven Zou
85783f3c0c fix(api):update stop job api
- update StopJob() of basic worker
- update UnSchedule() of basic scheduler
- update the policy store to get more data

fix #13599 , fix #13597

Signed-off-by: Steven Zou <szou@vmware.com>
2020-12-08 23:15:05 +08:00
He Weiwei
e92674a42a
feat: add cache library and enable it in config manager (#13525)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-08 17:40:03 +08:00
疯魔慕薇
9f8a743da9 Tecent TCR Provider
1. Docker image registry.
2. Helm chart registry.

Signed-off-by: 疯魔慕薇 <kfanjian@gmail.com>
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-12-08 15:44:18 +08:00
AllForNothing
7cddb5845f Improve refresh secret page for robot account UI
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-12-08 13:42:32 +08:00
stonezdj(Daojun Zhang)
ec2f251d63
Merge pull request #13685 from wy65701436/robot-name-conflict
fix robot name conflicate issue
2020-12-07 11:18:49 +08:00
Wenkai Yin(尹文开)
65b6ae08bb
Merge pull request #13618 from mmpei/official-feature-artifact-hub
Support replicate from artifact hub
2020-12-07 10:47:05 +08:00
Daniel Jiang
fef5317aef
Merge pull request #13382 from flaviodsr/fix_core_init
Fix deadlock on harbor-core initialization
2020-12-04 19:51:56 +08:00
danfengliu
56a35437b4
Change image source in e2e pytest (#13640)
Change source of most of test image samples from docker-hub to local building ones, so it will cost less docker-hub pull requests.
And some of cases like push cnab, they have to use docker-hub, but image samples in cnab test will cost 17 quotas, in this PR, we
replace those samples, now cnab case will cost 6 quotas.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-04 02:28:29 -08:00
danfengliu
ff50e2363d
Move proxy cache e2e test into a new CI job (#13690)
CI job timeout is 60 minutes, but proxy cache case timeout is 20 minutes, once
proxy cache case is timeout, it will reach CI job timeout, then cause CI job terminated
by timeout without running rest of test steps, expecially uploading useful harbor logs,
so move this test into a new clean job for saving time and debugging.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-04 02:27:59 -08:00
Will Sun
0840a10752
Merge pull request #13615 from AllForNothing/too-much-tag
Add limitation to artifact tags
2020-12-04 15:47:37 +08:00
Will Sun
cf9b73d9a0
Merge pull request #13662 from AllForNothing/robot
Add system robot account UI
2020-12-04 15:37:00 +08:00