Commit Graph

920 Commits

Author SHA1 Message Date
Wang Yan
60427e7187
build base image in CI (#12750)
In git action, use the local build base images instead of pulling from docker hub.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-18 14:11:52 +08:00
He Weiwei
f659523f50 fix(scan): fix the permission checking for artifact scanning
Closes #12778

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-17 09:55:38 +00:00
He Weiwei
f309896f2f refactor(api): generate project apis by go-swagger
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-15 16:10:57 +00:00
danfengliu
5cb239af1e
Fix echo parameter issue (#12775)
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-14 10:23:58 +08:00
danfengliu
e1eac2b4d0
Merge pull request #12688 from danfengliu/fix-upgrade-test-wrong-keyword-issue
Fix wrong keyword issue in upgrade test
2020-08-14 10:11:37 +08:00
Daniel Jiang
83de495147
Merge pull request #12772 from reasonerjt/update-root-cert-test
Refresh CA cert
2020-08-13 18:22:19 +08:00
Daniel Jiang
376929d58e Refresh CA cert
This commit refresh the harbor_ca.crt and revert the changes to generate a new CA each time in the test script.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-13 08:43:07 +00:00
danfengliu
8f116d7b38 Add exec priviledge to integration script
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-13 08:15:42 +00:00
danfengliu
601d1ef0eb Add git/golang/dns config in drone image
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-13 07:24:54 +00:00
danfengliu
9a2188641d Upgrade python and robot for drone build
1.Separate Dockerfile to drone and nightly e2e, one Dockerfile should cover
 both scenarios, it will be resolved later;
2.Fix docker api err type issue, API error has no message attibute;
3.Upgrade docker in nightly e2e images.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-12 10:26:12 +00:00
danfengliu
ab7767f539 Fix wrong keyword issue in upgrade test
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-10 14:47:38 +00:00
danfengliu
4968b6619b Recovery drone build script
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-10 14:23:44 +00:00
danfengliu
859ff2e1ef Modify e2e image for drone build
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-10 11:40:26 +00:00
danfengliu
07f5f549d0 Fix LDAP API test issue by upgrading python
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-09 06:37:28 +00:00
danfengliu
0521f9eed5
Merge pull request #12626 from danfengliu/upgrade-robotframework-and-libs
Upgrade robot-framework and Selenim library
2020-08-06 16:01:35 +08:00
danfengliu
3b8a2890f9 Upgrade robot-framework and Selenium library
1. Upgrade robot-framework to 3.1 and Selenium library to 4.4.0.
2. Fix a registry issue for clear filter text input.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-06 07:25:53 +00:00
Flávio Ramalho
3d3fabbd20
Make notary-remove-image-signature more flexible (#12536)
Instead of explicitly defining the expect path to /usr/local/bin/expect,
use the env command, so the expect executable is searched for and launched
from wherever it is first found.

Signed-off-by: Flávio Ramalho <framalho@suse.com>
2020-07-31 15:49:45 +08:00
danfengliu
da662f52d5
Merge pull request #12607 from danfengliu/add-gcs-s3-push-based-replic-test
Add GCS&S3 push-based replicatiton tests
2020-07-30 09:40:21 +08:00
danfengliu
8328e95b63 Add GCS&S3 push-based replicatiton tests
1. Add GCS push-based replicatiton tests;
   There is an issue against it, so it's commented out;
2. Add S3 push-based replicatiton tests;
3. Add filter registry keyword;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-29 12:44:49 +00:00
danfengliu
658cfcf3a2
Merge pull request #12427 from danfengliu/add-checkpoint-to-gc-untag-case
Add checkpoint of no deletion for untagged image
2020-07-28 21:27:26 +08:00
danfengliu
5d3080c2c2 Add checkpoint of no deletion for untagged image
1. Add checkpoint of no deletion for untagged image
2. rebase master.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-28 15:21:28 +08:00
stonezdj
7d97ae6ea2 Set LDAP groupname when PopulateGroup
Search ldap group name with default ldap group attribute name
fixes #10940

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-07-28 10:27:15 +08:00
danfengliu
f030de59a5
Merge pull request #12537 from danfengliu/add-oidc-user-onboard-test
Add OIDC user onboard test in nightly
2020-07-27 14:39:04 +08:00
danfengliu
8a691c5df1 Add OIDC user onboard test in nightly
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-27 13:56:18 +08:00
Wang Yan
eeb8fca255
add debugging env for GC time window (#12528)
* add debugging env for GC time window

For debugging, the tester/users wants to run GC to delete the removed artifact immediately instead of waitting for two hours, add the env(GC_BLOB_TIME_WINDOW) to meet this.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-22 11:09:01 +08:00
danfengliu
4bdcc1fa7d
Merge pull request #12515 from danfengliu/fix-replication-robot-script
Fix issues in replication tests
2020-07-20 16:19:19 +08:00
danfengliu
e20cb51d6b Fix issues in replication tests
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-20 09:42:55 +08:00
Ziming Zhang
8857e89e40 feature(redis) support redis sentinel
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-07-19 21:19:03 +08:00
danfengliu
af3a638980
Merge pull request #12498 from danfengliu/add-push-based-replication-test
Add push-based replication test in nightly
2020-07-17 11:01:38 +08:00
danfengliu
aa43afb601 Add push-based replication test in nightly
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-16 19:50:42 +08:00
Wang Yan
bad8f026fc
upgrade golang to v1.14.5 (#12489)
Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-16 16:20:54 +08:00
Wang Yan
d73265d10d
revise gc job to align non blocking gc (#12439)
two phases:
1, mark, select the gc candidates bases on the DB and mark them as status delete.
2, sweep, select the candidate and mark it as status deleting and remove it from backend and database.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-16 14:35:54 +08:00
AllForNothing
706c6bdb2b Rename project role
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-07-08 09:15:01 +08:00
danfengliu
0cb378d8fa Verify error message when deleting signed image
1. Verify error message when deleting signed image;
2. Add test case - Replication From Dockerhub;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-06 10:56:31 +08:00
AllForNothing
fff6f7529a Replace all whitelist with allowlist
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-06-24 16:17:17 +08:00
danfengliu
a8f159728b Add API E2E pytest of GC with untag flag enabled
1. Fix issue of keyword Go Into Repo, the verification logic could be more strict;
2. Add API E2E pytest of GC with untag flag enabled;
3. Add sleep in test_user_view_logs.py for delete log ocurred;
4. Test Case - Tag CRUD is not stable. Although add button was clicked, but the tag was'nt added successfully.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-23 14:48:04 +08:00
danfengliu
c5e5e9ec9f
Merge pull request #12234 from danfengliu/fix-keyword-go-into-repo
Add helm and helm3 CLI py-test
2020-06-16 17:28:47 +08:00
danfengliu
67ccc98c58 Add helm and helm3 CLI py-test
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-16 16:27:31 +08:00
danfengliu
1948a3ef66 Fix nightly issue caused by GC and CVE UI changed
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-13 20:28:27 +08:00
danfengliu
4b5080d44e Add singularity py-test
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-10 11:09:41 +08:00
Wang Yan
c993103e01
enable storage driver of gcs and oss (#12180)
Add go build tags for gcs and oss, otherwise these drivers cannot be registered and the error "StorageDriver is not regsited: GCS" will raise on registryctl launch under the setting of GCS storage.

These build tags are designed in the distribution, just refer to https://github.com/docker/distribution/blob/release/2.7/registry/storage/driver/gcs/gcs.go#L13

Pin the google cloud API to a old version is because distribution depends on it, otherwise go mode will use v0.17.0 that go-migrate is using as the dependency version, but this version will break the compile process with following error:
harbor/pkg/mod/google.golang.org/cloud@v0.0.0-20151119220103-975617b05ea8/storage/acl.go:65:16: invalid type assertion: v.(map[string]<inter>) (non-interface type *storage.ObjectAccessControl on left)
that's bacause another dependency google.golang.org/cloud requires the pinned version of google.golang.org/api.

The pinned package should be removed once https://github.com/docker/distribution/pull/3019 is merged, and distribution ships their v2.8.0

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-09 00:57:27 +08:00
danfengliu
e020b90bf0 add oras cli py-test
1. Add oras cli py-test;
2. Add env for notary url, allow to input different notary port instead of solid 4443;
3. Add retry for keyword Cannot Pull Image and make it longer during retry.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-08 11:10:07 +08:00
danfengliu
3f2850ffdf Scan result behavior changed after trivy upgrade
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-04 16:25:21 +08:00
danfengliu
ceaa0a57b3 Add replication schedule test
1. Add replication schedule test;
2. Add some sleep after project creation before push image to avoid push image v2 error.
   It fixed issue# 12094: Scan image vmware/photon:1.0 return unsupported occasionally.
3. Add some sleep in test_user_view_logs.py, can't get operation logs sometimes.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-02 15:51:10 +08:00
danfengliu
a732292941
Merge pull request #12002 from danfengliu/add-retry-for-go-into-index-2
Add retry for keyword go into index
2020-05-29 10:01:20 +08:00
danfengliu
5d8a308158 Add retry for keyword go into index
1. Add retry for keyword go into index;
2. Add retry for keyword go into project.
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-05-27 08:55:58 +00:00
danfengliu
ba48e55e4e Move docker prune cmd outside of api common install shell script
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-05-22 04:06:25 +00:00
danfengliu
a07cc7bb2a
Merge pull request #11967 from danfengliu/fix-nightly-log-verification
Modify nightly quotas error message verification
2020-05-20 14:38:37 +08:00
danfengliu
2b9a01ce59
Merge pull request #11963 from danfengliu/add-schedule-test-in-ui-test
Add Scan schedule test in nightly
2020-05-20 14:36:29 +08:00
danfengliu
b4a54adfeb Add Scan schedule test in nightly
1. Add Scan schedule test in nightly;
2. Remove redundancy steps when create a new project;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-05-20 12:16:33 +08:00
danfengliu
2fb90cf258 Modify nightly quotas error message verification
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-05-19 14:59:21 +08:00
wang yan
65f2abcb39 enable full set of conformance test
1, enable test of Content Discovery
2, enable test of Content Management

Signed-off-by: wang yan <wangyan@vmware.com>
2020-05-15 14:12:34 +08:00
danfengliu
ac19f02bf5 Enhance verificaton for tag retention pytest
1. Modify scan related keywords, some of keywords are not effective.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-05-12 16:41:47 +08:00
danfengliu
a8cfb40cc2
Merge pull request #11812 from danfengliu/fix-issue-of-gc-untag-case
Fix issue of GC untag images case
2020-04-30 20:17:37 +08:00
Daniel Jiang
2f2fe8ba68
Merge pull request #11801 from wy65701436/conformance-ci
add conformance test to CI
2020-04-30 19:49:07 +08:00
Daniel Jiang
8b9dc1fac6
Merge pull request #11779 from bitsf/cicd_fix_base_image
feat(cicd) add build date for base images
2020-04-30 19:13:16 +08:00
danfengliu
d10dfc8167 Fix issue of GC untag images case
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-30 16:10:03 +08:00
danfengliu
768b292b15 Add double confirm for clicking untag checkbox
1. Add double confirm for clicking untag checkbox;
2. Add retry for waiting GC completion;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-29 20:21:07 +08:00
wang yan
d9773248e2 add conformance test to CI
1, add a specrate git action for conformance test
2, use the OCI testing code to test Harbor master code on push

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-29 16:08:27 +08:00
danfengliu
cd0463047b Add tag for ui test case of add nuew user
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-28 19:47:47 +08:00
danfengliu
f8ce615e12 Add ctr command for pulling oci
1. Manifest list can be pulled by ctr;
2. ui-test missing key checkpoint, fix it by add checking scan detail result;
3. add tag retension untag image test;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-28 15:01:49 +08:00
danfengliu
16f6ad3688
Merge pull request #11768 from AllForNothing/nightly-login
Add new nightly case  admin add new users
2020-04-28 14:57:37 +08:00
Ziming Zhang
3c51ab556e feat(cicd) add build date for base images
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-04-28 12:56:37 +08:00
AllForNothing
8ff4003438 Add new nightly case admin add new users
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-04-28 11:10:06 +08:00
He Weiwei
8458f980d0 fix(swagger): enable security in swagger.yaml
1. Enable `security` in the swagger.yaml.
2. Include `basic` auth in `security` to make the generated python
client by `swagger-codegen-cli` work with basic authorization.
3. Include `anonymous` auth in `security` to make APIs of v2.0 generated
by `goswagger` work with `security` middleware.

Closes #11771

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-04-28 02:34:34 +00:00
DQ
599ca98c09 Hidden veriify client cert verfiy option
Remove to avoid replication access core from external_url issue

Signed-off-by: DQ <dengq@vmware.com>
2020-04-23 10:14:36 +08:00
Ziming Zhang
41e8bb91ce fix some cicd problems
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-04-21 14:34:45 +08:00
danfengliu
fdded0744a Clean up gitaction env and add tag for pytest
1. Add docker prune cmd to release some space;
2. Add tag for pytest in case of debugging requirement;
3. Replace image to smaller size in robot account pytest;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-20 18:19:28 +08:00
Danfeng Liu (c)
1cc5d0261b Add robot-account verification for upgrade test
1. Add Clair should be default scanner in upgrade test;
2. Add tag retention verification in upgrade test;
3. Add tag retention verification in upgrade test;
4. Add tag Immutability verification in upgrade test;
5. Add webhook verification in upgrade test;
6. Add CVE whitelist in upgrade test;

Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2020-04-16 22:16:33 +08:00
Yogi_Wang
589d05a894 nightly move switch scanner case to mutiple-scanner file
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-16 10:05:08 +08:00
Yogi_Wang
8835813439 [test-case] add switch scanner case
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-15 14:53:17 +08:00
jwangyangls
5cf72e3ba4
Merge pull request #11616 from jwangyangls/fix-bug-case
[Test] Fix view log error
2020-04-14 21:41:47 +08:00
Yogi_Wang
6799f7c4ad [test] Fix view log error
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-14 21:08:08 +08:00
AllForNothing
46320641b4 Add new nightly case for GC untagged images
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-04-14 09:25:06 +08:00
Will Sun
009662a317
Merge pull request #11426 from AllForNothing/nightly-tag
Add new nightly cases for tag CRUD
2020-04-13 12:15:18 +08:00
danfengliu
a869bba59f Modify quota error info due to product changed
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-10 21:06:05 +08:00
Wenkai Yin
812ea7814b Uncomment the verification logic in the ldap group test case
Fixes #11094, uncomment the verification logic in the ldap group test case

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-04-10 14:02:11 +08:00
danfengliu
5ec8ff70b2
Merge pull request #11474 from danfengliu/add-multi-scanner-robot
Add Multi-Scanner test file
2020-04-08 20:45:05 +08:00
danfengliu
66dfd59c80 Add Multi-Scanner test file
1. Add Multi-Scanner test file for harbor is deployed with multi scanners;
2. Modify notary doc description;
3. Add paragraph for e2e-api-python-based-scripting-guide;
4. Fix delete project issue;
5. Remove count qoutas in nightly and modify some of it;
6. Add Trivy in git hub offline action;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-08 16:26:31 +08:00
AllForNothing
e342b4ef0b Add new nightly cases for tag CRUD
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-04-08 09:56:27 +08:00
Ziming Zhang
572ebef685 feat(cicd) parameterize docker base image and external url
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-04-08 00:21:47 +08:00
Yogi_Wang
3775509ce5 Update APIs to only accept encoded repository name
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-07 22:40:39 +08:00
Qian Deng
1c8d5c6df5
Add spaces (#11471)
Fix typo of spaces cause failure

Signed-off-by: DQ <dengq@vmware.com>
2020-04-07 17:47:23 +08:00
DQ
1ae50b8d66 Remove migrator flags in script
Because migrator tool removed

Signed-off-by: DQ <dengq@vmware.com>
2020-04-07 14:57:10 +08:00
Daniel Jiang
db10720e80
Merge pull request #11406 from reasonerjt/reenable-token-auth-for-cli-new
Reenable token auth for cli
2020-04-07 08:55:25 +08:00
He Weiwei
80027c3b86
Merge pull request #11397 from ywk253100/200402_chart_api
Add a seperated swagger file for chart API
2020-04-07 00:05:49 +08:00
wang yan
44825e819e deprecate quota count on artifact
Fixes #11241

1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-06 16:56:11 +08:00
Wenkai Yin
8f8b4d5e8d Add a seperated swagger file for chart API
Add a seperated swagger file for chart API as these APIs have no version

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-04-06 16:30:26 +08:00
Daniel Jiang
08f9ffa000 Reenable token auth for cli
Docker CLI fails if it's not logged in upon seeing "basic" realm challenging while pinging the "/v2" endpoint. (#11266)
Some CLI will send HEAD to artifact endpoint before pushing (#11188)(#11271)

To fix such problems, this commit re-introduce the token auth flow to the CLIs.

For a HEAD request to "/v2/xxx" with no "Authoirzation" header, the v2_auth middleware populates the
"Www-Authenticate" header to redirect it to token endpoint with proper
requested scope.

It also adds security context to based on the content of the JWT which has the claims of the registry.
So a request from CLI carrying a token signed by the "/service/token" will have proper permissions.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-04-04 00:05:58 +08:00
Yogi_Wang
2610fe530f [Test Case] Add nightly case for CVE
1. add nightly case for cve
2. change translate words
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-03 16:59:45 +08:00
jwangyangls
4ea7b13215
Merge pull request #11385 from jwangyangls/nightly-case-trivy-2
[Nightly] Project Level Image Serverity Policy
2020-04-02 11:14:21 +08:00
danfengliu
aa26e624b9
Merge pull request #11379 from danfengliu/add-nightly-test-of-push-helm-chart
Add nightly test of push helm chart
2020-04-02 10:29:21 +08:00
Yogi_Wang
01f8291bb7 [Nightly] Project Level Image Serverity Policy
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-02 10:16:57 +08:00
danfengliu
e3b31bb8dd Add nightly test of push helm chart
1. Add push helm chart oci test in nightly;
2. Modify archive icon locator xpath;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-01 18:53:19 -07:00
danfengliu
f11e713ada
Merge pull request #11380 from jwangyangls/nightly-case-trivy-1
[Nightly] add case about trivy
2020-04-01 22:45:40 +08:00
Qian Deng
b1284da96b
Merge pull request #11360 from ninjadq/rever_chart_api_change
Rever chart api change
2020-04-01 18:58:57 +08:00
Yogi_Wang
24b57715ab [Night] add case about trivy
`
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-01 17:06:34 +08:00
AllForNothing
e6e3f0a6af Fix nightly cases for webhook
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-04-01 16:27:40 +08:00
danfengliu
7ad3b500ac
Merge pull request #11356 from danfengliu/add-nightly-test-of-push-manifest-list
Add nightly test case of push docker manifest list
2020-04-01 15:21:05 +08:00
danfengliu
8f9822d168 Add nightly test case of push docker manifest list
1. Add notary tool in E2E Dockerfile;
2. Add push docker manifest list test in nightly;
3. Modify document for e2e user guide;
4. Add CNAB tool  in E2E Dockerfile;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-01 06:51:13 +00:00