Commit Graph

80 Commits

Author SHA1 Message Date
YangJiao
a4a82d763a Add TestCase to download helmchart from UI
Signed-off-by: YangJiao <jiaoya@vmware.com>
2021-09-13 14:06:44 +08:00
danfengliu
3ef4dc17fc Fix notary trust directory issue and add login for each base image in makefile
1. Use root instead of ~ in notary parameter;
2. Fix tag immutability issue caused by GUI change;
3. Replace email domain name to harbor test;
4. Add login for each base image in makefile;
5. Add customize look test in nightly.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-05-20 18:23:13 +08:00
danfengliu
f0ebd17994 Add build base image step in build package git action workflow
Build base image step should be in build package workflow, and local base images build by new step should be removed since images have been pushed to docker hub.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-04-17 18:10:44 +08:00
danfengliu
a2fc1bcfaf Fix quotas test issue in upgrade pipeline
1. Change way for quotas verification in upgrade pipeline, prepare specific size of image, then it's an known value for verifcation;
2. Add notary key rotate test;
3. For issue brought by docker 20, clean containerd cache is the only effective way, so both dockerd and containerd should be cache cleard and restarted;
4. Upgrade E2E Dockerfile for importing readable file size package, and other issues;
5. Uncomment project level robot account test in nightly.
6. Get DNS from local setting, and set it into docker deamon config file;

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-03-10 06:27:22 +00:00
danfengliu
7fb9dbd0fa Upgrade docker and containerd
1. Fix E2E quotas issue, push the same image but with different name;
2. Add checkpoint for robot account test;
3. Upgraded docker and containerd in E2E image;
4. Package base image sample(busybox) into E2E image, so in E2E
container, all local docker images can be cleaned up, once base image is needed for
building image, it can be loaded locally;
5. Adapt OIDC service of supporting LDAP user, and add OIDC group user
test;
6. Restart docker deamon before content trust test, both in API and UI
test;
7. Add retry for keyword "Add A Tag Immutability Rule";
8. Fix tag retention test issue, missing click angle icon, and enhance
checkpoint of dry run and real run;
9. Fix schedule test issue for wrong cron string;
10. Disable quotas verification, it's not stable for script defect;

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-02-24 15:43:11 +08:00
Danfeng Liu (c)
ddf9d74135 Fix nightly keyword issues
1. Add retry for Get Statics Text keywords make this operation robust;
2. Replace image to one with slash in name for CNAB test, to cover more
condition;
3. Add replication rule deletion verfication in upgrade test;
4. Add non-admin user in pull operation for proxy cache test;
5. Add verification for quota display for upgrade test;
6. Add test for large size of image replication;
7. Add test checkpoint for system robot account py-test script;

Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2021-01-19 15:12:15 +08:00
Danfeng Liu (c)
ae05e3fdb0 Add Nightly GUI test for system robot account
1. Add GUI test for system robot account feature;
2. Fix issue of failing to catch docker pull image exception in API test.

Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2021-01-06 15:47:04 +08:00
danfengliu
3659abc62a Migrate gitlab replication test to nightly
Add 2 gitlab pull/push replication test scripts in replication pipeline.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-11 18:04:46 +08:00
danfengliu
2bf89e9356 Add docker-hub login before docker-hub pull
Due to docker-hub pull request rate limitation, we will use registed account to pull image
from docker-hub, therefore add docker-hub login in API tests.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-22 21:11:05 +08:00
danfengliu
1b127b16c5 Add API test scripts for tag immutability and scan signed image
1. Add 7 API python test scripts for tag immutability;
2. Swagger file has object struct defination issue on imuutability policy rule, because
   this policy is in shared with retention rule, therefore, they should be unified to
   the same  type of rule;
3. Fixed a timeout issue of action keyword, waiting for a long period;
4. Add API test scripts for scan signed image.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-03 10:10:06 +00:00
danfengliu
ab3e24a2e3 Add proxy cache python test scipts and upgrade containerd
Containerd in e2e image was using native one in ubuntu, it should be updated to the latest release.
And add proxy cache py-tests including pull image/manifest list by docker and ctr CLI.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-10-28 02:13:40 +00:00
danfengliu
00180b2a6f Add P2P Preheat UI Test
1. Fix ldap script issue, test case name should be in lowercase;
2. Add manifest list checkout point for proxy cache issue;
3. Add P2P Preheat policy CRUD Test.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-28 11:34:32 +08:00
danfengliu
3b8a2890f9 Upgrade robot-framework and Selenium library
1. Upgrade robot-framework to 3.1 and Selenium library to 4.4.0.
2. Fix a registry issue for clear filter text input.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-06 07:25:53 +00:00
danfengliu
8a691c5df1 Add OIDC user onboard test in nightly
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-27 13:56:18 +08:00
Ziming Zhang
8857e89e40 feature(redis) support redis sentinel
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-07-19 21:19:03 +08:00
danfengliu
e020b90bf0 add oras cli py-test
1. Add oras cli py-test;
2. Add env for notary url, allow to input different notary port instead of solid 4443;
3. Add retry for keyword Cannot Pull Image and make it longer during retry.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-06-08 11:10:07 +08:00
danfengliu
5d8a308158 Add retry for keyword go into index
1. Add retry for keyword go into index;
2. Add retry for keyword go into project.
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-05-27 08:55:58 +00:00
danfengliu
d10dfc8167 Fix issue of GC untag images case
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-30 16:10:03 +08:00
danfengliu
768b292b15 Add double confirm for clicking untag checkbox
1. Add double confirm for clicking untag checkbox;
2. Add retry for waiting GC completion;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-29 20:21:07 +08:00
danfengliu
66dfd59c80 Add Multi-Scanner test file
1. Add Multi-Scanner test file for harbor is deployed with multi scanners;
2. Modify notary doc description;
3. Add paragraph for e2e-api-python-based-scripting-guide;
4. Fix delete project issue;
5. Remove count qoutas in nightly and modify some of it;
6. Add Trivy in git hub offline action;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-08 16:26:31 +08:00
danfengliu
8f9822d168 Add nightly test case of push docker manifest list
1. Add notary tool in E2E Dockerfile;
2. Add push docker manifest list test in nightly;
3. Modify document for e2e user guide;
4. Add CNAB tool  in E2E Dockerfile;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-04-01 06:51:13 +00:00
danfengliu
77e3b0d828 Fix nightly ldap test cases
1. Modify robot account test, checkout "Never Expired" button;
2. Modify OIDC get user API;
3. Modify verification for docker pull command;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-03-27 20:24:43 +08:00
danfengliu
deb225b0f2 Modify elements locator in nightly tests
1. Modify get signature API in sign image test;
2. Modify locator for tag in artifact list;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-03-22 20:48:37 +08:00
Danfeng Liu (c)
1cdf7253cd Repull image when error
Docker pull command was enclosed to one keyword, in this keyword,
there're retry and log record.

Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-12-31 14:30:26 +08:00
Danfeng Liu (c)
76cbcfcbb9 Refect upgrade script, population will be distinguished by harbor version.
Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-12-26 14:08:47 +08:00
Danfeng Liu (c)
364df7a767 Modify robot keyword from press key to press keys
Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-12-12 17:01:31 +08:00
Wenkai Yin(尹文开)
d145f4baf4
Merge pull request #10034 from ywk253100/191128_clean
Clean up admiral-related code
2019-12-04 17:33:31 +08:00
Danfeng Liu (c)
7dbe84d9c6 1. Add nightly test case of Robot-Account; 2. In Go Into Repo keyword, filter element should not be clicked since list in loading state, so add some sleep and waiting for the cirlce icon is gone; 3. Move read-only test case in the end of common.robot.
Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-12-03 14:46:54 +08:00
Wenkai Yin
dd2bc0ecef Clean up admiral-related code
Clean up admiral-related code as it's useless

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-11-28 17:28:54 +08:00
Danfeng Liu (c)
f0750f73de 1. Add element verification for keyword Go Into Project; Make big loop effective by stop assert in keyword Retry Double Keywords When Error; 2. Quota string verification is too restrictive, make it loose a little.
Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-25 17:44:15 +08:00
Danfeng Liu (c)
b8a7598ffa The criteria of keyword Go Into Project is not strong for verify if it's in project repositroy page, so fix this issue by way of 'Retry Double Keywords When Error'.
Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-22 12:07:20 +08:00
sshijun
14364727ae Nightly case for tag-retention
Signed-off-by: sshijun <sshijun@vmware.com>
2019-11-11 16:58:33 +08:00
Yogi_Wang
58936aae42 Modify nightly case of scanner
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2019-11-11 13:19:37 +08:00
Yogi_Wang
988279cdcb Add webhook nightly case
Signed-off-by: Yogi_Wang <yawang@vmware.com>
2019-10-25 15:25:32 +08:00
danfengliu
ac0d3c991a Add nightly test case of 'Retag should be forbidden when system configuration read-only mode was set'.
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-10-10 17:13:19 +08:00
danfengliu
feeb6fb2d8 1. Nightly test case failed due to more vulnerbility CVE IDs in image redis, so add 2 new CVE IDs in CVE Whitelist test cases. 2. A Keyword parameter error was fixed in this PR.
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-10-02 16:43:50 +08:00
Danfeng Liu (c)
20dc245631 In nightly helm pipeline, test case 'Project Quotas Control Under GC' failed, it should be timing issue that the valuse in UI is not ready, so I add a keyword 'Retry Keyword When Return Value Mismatch' to retry value getting action.
Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-09-25 17:02:05 +08:00
Danfeng Liu (c)
5d1913842c Script test case for project quotas, there will be 2 or 3 test cases in this PR, like project quota edit, prject quota functionality.
Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-09-18 17:55:25 +08:00
danfengliu
7d681c623b Add AWS-ECR and Google-GCR replication test into nighly tests.
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-09-02 17:53:49 +08:00
danfengliu
7217ed5358 Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test.
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-28 14:01:43 +08:00
danfengliu
6818ce9b06
Adding retry for wait element visibility, in nightly test case Developer-Operate-Labels, project didn't show up when checking its visiblity, so add some wait for this check. (#8554)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-05 16:13:39 +08:00
danfengliu
01e3175647
In nightly test, there're some element click error occured in a rare occasion, so add a sleep to give it a try. (#8428)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-07-29 13:37:02 +08:00
danfengliu
a8300d91e6
Add a nightly test case helm CLI, in this case, we verify help repo add and helm push. (#8240)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-07-16 14:00:30 +08:00
danfengliu
432d0757f0
In nightly test, case Disable-Scan-Schedule always failed at action of edit button click, so we should add a comfirmation to make sure that this button will be disappeared after click action. (#8126)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-06-24 18:09:34 +08:00
danfengliu
74d39afbb0 Modify robot-framework keyword <Retry Double Keywords When Error>, when the second action is success, should quit keyword.
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-06-18 16:40:11 +08:00
danfengliu
d8b76c482a
Add 2 UI test cases which is replication rule edit and delete. (#7904)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-06-03 16:58:24 +08:00
danfengliu
e66d857fbc Add test case secret-verify for oidc-auth-mode feature by adding some keywords and one test cases.
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-05-27 14:31:14 +08:00
danfengliu
a503944110
OIDC authentication is a new feature in version 1.8.0, so add nightly GUI test cases for OIDC authentication scenario. (#7745)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-05-22 11:19:08 +08:00
danfengliu
dffb2bbd98 Due to replication NG feature already in master branch, old replication UI test cases have been moved out of Jenkins pipeline, and here test case <Replication Of Pull Images from DockerHub To Self> and <Replication Of Push Images from Self To Harbor> has been finished and ready for duty. Other modification is add retry for keywords in UserProfile.robot (#7611)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-05-09 15:44:48 +08:00
danfengliu
c320645ba1
many keywords in ToolKit.robot miss wait or retry when operating elements, and it has cause test case failure one time, so I modify all keywords in ToolKit.robot. (#7340)
Signed-off-by: danfengliu <danfengl@vmware.com>
2019-04-11 15:15:35 +08:00