Commit Graph

999 Commits

Author SHA1 Message Date
DQ
66fd4a9728 change label project id to project name
project_name is more accurate

Signed-off-by: DQ <dengq@vmware.com>
2021-01-26 16:15:04 +08:00
DQ
25ea2b1c82 Use apt to install python in ci
To avoid upstream get_pip.py changes

Signed-off-by: DQ <dengq@vmware.com>
2021-01-25 16:54:09 +08:00
DQ
28ae77e5c6 Aggregate metrics
1. Add operation id in ctx in baseapi before prepare
2. add operation id for registry proxy request
3. use url for other request

Signed-off-by: DQ <dengq@vmware.com>
2021-01-25 09:59:10 +08:00
Wang Yan
dba229d0df
build third party binaries in CI (#14019)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-22 11:33:42 +08:00
danfengliu
d4b7888098
Merge pull request #13882 from danfengliu/fix-nightly-keyword-issues
Add tests and fix nightly issues
2021-01-19 17:28:52 +08:00
Danfeng Liu (c)
ddf9d74135 Fix nightly keyword issues
1. Add retry for Get Statics Text keywords make this operation robust;
2. Replace image to one with slash in name for CNAB test, to cover more
condition;
3. Add replication rule deletion verfication in upgrade test;
4. Add non-admin user in pull operation for proxy cache test;
5. Add verification for quota display for upgrade test;
6. Add test for large size of image replication;
7. Add test checkpoint for system robot account py-test script;

Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2021-01-19 15:12:15 +08:00
Qian Deng
d06ce6efd5
Merge pull request #13922 from ninjadq/add_e2e_test_for_metrics
Add e2e test for harbor metrics
2021-01-18 11:04:11 +08:00
Dirk Mueller
903707d6c2
Fix repoitory -> repository typo (#13794)
Signed-off-by: Dirk Mueller <dirk@dmllr.de>
2021-01-16 19:34:43 +08:00
Will Sun
b7c5fc0562
Modify i18n text for push permission (#14015)
Signed-off-by: AllForNothing <sshijun@vmware.com>
2021-01-15 15:26:48 +08:00
danfengliu
a576ee8694 Pass git action secret tp push image function in build workflow
Signed-off-by: danfengliu <danfengl@vmware.com>
2021-01-12 10:01:50 +08:00
danfengliu
5f713f22fd Push harbor build images to docker-hub in build workflow
Signed-off-by: danfengliu <danfengl@vmware.com>
2021-01-11 18:24:57 +08:00
Ziming Zhang
5bff3e94a0 fix(immutability) swagger test case
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-01-11 03:21:12 +00:00
Ziming Zhang
be58c1d3ff fix(retention) swagger test case
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-01-11 03:21:12 +00:00
DQ
27a9f71e26 Add e2e test for harbor metrics
Signed-off-by: DQ <dengq@vmware.com>
2021-01-09 04:12:32 +08:00
Ziming Zhang
39fb500318 feat(retention) refactor to use go swagger api
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-01-08 07:09:28 +00:00
Will Sun
f566748c77
Fix nightly case by add to scan button (#13937)
Signed-off-by: AllForNothing <sshijun@vmware.com>
2021-01-08 15:01:26 +08:00
danfengliu
f57dc13081 Add py-test for system robot account
In this py-test, add a new system robot account, and verify all project privileges.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-01-08 12:01:53 +08:00
danfengliu
511bd86930
Merge pull request #13849 from danfengliu/add-gui-test-for-system-robot-account
Ad  nightly GUI test for system robot account
2021-01-06 16:59:05 +08:00
danfengliu
b5607850af
Merge pull request #13834 from danfengliu/clear-local-useless-images-in-nightly
Clear local images in nightly test
2021-01-06 16:55:22 +08:00
Danfeng Liu (c)
ae05e3fdb0 Add Nightly GUI test for system robot account
1. Add GUI test for system robot account feature;
2. Fix issue of failing to catch docker pull image exception in API test.

Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2021-01-06 15:47:04 +08:00
danfengliu
076ebb5ee8 Clear local images in nightly test
Remove local image right after pushing operation, then pull operation can be triggered by docker 20.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-01-04 10:25:02 +08:00
Wang Yan
7a8a8fa104
upgrade go version to v1.15.6 (#13836)
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-23 18:53:09 +08:00
He Weiwei
ce6ed3eeb7 refactor(api): move scan all apis to go-swagger
Move scan all APIs from beego to go-swagger.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-21 02:17:02 +00:00
danfengliu
4f0842bd23 Add py-test for scan manifest list and CNAB bundle
Due to complicate logic of scan report in multi-level artifacts, should add scan tests
for into manifest list and CNAB bundle python test suit.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-18 16:37:53 +08:00
Wang Yan
01eb60c36b
add permission checking for gc operation (#13756)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-15 23:27:12 +08:00
Wenkai Yin(尹文开)
6569016d35
Merge pull request #13139 from wy65701436/migrate-gc
Migrate gc to task manager
2020-12-14 10:43:44 +08:00
danfengliu
3659abc62a Migrate gitlab replication test to nightly
Add 2 gitlab pull/push replication test scripts in replication pipeline.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-11 18:04:46 +08:00
wang yan
1bb79d402d update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 16:08:52 +08:00
danfengliu
ad7581d9b3 Migrate gitlab replication test to nightly
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-09 12:00:38 +08:00
danfengliu
56a35437b4
Change image source in e2e pytest (#13640)
Change source of most of test image samples from docker-hub to local building ones, so it will cost less docker-hub pull requests.
And some of cases like push cnab, they have to use docker-hub, but image samples in cnab test will cost 17 quotas, in this PR, we
replace those samples, now cnab case will cost 6 quotas.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-04 02:28:29 -08:00
danfengliu
ff50e2363d
Move proxy cache e2e test into a new CI job (#13690)
CI job timeout is 60 minutes, but proxy cache case timeout is 20 minutes, once
proxy cache case is timeout, it will reach CI job timeout, then cause CI job terminated
by timeout without running rest of test steps, expecially uploading useful harbor logs,
so move this test into a new clean job for saving time and debugging.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-12-04 02:27:59 -08:00
danfengliu
792a3c8c27
Merge pull request #13625 from danfengliu/fix-scan-all-py-test-script
Fix scan all python test script issue of missing test step
2020-12-02 09:17:10 +08:00
Wang Yan
732e9a21cd
updates on robot accounts (#13623)
* updates on robot accounts

1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account

Signed-off-by: wang yan <wangyan@vmware.com>

* update code per review comments

1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-01 18:31:34 +08:00
danfengliu
b36a4cf3be Fix scan all python test script issue of missing test step
1. In scan all py-test, missing a important test step for checkin scan all status,
2. which indicate the ending of scan all job.
In swagger, ongoig field is missing in stats struct.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-27 20:27:37 +08:00
DQ
590212b485 Remove clair related code
- clair code in harbor core
- clair code in frontend
- clair code in robotcase

Signed-off-by: DQ <dengq@vmware.com>
2020-11-27 14:01:04 +08:00
Wang Yan
02846194e0 parent 8e61a3ea31
author Wang Yan <wangyan@vmware.com> 1605849192 +0800
committer Wang Yan <wangyan@vmware.com> 1606361046 +0800

update code per review comments

Signed-off-by: wang yan <wangyan@vmware.com>
2020-11-26 14:10:12 +08:00
Wenkai Yin(尹文开)
fe8b628f0c
Merge pull request #13437 from ywk253100/200929_replication
Refactor the replication execution
2020-11-24 10:38:22 +08:00
Wenkai Yin
294385c34d Refactor the replication execution
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-11-23 14:24:10 +08:00
danfengliu
2bf89e9356 Add docker-hub login before docker-hub pull
Due to docker-hub pull request rate limitation, we will use registed account to pull image
from docker-hub, therefore add docker-hub login in API tests.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-22 21:11:05 +08:00
danfengliu
5d0e6d6dbf Add p2p python test script
Add p2p python test including create disstribution instance and project p2p preheat policy.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-12 05:41:35 +00:00
danfengliu
a70753d382 Add p2p python test script
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-11 14:55:23 +08:00
danfengliu
444de8536a Suppress urllib3 warning and fix nightly element locator issue and remove switch scanner
1. Since upgrade python to 3.7, some urllib3 warnings come out during
   execution, because missing disconnection when using request by docker
   library, it's not affect script execution, therefore ignore them first.
2. Replication tests failed in nightly due to new UI code, some element
   locators are not valid anymore;
3. Remove switch scanner test steps, since Clair was removed.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-10 10:04:46 +08:00
danfengliu
1b127b16c5 Add API test scripts for tag immutability and scan signed image
1. Add 7 API python test scripts for tag immutability;
2. Swagger file has object struct defination issue on imuutability policy rule, because
   this policy is in shared with retention rule, therefore, they should be unified to
   the same  type of rule;
3. Fixed a timeout issue of action keyword, waiting for a long period;
4. Add API test scripts for scan signed image.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-11-03 10:10:06 +00:00
danfengliu
691168b8cf
Merge pull request #13359 from danfengliu/optimize-upgrade-pipeline-of-pull-push-image
Optimize upgrade pipeline script of push image
2020-10-29 14:31:30 +08:00
danfengliu
ab3e24a2e3 Add proxy cache python test scipts and upgrade containerd
Containerd in e2e image was using native one in ubuntu, it should be updated to the latest release.
And add proxy cache py-tests including pull image/manifest list by docker and ctr CLI.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-10-28 02:13:40 +00:00
danfengliu
cff4f529cd Optimize upgrade pipeline script of push image
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-10-26 15:16:18 +08:00
danfengliu
b388078a3e Upgrade containerd in e2e image
Containerd in e2e image was using native one in ubuntu, it should be updated to the latest release.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-10-21 08:14:01 +00:00
danfengliu
6cdae44dc2
Merge pull request #13235 from danfengliu/fix-issue-for-upgrade-pipeline
Replication UI changes break upgrade pipeline
2020-10-14 12:06:24 +08:00
danfengliu
4bc7444602 Add UI test for LDAP admin group dn setting
1. Verify LDAP group admin in UI;
2. Update LDAP group admin need to verify the AdminRoleInAuth, get current user will return this field as true.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-10-14 11:05:09 +08:00
danfengliu
8285db8def Replication UI changes break upgrade pipeline
Remove duplicate keyword for editing replication rule.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-10-14 11:00:14 +08:00
Will Sun
3d5f8106f2
Merge pull request #13176 from AllForNothing/fix-nightly
Fix nightly case for replication
2020-09-30 18:29:36 +08:00
AllForNothing
3f213df595 Fix nightly case for replication
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-09-30 16:59:11 +08:00
danfengliu
baf7cda3e2
Merge pull request #13164 from danfengliu/add-build-package-action-in-github
Add harbor build in git action workflow
2020-09-30 09:55:26 +08:00
danfengliu
12f4f73c8a Add harbor build in gith action workflow
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-29 09:18:18 +00:00
danfengliu
fc5743b355
Merge pull request #13128 from danfengliu/fix-a-typo-in-upgrade-prepare-script
Fix typo in upgrade prepare script
2020-09-28 17:54:51 +08:00
danfengliu
e4894017cd Fix typo in upgrade prepare scritp
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-28 10:36:22 +08:00
AllForNothing
8d9a18b539 Update UI package to latest version
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-09-25 09:46:05 +08:00
Will Sun
18f41bad88
Merge pull request #13085 from AllForNothing/replication
Add disable/enable function to replication rules
2020-09-24 17:35:51 +08:00
danfengliu
48510fde08
Merge branch 'master' into add-2.1-upgrade-pipeline-in-nightly 2020-09-22 13:48:22 +08:00
danfengliu
ef53edf553 Recovery nightly setup
1. Recovery nightly setup;
2. Add retry for replication rule selection.
3. Add retry for CNAB test.
4. Rerun keyword of push CNAB

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-22 02:41:50 +00:00
danfengliu
e2989e3b93 Add 2.0 & 2.1 upgrade pipeline in nightly
1. Add 2.0 upgrade pipeline in nightly;
2. Add 2.1 upgrade pipeline in nightly.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-22 02:14:40 +00:00
AllForNothing
abdbd79ed4 Add disable/enable function to replication rules
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-09-17 10:29:31 +08:00
danfengliu
7e097afde8
Remove ca parameter in helm test case (#13069)
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-16 16:31:45 +08:00
danfengliu
ebef11476a
Merge pull request #13002 from danfengliu/upgrade-docker-in-drone-dockerfile
Create Dockerfile for E2E py-test
2020-09-15 14:16:11 +08:00
danfengliu
d99ea887f1
Merge pull request #12989 from danfengliu/Add-Checkpoint-for-LDAP-group-py-test
Add checkpoint for LDAP group py-test
2020-09-15 14:15:31 +08:00
danfengliu
86fb6fdc65 Add checkpoint for LDAP group py-test
LDAP group has different role, user in group has the same role, as groups with different roles were added in project
member list, user should act like in different roles. for admin and dev role, there should be checkpoints to verify priviledges of each own.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-14 21:15:09 +08:00
danfengliu
0dda643bc5 Create Dockerfile for E2E py-test
Robot tests have been always executed in docker container, but API pyt-test is not,
so rebuild Dockerfile for it. During that, some of tools have been upgraded, so API py-test
 scripts need some modification.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-14 02:36:18 +00:00
stonezdj(Daojun Zhang)
97b9cc2d5e
Merge pull request #12997 from stonezdj/200902_ldap_filter_fail
Add ldap filter syntax validation when create search filter
2020-09-09 16:21:01 +08:00
danfengliu
f90f9a6984
Fix nightly issue caused UI changing (#13006)
Some UI text display changed, it cause failure in nightly.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-09 15:20:13 +08:00
stonezdj
b9752f3112 Add ldap filter syntax validation when create search filter
Correct ldap search filter is enclosed with '(' and ')'
Search ldap group with the ldap group base DN instead of group DN
Fixes #12613 LDAP Group Filter and Group Base DN have no affect

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-09-09 10:07:07 +08:00
danfengliu
34e2490154 Fix git action CI python package installation
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-03 20:41:55 +08:00
danfengliu
2ca5c11675 Fix git action CI missing python sit packages issue
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-03 17:34:51 +08:00
danfengliu
931bde0df8 Fix issue of missing swagger package for python API test
1. Python API test can't import packages from swagger maker, the reason is that python install packages to site-packages instead of dist-packages,
   so the current solution is to copy pakages from site back to dist, I will try to find a way of using ENV variable to fix this;

2. Remove a python library no longer be used.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-09-02 15:18:23 +08:00
danfengliu
00180b2a6f Add P2P Preheat UI Test
1. Fix ldap script issue, test case name should be in lowercase;
2. Add manifest list checkout point for proxy cache issue;
3. Add P2P Preheat policy CRUD Test.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-28 11:34:32 +08:00
stonezdj(Daojun Zhang)
b2ed195df5
Merge pull request #12785 from danfengliu/add-distribution-ui-test
Add distribution UI test in nightly
2020-08-21 14:19:05 +08:00
danfengliu
37e280c49f Add proxy-cache and distribution UI test in nightly
1. Add proxy-cache UI test in nightly;
2. Add distribution UI test in nightly.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-21 02:08:06 +00:00
danfengliu
143062fe20
Merge pull request #12788 from danfengliu/upgrade-robotframework-in-git-action
upgrade robotframework in git action
2020-08-20 20:08:36 +08:00
danfengliu
3f5bd9ae0a Re-script ldap API script
1. upgrade robotframework in git action
2. Re-script ldap API script:test_assign_role_to_ldap_group.py

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-20 11:24:51 +00:00
Dirk Mueller
08a4d8efd2
Update to golang 1.14.7 (#12809)
We should use a golang that isn't having security issues.

This includes:
* go1.14.6 (released 2020/07/16) includes fixes to the go command, the
compiler, the linker, vet, and the database/sql, encoding/json,
net/http, reflect, and testing packages. See the Go 1.14.6 milestone on
our issue tracker for details.

* go1.14.7 (released 2020/08/06) includes security fixes to the
encoding/binary package. See the Go 1.14.7 milestone on our issue
tracker for details (CVE-2020-16845)

Signed-off-by: Dirk Mueller <dirk@dmllr.de>
Signed-off-by: Dirk Mueller <dmueller@suse.com>
2020-08-20 15:38:35 +08:00
Wang Yan
60427e7187
build base image in CI (#12750)
In git action, use the local build base images instead of pulling from docker hub.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-18 14:11:52 +08:00
He Weiwei
f659523f50 fix(scan): fix the permission checking for artifact scanning
Closes #12778

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-17 09:55:38 +00:00
He Weiwei
f309896f2f refactor(api): generate project apis by go-swagger
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-15 16:10:57 +00:00
danfengliu
5cb239af1e
Fix echo parameter issue (#12775)
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-14 10:23:58 +08:00
danfengliu
e1eac2b4d0
Merge pull request #12688 from danfengliu/fix-upgrade-test-wrong-keyword-issue
Fix wrong keyword issue in upgrade test
2020-08-14 10:11:37 +08:00
Daniel Jiang
83de495147
Merge pull request #12772 from reasonerjt/update-root-cert-test
Refresh CA cert
2020-08-13 18:22:19 +08:00
Daniel Jiang
376929d58e Refresh CA cert
This commit refresh the harbor_ca.crt and revert the changes to generate a new CA each time in the test script.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-13 08:43:07 +00:00
danfengliu
8f116d7b38 Add exec priviledge to integration script
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-13 08:15:42 +00:00
danfengliu
601d1ef0eb Add git/golang/dns config in drone image
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-13 07:24:54 +00:00
danfengliu
9a2188641d Upgrade python and robot for drone build
1.Separate Dockerfile to drone and nightly e2e, one Dockerfile should cover
 both scenarios, it will be resolved later;
2.Fix docker api err type issue, API error has no message attibute;
3.Upgrade docker in nightly e2e images.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-12 10:26:12 +00:00
danfengliu
ab7767f539 Fix wrong keyword issue in upgrade test
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-10 14:47:38 +00:00
danfengliu
4968b6619b Recovery drone build script
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-10 14:23:44 +00:00
danfengliu
859ff2e1ef Modify e2e image for drone build
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-10 11:40:26 +00:00
danfengliu
07f5f549d0 Fix LDAP API test issue by upgrading python
Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-09 06:37:28 +00:00
danfengliu
0521f9eed5
Merge pull request #12626 from danfengliu/upgrade-robotframework-and-libs
Upgrade robot-framework and Selenim library
2020-08-06 16:01:35 +08:00
danfengliu
3b8a2890f9 Upgrade robot-framework and Selenium library
1. Upgrade robot-framework to 3.1 and Selenium library to 4.4.0.
2. Fix a registry issue for clear filter text input.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-08-06 07:25:53 +00:00
Flávio Ramalho
3d3fabbd20
Make notary-remove-image-signature more flexible (#12536)
Instead of explicitly defining the expect path to /usr/local/bin/expect,
use the env command, so the expect executable is searched for and launched
from wherever it is first found.

Signed-off-by: Flávio Ramalho <framalho@suse.com>
2020-07-31 15:49:45 +08:00
danfengliu
da662f52d5
Merge pull request #12607 from danfengliu/add-gcs-s3-push-based-replic-test
Add GCS&S3 push-based replicatiton tests
2020-07-30 09:40:21 +08:00
danfengliu
8328e95b63 Add GCS&S3 push-based replicatiton tests
1. Add GCS push-based replicatiton tests;
   There is an issue against it, so it's commented out;
2. Add S3 push-based replicatiton tests;
3. Add filter registry keyword;

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-29 12:44:49 +00:00
danfengliu
658cfcf3a2
Merge pull request #12427 from danfengliu/add-checkpoint-to-gc-untag-case
Add checkpoint of no deletion for untagged image
2020-07-28 21:27:26 +08:00
danfengliu
5d3080c2c2 Add checkpoint of no deletion for untagged image
1. Add checkpoint of no deletion for untagged image
2. rebase master.

Signed-off-by: danfengliu <danfengl@vmware.com>
2020-07-28 15:21:28 +08:00