Upstream
/
harbor
mirror of https://github.com/goharbor/harbor.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Add Multi-Scanner test file 

1. Add Multi-Scanner test file for harbor is deployed with multi scanners;
2. Modify notary doc description;
3. Add paragraph for e2e-api-python-based-scripting-guide;
4. Fix delete project issue;
5. Remove count qoutas in nightly and modify some of it;
6. Add Trivy in git hub offline action;

Signed-off-by: danfengliu <danfengl@vmware.com>
This commit is contained in:
danfengliu 2020-04-07 17:28:09 +08:00
parent 5bcd015d6f
commit 66dfd59c80
10 changed files with 106 additions and 101 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
docs/1.10/build-customize-contribute/e2e-api-python-based-scripting-guide.md
View File

@ -3,7 +3,7 @@ title: E2E (API) Python Based Test Scripting Guide
draft: true
---
**Preparation**
#### Preparation ####
After getting Harbor source code (git clone https://github.com/goharbor/harbor.git), Harbor E2E API test scripts can be found in tests/apitests/python directory.
@ -30,7 +30,7 @@ Harbor E2E API test scripts will import python library under archive "harborclie
Until now, we have all preparation work done.
**Scripting**
#### Scripting ####
As you can see, we will use python library made by "make swagger_client", in this library, we have all API functions and models, but for more convenience we encapsulate one more level in archive "library", so the script structure is as bellow:
@ -47,8 +47,8 @@ As you can see, we will use python library made by "make swagger_client", in thi
You can add both library code and script code, since not all APIs have been encapsulated.
**Execution Example**
```
#### Manual Execution Example ####
root@harbor:/harbor/code/harbor# `python ./tests/apitests/python/test_add_sys_label_to_tag.py`
2020-03-11 13:40:07,269 DEBUG Starting new HTTPS connection (1): 1.1.1.1:443
@ -67,10 +67,15 @@ header: Content-Security-Policy: frame-ancestors 'none'
2020-03-11 13:40:07,482 DEBUG https://1.1.1.1:443 "POST /api/v2.0/users HTTP/1.1" 201 0
2020-03-11 13:40:07,483 DEBUG response body:
...
......
......
......
#### How To Trigger Script In CI ####
If you like to have your scripts running in CI which is for verification of pull requests, please add your scripts into *https://github.com/goharbor/harbor/blob/master/tests/robot-cases/Group0-BAT/API_DB.robot* file, then scripts can be triggered once there is a pull request.
...
...
```

2
docs/1.10/working-with-projects/project-configuration/implementing-content-trust.md
View File

@ -19,7 +19,7 @@ The root key is generated at: `/root/.docker/trust/private/root_keys`
You will also be asked to enter a new passphrase for the image. This is generated at `/root/.docker/trust/private/tuf_keys/[registry name] /[imagepath]`.
If you are using a self-signed cert, make sure to copy the CA cert into `/etc/docker/certs.d/10.117.169.182` and `$HOME/.docker/tls/10.117.169.182:4443/`. When an image is signed, it is indicated in the Web UI.
When an image is signed, it has a tick shown in UI; otherwise, a cross sign(X) is displayed instead.
A signed image will have a checkbox next to it, otherwise an X is displayed instead.
If you want to remove a tag signature from harbor, you can use 'notary remove' command:

4
tests/ci/distro_installer.sh
View File

@ -3,5 +3,5 @@ set -x
set -e
sudo make package_online VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true HTTPPROXY=
sudo make package_offline VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true HTTPPROXY=
sudo make package_online VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true TRIVYFLAG=true HTTPPROXY=
sudo make package_offline VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true TRIVYFLAG=true HTTPPROXY=

2
tests/resources/Harbor-Pages/Configuration.robot
View File

@ -334,5 +334,5 @@ Get Project Count Quota Text From Project Quotas List
Get Project Storage Quota Text From Project Quotas List
[Arguments] ${project_name}
Switch To Project Quotas
${storage_quota}= Get Text xpath=//project-quotas//clr-datagrid//clr-dg-row[contains(.,'${project_name}')]//clr-dg-cell[4]//label
${storage_quota}= Get Text xpath=//project-quotas//clr-datagrid//clr-dg-row[contains(.,'${project_name}')]//clr-dg-cell[3]//label
[Return] ${storage_quota}

13
tests/resources/TestCaseBody.robot
View File

@ -133,13 +133,14 @@ Body Of Admin Push Signed Image
Delete A Project Without Sign In Harbor
[Arguments] ${harbor_ip}=${ip} ${username}=${HARBOR_ADMIN} ${password}=${HARBOR_PASSWORD}
${d}= Get Current Date result_format=%m%s
Create An New Project project${d}
Push Image ${harbor_ip} ${username} ${password} project${d} hello-world
Project Should Not Be Deleted project${d}
Go Into Project project${d}
Delete Repo project${d}
${project_name}= Set Variable 000${d}
Create An New Project ${project_name}
Push Image ${harbor_ip} ${username} ${password} ${project_name} hello-world
Project Should Not Be Deleted ${project_name}
Go Into Project ${project_name}
Delete Repo ${project_name}
Navigate To Projects
Project Should Be Deleted project${d}
Project Should Be Deleted ${project_name}
Manage Project Member Without Sign In Harbor
[Arguments] ${sign_in_user} ${sign_in_pwd} ${test_user1}=user005 ${test_user2}=user006 ${is_oidc_mode}=${false}

4
tests/resources/Util.robot
View File

@ -228,7 +228,7 @@ Command Should be Failed
Retry Keyword When Error
[Arguments] ${keyword} @{elements}
:For ${n} IN RANGE 1 6
:For ${n} IN RANGE 1 3
\ Log To Console Trying ${keyword} elements @{elements} ${n} times ...
\ ${out} Run Keyword And Ignore Error ${keyword} @{elements}
\ Log To Console Return value is ${out} and ${out[0]}
@ -253,7 +253,7 @@ Retry Keyword When Return Value Mismatch
Retry Double Keywords When Error
[Arguments] ${keyword1} ${element1} ${keyword2} ${element2} ${DoAssert}=${true}
:For ${n} IN RANGE 1 5
:For ${n} IN RANGE 1 3
\ Log To Console Trying ${keyword1} and ${keyword2} ${n} times ...
\ ${out1} Run Keyword And Ignore Error ${keyword1} ${element1}
\ Capture Page Screenshot

2
tests/robot-cases/Group0-BAT/API_DB.robot
View File

@ -67,3 +67,5 @@ Test Case - Scan Image
Harbor API Test ./tests/apitests/python/test_scan_image_artifact.py
Test Case - Scan All Images
Harbor API Test ./tests/apitests/python/test_system_level_scan_all.py
Test Case - Registry API
Harbor API Test ./tests/apitests/python/test_registry_api.py

128
tests/robot-cases/Group1-Nightly/Common.robot
View File

@ -436,51 +436,14 @@ Test Case - Copy A Image
Test Case - Create An New Project With Quotas Set
Init Chrome Driver
${d}= Get Current Date result_format=%m%s
${count_quota}= Set Variable 1234
${storage_quota}= Set Variable 600
${storage_quota_unit}= Set Variable GB
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project project${d} count_quota=${count_quota} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit}
${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d}
Should Be Equal As Strings ${count_quota_ret} 0 of ${count_quota}
Create An New Project project${d} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit}
${storage_quota_ret}= Get Project Storage Quota Text From Project Quotas List project${d}
Should Be Equal As Strings ${storage_quota_ret} 0Byte of ${storage_quota}${storage_quota_unit}
Close Browser
Test Case - Project Image And Chart Artifact Count Quotas Dispaly And Control
Init Chrome Driver
${d}= Get Current Date result_format=%m%s
${count_quota}= Set Variable 2
${storage_quota}= Set Variable 500
${storage_quota_unit}= Set Variable MB
${image}= Set Variable redis
#For docker-hub registry
#${sha256}= Set Variable 9755880356c4ced4ff7745bafe620f0b63dd17747caedba72504ef7bac882089
#For internal CPE harbor registry
${sha256}= Set Variable 0e67625224c1da47cb3270e7a861a83e332f708d3d89dde0cbed432c94824d9a
${image_size}= Set Variable 34.14MB
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project project${d} count_quota=${count_quota} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit}
Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} sha256=${sha256}
${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d}
Should Be Equal As Strings ${count_quota_ret} 1 of ${count_quota}
${storage_quota_ret}= Get Project Storage Quota Text From Project Quotas List project${d}
Should Be Equal As Strings ${storage_quota_ret} ${image_size} of ${storage_quota}${storage_quota_unit}
#Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} alpine
Go Into Project project${d}
Switch To Project Charts
Upload Chart files
Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox err_msg=Quota exceeded when processing the request of adding 1 of count resource, which when updated to current usage of 2 will exceed the configured upper limit of 2
${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d}
Should Be Equal As Strings ${count_quota_ret} 2 of ${count_quota}
Go Into Project project${d}
Delete Repo project${d}/${image}
${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d}
Should Be Equal As Strings ${count_quota_ret} 1 of ${count_quota}
Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox
Close Browser
# Make sure image logstash was pushed to harbor for the 1st time, so GC will delete it.
Test Case - Project Quotas Control Under GC
Init Chrome Driver
@ -533,14 +496,15 @@ Test Case - Project Storage Quotas Dispaly And Control
Test Case - Project Quotas Control Under Copy
Init Chrome Driver
${d}= Get Current Date result_format=%m%s
${count_quota}= Set Variable 1
${image_a}= Set Variable redis
${image_b}= Set Variable logstash
${image_a_ver}= Set Variable 5.0
${image_b_ver}= Set Variable 6.8.3
${image_a_ver}= Set Variable donotremove5.0
${image_b_ver}= Set Variable do_not_remove_6.8.3
${storage_quota}= Set Variable 330
${storage_quota_unit}= Set Variable MB
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project project_a_${d}
Create An New Project project_b_${d} count_quota=${count_quota}
Create An New Project project_b_${d} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit}
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project_a_${d} ${image_a} tag=${image_a_ver} tag1=${image_a_ver}
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project_a_${d} ${image_b} tag=${image_b_ver} tag1=${image_b_ver}
Go Into Project project_a_${d}
@ -625,44 +589,6 @@ Test Case - Robot Account
Push image ${ip} robot${d} ${token} project${d} hello-world:latest is_robot=${true}
Pull image ${ip} robot${d} ${token} project${d} hello-world:latest is_robot=${true}
Test Case - Read Only Mode
Init Chrome Driver
${d}= Get Current Date result_format=%m%s
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project project${d}
Enable Read Only
Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest
Disable Read Only
Sleep 5
Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest
Close Browser
Test Case - Can Not Copy Image In ReadOnly Mode
Init Chrome Driver
${random_num1}= Get Current Date result_format=%m%s
${random_num2}= Evaluate str(random.randint(1000,9999)) modules=random
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project project${random_num1}
Create An New Project project${random_num1}${random_num2}
Go Into Project project${random_num1} has_image=${false}
Sleep 1
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${random_num1} redis ${image_tag}
Sleep 1
Enable Read Only
Go Into Repo project${random_num1}/redis
Copy Image ${image_tag} project${random_num1}${random_num2} ${target_image_name}
Retry Wait Element Not Visible ${repo_retag_confirm_dlg}
Navigate To Projects
Go Into Project project${random_num1}${random_num2} has_image=${false}
Sleep 10
Go Into Project project${random_num1}${random_num2} has_image=${false}
Disable Read Only
Close Browser
Test Case - Push Docker Manifest Index and Display
Init Chrome Driver
${d}= Get Current Date result_format=%m%s
@ -695,6 +621,7 @@ Test Case - Push Docker Manifest Index and Display
Go Into Repo test${d}/index${d}
Retry Double Keywords When Error Go Into Index index_name=${null} Page Should Contain Element ${tag_table_column_os_arch}
Page Should Contain Element ${artifact_rows} limit=2
Close Browser
Test Case - Push CNAB Bundle and Display
Init Chrome Driver
@ -716,6 +643,7 @@ Test Case - Push CNAB Bundle and Display
Go Into Repo test${d}/cnab${d}
Retry Double Keywords When Error Go Into Index index_name=${null} Page Should Contain Element ${tag_table_column_os_arch}
Page Should Contain Element ${artifact_rows} limit=3
Close Browser
Test Case - Push Helm Chart and Display
Init Chrome Driver
@ -738,3 +666,43 @@ Test Case - Push Helm Chart and Display
Go Into Project test${d}
Wait Until Page Contains test${d}/${repo_name}
Retry Double Keywords When Error Go Into Repo test${d}/${repo_name} Page Should Contain Element ${tag_table_column_vulnerabilities}
Close Browser
Test Case - Can Not Copy Image In ReadOnly Mode
Init Chrome Driver
${random_num1}= Get Current Date result_format=%m%s
${random_num2}= Evaluate str(random.randint(1000,9999)) modules=random
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project project${random_num1}
Create An New Project project${random_num1}${random_num2}
Go Into Project project${random_num1} has_image=${false}
Sleep 1
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${random_num1} redis ${image_tag}
Sleep 1
Enable Read Only
Go Into Repo project${random_num1}/redis
Copy Image ${image_tag} project${random_num1}${random_num2} ${target_image_name}
Retry Wait Element Not Visible ${repo_retag_confirm_dlg}
Navigate To Projects
Go Into Project project${random_num1}${random_num2} has_image=${false}
Sleep 10
Go Into Project project${random_num1}${random_num2} has_image=${false}
Disable Read Only
Sleep 10
Close Browser
Test Case - Read Only Mode
Init Chrome Driver
${d}= Get Current Date result_format=%m%s
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project project${d}
Enable Read Only
Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest
Disable Read Only
Sleep 5
Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest
Close Browser

2
tests/robot-cases/Group1-Nightly/Upgrade.robot
View File

@ -27,7 +27,7 @@ Test Case - Manage project publicity
Body Of Manage project publicity
Test Case - Scan A Tag In The Repo
Body Of Scan A Tag In The Repo
Body Of Scan A Tag In The Repo Use Trivy
Test Case - List Helm Charts
Body Of List Helm Charts

29
tests/robot-cases/Group1-Nightly/multi_scanners.robot Normal file
View File

@ -0,0 +1,29 @@
// Copyright (c) 2017 VMware, Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
*** Settings ***
Documentation Harbor BATs
Resource ../../resources/Util.robot
Default Tags Nightly
*** Variables ***
${HARBOR_URL} https://${ip}
${SSH_USER} root
${HARBOR_ADMIN} admin
*** Test Cases ***
Test Case - Get Harbor Version
#Just get harbor version and log it
Get Harbor Version