diff --git a/docs/1.10/build-customize-contribute/e2e-api-python-based-scripting-guide.md b/docs/1.10/build-customize-contribute/e2e-api-python-based-scripting-guide.md index 1055a2f7a..3c4597aa6 100644 --- a/docs/1.10/build-customize-contribute/e2e-api-python-based-scripting-guide.md +++ b/docs/1.10/build-customize-contribute/e2e-api-python-based-scripting-guide.md @@ -3,7 +3,7 @@ title: E2E (API) Python Based Test Scripting Guide draft: true --- -**Preparation** +#### Preparation #### After getting Harbor source code (git clone https://github.com/goharbor/harbor.git), Harbor E2E API test scripts can be found in tests/apitests/python directory. @@ -30,7 +30,7 @@ Harbor E2E API test scripts will import python library under archive "harborclie Until now, we have all preparation work done. -**Scripting** +#### Scripting #### As you can see, we will use python library made by "make swagger_client", in this library, we have all API functions and models, but for more convenience we encapsulate one more level in archive "library", so the script structure is as bellow: @@ -47,8 +47,8 @@ As you can see, we will use python library made by "make swagger_client", in thi You can add both library code and script code, since not all APIs have been encapsulated. -**Execution Example** -``` +#### Manual Execution Example #### + root@harbor:/harbor/code/harbor# `python ./tests/apitests/python/test_add_sys_label_to_tag.py` 2020-03-11 13:40:07,269 DEBUG Starting new HTTPS connection (1): 1.1.1.1:443 @@ -67,10 +67,15 @@ header: Content-Security-Policy: frame-ancestors 'none' 2020-03-11 13:40:07,482 DEBUG https://1.1.1.1:443 "POST /api/v2.0/users HTTP/1.1" 201 0 2020-03-11 13:40:07,483 DEBUG response body: -... +...... + +...... + +...... + +#### How To Trigger Script In CI #### + +If you like to have your scripts running in CI which is for verification of pull requests, please add your scripts into *https://github.com/goharbor/harbor/blob/master/tests/robot-cases/Group0-BAT/API_DB.robot* file, then scripts can be triggered once there is a pull request. -... -... -``` diff --git a/docs/1.10/working-with-projects/project-configuration/implementing-content-trust.md b/docs/1.10/working-with-projects/project-configuration/implementing-content-trust.md index 464e7d640..93384901c 100644 --- a/docs/1.10/working-with-projects/project-configuration/implementing-content-trust.md +++ b/docs/1.10/working-with-projects/project-configuration/implementing-content-trust.md @@ -19,7 +19,7 @@ The root key is generated at: `/root/.docker/trust/private/root_keys` You will also be asked to enter a new passphrase for the image. This is generated at `/root/.docker/trust/private/tuf_keys/[registry name] /[imagepath]`. If you are using a self-signed cert, make sure to copy the CA cert into `/etc/docker/certs.d/10.117.169.182` and `$HOME/.docker/tls/10.117.169.182:4443/`. When an image is signed, it is indicated in the Web UI. -When an image is signed, it has a tick shown in UI; otherwise, a cross sign(X) is displayed instead. +A signed image will have a checkbox next to it, otherwise an X is displayed instead. If you want to remove a tag signature from harbor, you can use 'notary remove' command: diff --git a/tests/ci/distro_installer.sh b/tests/ci/distro_installer.sh index b4ee8a76f..daa76a9cb 100755 --- a/tests/ci/distro_installer.sh +++ b/tests/ci/distro_installer.sh @@ -3,5 +3,5 @@ set -x set -e -sudo make package_online VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true HTTPPROXY= -sudo make package_offline VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true HTTPPROXY= +sudo make package_online VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true TRIVYFLAG=true HTTPPROXY= +sudo make package_offline VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true TRIVYFLAG=true HTTPPROXY= diff --git a/tests/resources/Harbor-Pages/Configuration.robot b/tests/resources/Harbor-Pages/Configuration.robot index f2951c986..cd4fa9e06 100644 --- a/tests/resources/Harbor-Pages/Configuration.robot +++ b/tests/resources/Harbor-Pages/Configuration.robot @@ -334,5 +334,5 @@ Get Project Count Quota Text From Project Quotas List Get Project Storage Quota Text From Project Quotas List [Arguments] ${project_name} Switch To Project Quotas - ${storage_quota}= Get Text xpath=//project-quotas//clr-datagrid//clr-dg-row[contains(.,'${project_name}')]//clr-dg-cell[4]//label + ${storage_quota}= Get Text xpath=//project-quotas//clr-datagrid//clr-dg-row[contains(.,'${project_name}')]//clr-dg-cell[3]//label [Return] ${storage_quota} diff --git a/tests/resources/TestCaseBody.robot b/tests/resources/TestCaseBody.robot index 194970f32..4347c8bf7 100644 --- a/tests/resources/TestCaseBody.robot +++ b/tests/resources/TestCaseBody.robot @@ -133,13 +133,14 @@ Body Of Admin Push Signed Image Delete A Project Without Sign In Harbor [Arguments] ${harbor_ip}=${ip} ${username}=${HARBOR_ADMIN} ${password}=${HARBOR_PASSWORD} ${d}= Get Current Date result_format=%m%s - Create An New Project project${d} - Push Image ${harbor_ip} ${username} ${password} project${d} hello-world - Project Should Not Be Deleted project${d} - Go Into Project project${d} - Delete Repo project${d} + ${project_name}= Set Variable 000${d} + Create An New Project ${project_name} + Push Image ${harbor_ip} ${username} ${password} ${project_name} hello-world + Project Should Not Be Deleted ${project_name} + Go Into Project ${project_name} + Delete Repo ${project_name} Navigate To Projects - Project Should Be Deleted project${d} + Project Should Be Deleted ${project_name} Manage Project Member Without Sign In Harbor [Arguments] ${sign_in_user} ${sign_in_pwd} ${test_user1}=user005 ${test_user2}=user006 ${is_oidc_mode}=${false} diff --git a/tests/resources/Util.robot b/tests/resources/Util.robot index de2f44b09..ee3cd5aa3 100644 --- a/tests/resources/Util.robot +++ b/tests/resources/Util.robot @@ -228,7 +228,7 @@ Command Should be Failed Retry Keyword When Error [Arguments] ${keyword} @{elements} - :For ${n} IN RANGE 1 6 + :For ${n} IN RANGE 1 3 \ Log To Console Trying ${keyword} elements @{elements} ${n} times ... \ ${out} Run Keyword And Ignore Error ${keyword} @{elements} \ Log To Console Return value is ${out} and ${out[0]} @@ -253,7 +253,7 @@ Retry Keyword When Return Value Mismatch Retry Double Keywords When Error [Arguments] ${keyword1} ${element1} ${keyword2} ${element2} ${DoAssert}=${true} - :For ${n} IN RANGE 1 5 + :For ${n} IN RANGE 1 3 \ Log To Console Trying ${keyword1} and ${keyword2} ${n} times ... \ ${out1} Run Keyword And Ignore Error ${keyword1} ${element1} \ Capture Page Screenshot diff --git a/tests/robot-cases/Group0-BAT/API_DB.robot b/tests/robot-cases/Group0-BAT/API_DB.robot index 5cf0752d1..4288c1561 100644 --- a/tests/robot-cases/Group0-BAT/API_DB.robot +++ b/tests/robot-cases/Group0-BAT/API_DB.robot @@ -67,3 +67,5 @@ Test Case - Scan Image Harbor API Test ./tests/apitests/python/test_scan_image_artifact.py Test Case - Scan All Images Harbor API Test ./tests/apitests/python/test_system_level_scan_all.py +Test Case - Registry API + Harbor API Test ./tests/apitests/python/test_registry_api.py diff --git a/tests/robot-cases/Group1-Nightly/Common.robot b/tests/robot-cases/Group1-Nightly/Common.robot index 697f758f8..a1431b189 100644 --- a/tests/robot-cases/Group1-Nightly/Common.robot +++ b/tests/robot-cases/Group1-Nightly/Common.robot @@ -436,51 +436,14 @@ Test Case - Copy A Image Test Case - Create An New Project With Quotas Set Init Chrome Driver ${d}= Get Current Date result_format=%m%s - ${count_quota}= Set Variable 1234 ${storage_quota}= Set Variable 600 ${storage_quota_unit}= Set Variable GB Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} - Create An New Project project${d} count_quota=${count_quota} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit} - ${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d} - Should Be Equal As Strings ${count_quota_ret} 0 of ${count_quota} + Create An New Project project${d} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit} ${storage_quota_ret}= Get Project Storage Quota Text From Project Quotas List project${d} Should Be Equal As Strings ${storage_quota_ret} 0Byte of ${storage_quota}${storage_quota_unit} Close Browser -Test Case - Project Image And Chart Artifact Count Quotas Dispaly And Control - Init Chrome Driver - ${d}= Get Current Date result_format=%m%s - ${count_quota}= Set Variable 2 - ${storage_quota}= Set Variable 500 - ${storage_quota_unit}= Set Variable MB - ${image}= Set Variable redis - #For docker-hub registry - #${sha256}= Set Variable 9755880356c4ced4ff7745bafe620f0b63dd17747caedba72504ef7bac882089 - #For internal CPE harbor registry - ${sha256}= Set Variable 0e67625224c1da47cb3270e7a861a83e332f708d3d89dde0cbed432c94824d9a - - ${image_size}= Set Variable 34.14MB - Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} - Create An New Project project${d} count_quota=${count_quota} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit} - Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} sha256=${sha256} - ${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d} - Should Be Equal As Strings ${count_quota_ret} 1 of ${count_quota} - ${storage_quota_ret}= Get Project Storage Quota Text From Project Quotas List project${d} - Should Be Equal As Strings ${storage_quota_ret} ${image_size} of ${storage_quota}${storage_quota_unit} - #Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} alpine - Go Into Project project${d} - Switch To Project Charts - Upload Chart files - Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox err_msg=Quota exceeded when processing the request of adding 1 of count resource, which when updated to current usage of 2 will exceed the configured upper limit of 2 - ${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d} - Should Be Equal As Strings ${count_quota_ret} 2 of ${count_quota} - Go Into Project project${d} - Delete Repo project${d}/${image} - ${count_quota_ret}= Get Project Count Quota Text From Project Quotas List project${d} - Should Be Equal As Strings ${count_quota_ret} 1 of ${count_quota} - Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox - Close Browser - # Make sure image logstash was pushed to harbor for the 1st time, so GC will delete it. Test Case - Project Quotas Control Under GC Init Chrome Driver @@ -533,14 +496,15 @@ Test Case - Project Storage Quotas Dispaly And Control Test Case - Project Quotas Control Under Copy Init Chrome Driver ${d}= Get Current Date result_format=%m%s - ${count_quota}= Set Variable 1 ${image_a}= Set Variable redis ${image_b}= Set Variable logstash - ${image_a_ver}= Set Variable 5.0 - ${image_b_ver}= Set Variable 6.8.3 + ${image_a_ver}= Set Variable donotremove5.0 + ${image_b_ver}= Set Variable do_not_remove_6.8.3 + ${storage_quota}= Set Variable 330 + ${storage_quota_unit}= Set Variable MB Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} Create An New Project project_a_${d} - Create An New Project project_b_${d} count_quota=${count_quota} + Create An New Project project_b_${d} storage_quota=${storage_quota} storage_quota_unit=${storage_quota_unit} Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project_a_${d} ${image_a} tag=${image_a_ver} tag1=${image_a_ver} Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project_a_${d} ${image_b} tag=${image_b_ver} tag1=${image_b_ver} Go Into Project project_a_${d} @@ -625,44 +589,6 @@ Test Case - Robot Account Push image ${ip} robot${d} ${token} project${d} hello-world:latest is_robot=${true} Pull image ${ip} robot${d} ${token} project${d} hello-world:latest is_robot=${true} -Test Case - Read Only Mode - Init Chrome Driver - ${d}= Get Current Date result_format=%m%s - Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} - Create An New Project project${d} - - Enable Read Only - Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest - - Disable Read Only - Sleep 5 - Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest - Close Browser - -Test Case - Can Not Copy Image In ReadOnly Mode - Init Chrome Driver - ${random_num1}= Get Current Date result_format=%m%s - ${random_num2}= Evaluate str(random.randint(1000,9999)) modules=random - - Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} - Create An New Project project${random_num1} - Create An New Project project${random_num1}${random_num2} - - Go Into Project project${random_num1} has_image=${false} - Sleep 1 - Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${random_num1} redis ${image_tag} - Sleep 1 - Enable Read Only - Go Into Repo project${random_num1}/redis - Copy Image ${image_tag} project${random_num1}${random_num2} ${target_image_name} - Retry Wait Element Not Visible ${repo_retag_confirm_dlg} - Navigate To Projects - Go Into Project project${random_num1}${random_num2} has_image=${false} - Sleep 10 - Go Into Project project${random_num1}${random_num2} has_image=${false} - Disable Read Only - Close Browser - Test Case - Push Docker Manifest Index and Display Init Chrome Driver ${d}= Get Current Date result_format=%m%s @@ -695,6 +621,7 @@ Test Case - Push Docker Manifest Index and Display Go Into Repo test${d}/index${d} Retry Double Keywords When Error Go Into Index index_name=${null} Page Should Contain Element ${tag_table_column_os_arch} Page Should Contain Element ${artifact_rows} limit=2 + Close Browser Test Case - Push CNAB Bundle and Display Init Chrome Driver @@ -716,6 +643,7 @@ Test Case - Push CNAB Bundle and Display Go Into Repo test${d}/cnab${d} Retry Double Keywords When Error Go Into Index index_name=${null} Page Should Contain Element ${tag_table_column_os_arch} Page Should Contain Element ${artifact_rows} limit=3 + Close Browser Test Case - Push Helm Chart and Display Init Chrome Driver @@ -738,3 +666,43 @@ Test Case - Push Helm Chart and Display Go Into Project test${d} Wait Until Page Contains test${d}/${repo_name} Retry Double Keywords When Error Go Into Repo test${d}/${repo_name} Page Should Contain Element ${tag_table_column_vulnerabilities} + Close Browser + +Test Case - Can Not Copy Image In ReadOnly Mode + Init Chrome Driver + ${random_num1}= Get Current Date result_format=%m%s + ${random_num2}= Evaluate str(random.randint(1000,9999)) modules=random + + Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} + Create An New Project project${random_num1} + Create An New Project project${random_num1}${random_num2} + + Go Into Project project${random_num1} has_image=${false} + Sleep 1 + Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${random_num1} redis ${image_tag} + Sleep 1 + Enable Read Only + Go Into Repo project${random_num1}/redis + Copy Image ${image_tag} project${random_num1}${random_num2} ${target_image_name} + Retry Wait Element Not Visible ${repo_retag_confirm_dlg} + Navigate To Projects + Go Into Project project${random_num1}${random_num2} has_image=${false} + Sleep 10 + Go Into Project project${random_num1}${random_num2} has_image=${false} + Disable Read Only + Sleep 10 + Close Browser + +Test Case - Read Only Mode + Init Chrome Driver + ${d}= Get Current Date result_format=%m%s + Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} + Create An New Project project${d} + + Enable Read Only + Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest + + Disable Read Only + Sleep 5 + Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} busybox:latest + Close Browser \ No newline at end of file diff --git a/tests/robot-cases/Group1-Nightly/Upgrade.robot b/tests/robot-cases/Group1-Nightly/Upgrade.robot index 871a4013a..5138a5668 100644 --- a/tests/robot-cases/Group1-Nightly/Upgrade.robot +++ b/tests/robot-cases/Group1-Nightly/Upgrade.robot @@ -27,7 +27,7 @@ Test Case - Manage project publicity Body Of Manage project publicity Test Case - Scan A Tag In The Repo - Body Of Scan A Tag In The Repo + Body Of Scan A Tag In The Repo Use Trivy Test Case - List Helm Charts Body Of List Helm Charts diff --git a/tests/robot-cases/Group1-Nightly/multi_scanners.robot b/tests/robot-cases/Group1-Nightly/multi_scanners.robot new file mode 100644 index 000000000..6a5872527 --- /dev/null +++ b/tests/robot-cases/Group1-Nightly/multi_scanners.robot @@ -0,0 +1,29 @@ + +// Copyright (c) 2017 VMware, Inc. All Rights Reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +*** Settings *** +Documentation Harbor BATs +Resource ../../resources/Util.robot +Default Tags Nightly + +*** Variables *** +${HARBOR_URL} https://${ip} +${SSH_USER} root +${HARBOR_ADMIN} admin + +*** Test Cases *** +Test Case - Get Harbor Version +#Just get harbor version and log it + Get Harbor Version \ No newline at end of file