Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-11-01 08:09:59 +01:00
Code Issues Projects Releases Wiki Activity
b566375d58
harbor/tests/robot-cases/Group1-Nightly/Clair.robot

239 lines
11 KiB
Plaintext
Raw Normal View History

Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
// Copyright (c) 2017 VMware, Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
*** Settings ***
Documentation Harbor BATs
Resource ../../resources/Util.robot
Default Tags Nightly
*** Variables ***
${HARBOR_URL} https://${ip}
${SSH_USER} root
${HARBOR_ADMIN} admin
*** Test Cases ***
Modify nightly case of scanner Signed-off-by: Yogi_Wang <yawang@vmware.com>
2019-11-07 05:53:24 +01:00
Test Case - Clair Is Default Scanner And It Is Immutable
Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
Init Chrome Driver
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Modify nightly case of scanner Signed-off-by: Yogi_Wang <yawang@vmware.com>
2019-11-07 05:53:24 +01:00
Switch To Scanners Page
Should Display The Default Clair Scanner
Clair Is Immutable Scanner
Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
Test Case - Disable Scan Schedule
Init Chrome Driver
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Switch To Vulnerability Page
Disable Scan Schedule
Logout Harbor
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Switch To Vulnerability Page
Retry Wait Until Page Contains None
Close Browser
Test Case - Scan A Tag In The Repo
Body Of Scan A Tag In The Repo
Test Case - Scan As An Unprivileged User
Init Chrome Driver
Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} library hello-world
Sign In Harbor ${HARBOR_URL} user024 Test1@34
Go Into Project library
Go Into Repo hello-world
Select Object latest
Scan Is Disabled
Close Browser
Test Case - Scan Image With Empty Vul
Init Chrome Driver
Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} library busybox
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Go Into Project library
Go Into Repo busybox
Scan Repo latest Succeed
Move To Summary Chart
Modify nightly case of scanner Signed-off-by: Yogi_Wang <yawang@vmware.com>
2019-11-07 05:53:24 +01:00
Wait Until Page Contains No vulnerability
Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
Close Browser
Test Case - Manual Scan All
Init Chrome Driver
Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} library redis
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Switch To Vulnerability Page
Modify scan now nightly case Signed-off-by: Yogi_Wang <yawang@vmware.com>
2019-11-12 11:47:15 +01:00
Trigger Scan Now And Wait Until The Result Appears
Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
Navigate To Projects
Go Into Project library
Go Into Repo redis
Summary Chart Should Display latest
Close Browser
Test Case - View Scan Error
Init Chrome Driver
${d}= get current date result_format=%m%s
Sign In Harbor ${HARBOR_URL} user026 Test1@34
Create An New Project project${d}
Push Image ${ip} user026 Test1@34 project${d} vmware/photon:1.0
Go Into Project project${d}
Go Into Repo project${d}/vmware/photon
Scan Repo 1.0 Fail
View Scan Error Log
Close Browser
Test Case - Scan Image On Push
[Tags] run-once
Init Chrome Driver
Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} library hello-world
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Go Into Project library
Goto Project Config
Enable Scan On Push
Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} library memcached
Navigate To Projects
Go Into Project library
Go Into Repo memcached
Summary Chart Should Display latest
Close Browser
Test Case - View Scan Results
[Tags] run-once
Init Chrome Driver
${d}= get current date result_format=%m%s
Sign In Harbor ${HARBOR_URL} user025 Test1@34
Create An New Project project${d}
Push Image ${ip} user025 Test1@34 project${d} tomcat
Go Into Project project${d}
Go Into Repo project${d}/tomcat
Scan Repo latest Succeed
Summary Chart Should Display latest
View Repo Scan Details
Close Browser
Test Case - Project Level Image Serverity Policy
[Tags] run-once
Init Chrome Driver
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
${d}= get current date result_format=%m%s
Switch registry from docker-hub ot internal harbor registry In nightly test, images for tests were pulled from docker-hub, and sometime in an rare occasion, client will fail to contact docker-hub like timeout error, so we like to pull image from an internal harbor registry with no network issue invovled. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2020-01-14 04:11:12 +01:00
#For docker-hub registry
#${sha256}= Set Variable 9755880356c4ced4ff7745bafe620f0b63dd17747caedba72504ef7bac882089
#For internal CPE harbor registry
${sha256}= Set Variable 0e67625224c1da47cb3270e7a861a83e332f708d3d89dde0cbed432c94824d9a
As the background of clair updated was disabled, and we populate clair db right after harbor deployed with existing data, so all clair relevant test cases should be update as the data of this clair population. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-15 09:30:08 +01:00
${image}= Set Variable redis
Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
Create An New Project project${d}
Script test case for project quotas, there will be 2 or 3 test cases in this PR, like project quota edit, prject quota functionality. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-09-12 07:56:56 +02:00
Push Image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} sha256=${sha256}
Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
Go Into Project project${d}
Go Into Repo ${image}
Scan Repo ${sha256} Succeed
Navigate To Projects
Go Into Project project${d}
As the background of clair updated was disabled, and we populate clair db right after harbor deployed with existing data, so all clair relevant test cases should be update as the data of this clair population. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-15 09:30:08 +01:00
Set Vulnerabilty Serverity 3
Currently, nightly test pipeline was only in one level category of auth_mode and upgrade, now we like to add a level base on the original level which is harbor deployment parameters. (#8398) Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-01 09:10:54 +02:00
Cannot pull image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} tag=${sha256}
Close Browser
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
#Important Note: All CVE IDs in CVE Whitelist cases must unique!
Test Case - Verfiy System Level CVE Whitelist
[Tags] run-once
Init Chrome Driver
1.Add CVE white list API test into API_DB.robot;2.Random data for prject name should use lower case %m%s;3.Replace from harbor to self. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-09-04 07:52:56 +02:00
${d}= Get Current Date result_format=%m%s
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
${image}= Set Variable mariadb
Switch registry from docker-hub ot internal harbor registry In nightly test, images for tests were pulled from docker-hub, and sometime in an rare occasion, client will fail to contact docker-hub like timeout error, so we like to pull image from an internal harbor registry with no network issue invovled. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2020-01-14 04:11:12 +01:00
#For docker-hub registry
#${sha256}= Set Variable c396eb803be99041e69eed84b0eb880d5474a6b2c1fd5a84268ce0420088d20d
#For internal CPE harbor registry
${sha256}= Set Variable b5e273ed46d2b5a1c96bf8f3ae37aa5e90c6c481e7f7ae66744610d7df79cbd1
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
${signin_user}= Set Variable user025
${signin_pwd}= Set Variable Test1@34
Sign In Harbor ${HARBOR_URL} ${signin_user} ${signin_pwd}
Create An New Project project${d}
Script test case for project quotas, there will be 2 or 3 test cases in this PR, like project quota edit, prject quota functionality. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-09-12 07:56:56 +02:00
Push Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} sha256=${sha256}
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Go Into Project project${d}
As the background of clair updated was disabled, and we populate clair db right after harbor deployed with existing data, so all clair relevant test cases should be update as the data of this clair population. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-15 09:30:08 +01:00
Set Vulnerabilty Serverity 2
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Cannot Pull image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Go Into Project project${d}
Go Into Repo project${d}/${image}
Scan Repo ${sha256} Succeed
Logout Harbor
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Switch To Configure
Switch To Configuration System Setting
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
Add Items To System CVE Whitelist CVE-2019-13050\nCVE-2018-19591\nCVE-2018-11236\nCVE-2018-11237\nCVE-2019-13627\nCVE-2018-20839\nCVE-2019-2923\nCVE-2019-2922\nCVE-2019-2911\nCVE-2019-2914\nCVE-2019-2924\nCVE-2019-2910\nCVE-2019-2938\nCVE-2019-2993\nCVE-2019-2974\nCVE-2019-2960\nCVE-2019-2948\nCVE-2019-2946
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Cannot Pull image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
Add Items To System CVE Whitelist CVE-2019-2969
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Pull Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Replace CVE sample image from other to redis in union, redis is stable in relative, so test result will be stable relatively Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-09-09 05:52:56 +02:00
Delete Top Item In System CVE Whitelist count=6
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Cannot Pull image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Close Browser
Test Case - Verfiy Project Level CVE Whitelist
[Tags] run-once
Init Chrome Driver
1.Add CVE white list API test into API_DB.robot;2.Random data for prject name should use lower case %m%s;3.Replace from harbor to self. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-09-04 07:52:56 +02:00
${d}= Get Current Date result_format=%m%s
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
${image}= Set Variable mariadb
Switch registry from docker-hub ot internal harbor registry In nightly test, images for tests were pulled from docker-hub, and sometime in an rare occasion, client will fail to contact docker-hub like timeout error, so we like to pull image from an internal harbor registry with no network issue invovled. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2020-01-14 04:11:12 +01:00
#For docker-hub registry
#${sha256}= Set Variable c396eb803be99041e69eed84b0eb880d5474a6b2c1fd5a84268ce0420088d20d
#For internal CPE harbor registry
${sha256}= Set Variable b5e273ed46d2b5a1c96bf8f3ae37aa5e90c6c481e7f7ae66744610d7df79cbd1
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
${signin_user}= Set Variable user025
${signin_pwd}= Set Variable Test1@34
Sign In Harbor ${HARBOR_URL} ${signin_user} ${signin_pwd}
Create An New Project project${d}
Script test case for project quotas, there will be 2 or 3 test cases in this PR, like project quota edit, prject quota functionality. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-09-12 07:56:56 +02:00
Push Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} sha256=${sha256}
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Pull Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Go Into Project project${d}
As the background of clair updated was disabled, and we populate clair db right after harbor deployed with existing data, so all clair relevant test cases should be update as the data of this clair population. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-15 09:30:08 +01:00
Set Vulnerabilty Serverity 2
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Cannot Pull image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Go Into Project project${d}
Go Into Repo project${d}/${image}
Scan Repo ${sha256} Succeed
Go Into Project project${d}
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
Add Items to Project CVE Whitelist CVE-2019-13050\nCVE-2018-19591\nCVE-2018-11236\nCVE-2018-11237\nCVE-2019-13627\nCVE-2018-20839\nCVE-2019-2923\nCVE-2019-2922\nCVE-2019-2911\nCVE-2019-2914\nCVE-2019-2924\nCVE-2019-2910\nCVE-2019-2938\nCVE-2019-2993\nCVE-2019-2974\nCVE-2019-2960\nCVE-2019-2948\nCVE-2019-2946
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Cannot Pull image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
Add Items to Project CVE Whitelist CVE-2019-2969
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Pull Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Delete Top Item In Project CVE Whitelist
Cannot Pull image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Close Browser
Test Case - Verfiy Project Level CVE Whitelist By Quick Way of Add System
[Tags] run-once
Init Chrome Driver
1.Add CVE white list API test into API_DB.robot;2.Random data for prject name should use lower case %m%s;3.Replace from harbor to self. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-09-04 07:52:56 +02:00
${d}= Get Current Date result_format=%m%s
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
${image}= Set Variable mariadb
Switch registry from docker-hub ot internal harbor registry In nightly test, images for tests were pulled from docker-hub, and sometime in an rare occasion, client will fail to contact docker-hub like timeout error, so we like to pull image from an internal harbor registry with no network issue invovled. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2020-01-14 04:11:12 +01:00
#For docker-hub registry
#${sha256}= Set Variable c396eb803be99041e69eed84b0eb880d5474a6b2c1fd5a84268ce0420088d20d
#For internal CPE harbor registry
${sha256}= Set Variable b5e273ed46d2b5a1c96bf8f3ae37aa5e90c6c481e7f7ae66744610d7df79cbd1
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
${signin_user}= Set Variable user025
${signin_pwd}= Set Variable Test1@34
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Switch To Configure
Switch To Configuration System Setting
Modify 2 points, 1st is add wait time right after scan all; 2nd is add name length for project name to avoid naming repeatly. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-21 07:03:59 +01:00
Add Items To System CVE Whitelist CVE-2019-13050\nCVE-2018-19591\nCVE-2018-11236\nCVE-2018-11237\nCVE-2019-13627\nCVE-2018-20839\nCVE-2019-2923\nCVE-2019-2922\nCVE-2019-2911\nCVE-2019-2914\nCVE-2019-2924\nCVE-2019-2910\nCVE-2019-2938\nCVE-2019-2993\nCVE-2019-2974\nCVE-2019-2960\nCVE-2019-2948\nCVE-2019-2946\nCVE-2019-2969
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Logout Harbor
Sign In Harbor ${HARBOR_URL} ${signin_user} ${signin_pwd}
Create An New Project project${d}
Script test case for project quotas, there will be 2 or 3 test cases in this PR, like project quota edit, prject quota functionality. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-09-12 07:56:56 +02:00
Push Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} sha256=${sha256}
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Go Into Project project${d}
As the background of clair updated was disabled, and we populate clair db right after harbor deployed with existing data, so all clair relevant test cases should be update as the data of this clair population. Signed-off-by: Danfeng Liu (c) <danfengl@vmware.com>
2019-11-15 09:30:08 +01:00
Set Vulnerabilty Serverity 2
Add CVE Whitelist test cases into nightly test, first test case is system CVE Whitelist test. Signed-off-by: danfengliu <danfengl@vmware.com>
2019-08-21 08:12:12 +02:00
Go Into Project project${d}
Go Into Repo project${d}/${image}
Scan Repo ${sha256} Succeed
Pull Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Go Into Project project${d}
Set Project To Project Level CVE Whitelist
Cannot Pull image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Add System CVE Whitelist to Project CVE Whitelist By Add System Button Click
Pull Image ${ip} ${signin_user} ${signin_pwd} project${d} ${image} tag=${sha256}
Close Browser
Reference in New Issue Copy Permalink