Upstream
/
harbor
mirror of https://github.com/goharbor/harbor.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Enhance webhook testcase (#17031) 

Increase retry to improve test case stability.

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
This commit is contained in:
Yang Jiao 2022-06-21 14:28:43 +08:00 committed by GitHub
parent 7024442f25
commit a594f5649d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 151 additions and 108 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
tests/resources/Harbor-Pages/Project-Webhooks.robot
View File

@ -11,9 +11,9 @@ Switch To Project Webhooks
Sleep 1
Create A New Webhook
[Arguments] ${webhook_name} ${webhook_endpoint_url} @{event_type}
Retry Element Click ${new_webhook_button_xpath}
Retry Text Input ${webhook_name_xpath} ${webhook_name}
[Arguments] ${webhook_name} ${webhook_endpoint_url} ${event_type}=@{EMPTY}
Retry Element Click ${new_webhook_button_xpath}
Retry Text Input ${webhook_name_xpath} ${webhook_name}
Retry Text Input ${webhook_endpoint_id_xpath} ${webhook_endpoint_url}
${len}= Get Length ${event_type}
Run Keyword If ${len} > 0 Select Event Type @{event_type}
@ -22,7 +22,7 @@ Create A New Webhook
Select Event Type
[Arguments] @{event_type}
@{elements}= Get WebElements //form//div[contains(@class,'clr-control-inline')]//label
${elements}= Get WebElements //form//div[contains(@class,'clr-control-inline')]//label
FOR ${element} IN @{elements}
Retry Element Click ${element}
END

115
tests/resources/TestCaseBody.robot
View File

@ -475,3 +475,118 @@ Body Of Stop Scan And Stop Scan All
Prepare Image Package Test Files
[Arguments] ${files_path}
${rc} ${output}= Run And Return Rc And Output bash tests/robot-cases/Group0-Util/prepare_imgpkg_test_files.sh ${files_path}
Verify Webhook By Artifact Pushed Event
[Arguments] ${project_name} ${image} ${tag} ${user} ${pwd} ${webhook_handle}
Switch Window ${webhook_handle}
Delete All Requests
Push Image With Tag ${ip} ${user} ${pwd} ${project_name} ${image} ${tag}
&{artifact_pushed_property}= Create Dictionary type=PUSH_ARTIFACT operator=${user} namespace=${project_name} name=${image} tag=${tag}
Verify Request &{artifact_pushed_property}
Clean All Local Images
Verify Webhook By Artifact Pulled Event
[Arguments] ${project_name} ${image} ${tag} ${user} ${pwd} ${webhook_handle}
Switch Window ${webhook_handle}
Delete All Requests
Clean All Local Images
Docker Login ${ip} ${user} ${pwd}
Docker Pull ${ip}/${project_name}/${image}:${tag}
Docker Logout ${ip}
&{artifact_pulled_property}= Create Dictionary type=PULL_ARTIFACT operator=${user} namespace=${project_name} name=${image}
Verify Request &{artifact_pulled_property}
Verify Webhook By Artifact Deleted Event
[Arguments] ${project_name} ${image} ${tag} ${user} ${harbor_handle} ${webhook_handle}
Switch Window ${webhook_handle}
Delete All Requests
Switch Window ${harbor_handle}
Go Into Project ${project_name}
Go Into Repo ${project_name}/${image}
@{tag_list} Create List ${tag}
Multi-delete Artifact @{tag_list}
Switch Window ${webhook_handle}
&{artifact_deleted_property}= Create Dictionary type=DELETE_ARTIFACT operator=${user} namespace=${project_name} name=${image} tag=${tag}
Verify Request &{artifact_deleted_property}
Verify Webhook By Scanning Finished Event
[Arguments] ${project_name} ${image} ${tag} ${harbor_handle} ${webhook_handle}
Switch Window ${webhook_handle}
Delete All Requests
Switch Window ${harbor_handle}
Go Into Project ${project_name}
Go Into Repo ${project_name}/${image}
Scan Repo ${tag} Succeed
Switch Window ${webhook_handle}
&{scanning_finished_property}= Create Dictionary type=SCANNING_COMPLETED scan_status=Success namespace=${project_name} tag=${tag} name=${image}
Verify Request &{scanning_finished_property}
Verify Webhook By Scanning Stopped Event
[Arguments] ${project_name} ${image} ${tag} ${harbor_handle} ${webhook_handle}
Switch Window ${webhook_handle}
Delete All Requests
Switch Window ${harbor_handle}
Scan Artifact ${project_name} ${image}
Stop Scan Artifact
Check Scan Artifact Job Status Is Stopped
Switch Window ${webhook_handle}
&{scanning_stopped_property}= Create Dictionary type=SCANNING_STOPPED scan_status=Stopped namespace=${project_name} tag=${tag} name=${image}
Verify Request &{scanning_stopped_property}
Verify Webhook By Tag Retention Finished Event
[Arguments] ${project_name} ${image} ${tag1} ${tag2} ${harbor_handle} ${webhook_handle}
Switch Window ${webhook_handle}
Delete All Requests
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} ${project_name} ${image} ${tag1} ${tag1}
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} ${project_name} ${image} ${tag2} ${tag2}
Switch Window ${harbor_handle}
Go Into Project ${project_name}
Switch To Tag Retention
Execute Run ${image}
Switch Window ${webhook_handle}
&{tag_retention_finished_property}= Create Dictionary type=TAG_RETENTION operator=MANUAL project_name=${project_name} name_tag=${image}:${tag2} status=SUCCESS
Verify Request &{tag_retention_finished_property}
Verify Webhook By Replication Finished Event
[Arguments] ${project_name} ${project_dest_name} ${replication_rule_name} ${harbor_handle} ${webhook_handle}
Switch Window ${webhook_handle}
Delete All Requests
Switch Window ${harbor_handle}
Switch To Replication Manage
Select Rule And Replicate ${replication_rule_name}
Retry Wait Until Page Contains Succeeded
Switch Window ${webhook_handle}
&{replication_finished_property}= Create Dictionary type=REPLICATION operator=MANUAL registry_type=harbor harbor_hostname=${ip}
Verify Request &{replication_finished_property}
Verify Webhook By Quota Near Threshold Event And Quota Exceed Event
[Arguments] ${webhook_endpoint_url} ${harbor_handle} ${webhook_handle}
${d}= Get Current Date result_format=%m%s
${image}= Set Variable nginx
${tag1}= Set Variable 1.17.6
${tag2}= Set Variable 1.14.0
${storage_quota}= Set Variable 50
Create An New Project And Go Into Project project${d} storage_quota=${storage_quota} storage_quota_unit=MiB
Switch To Project Webhooks
${event_type} Create List Quota near threshold
Create A New Webhook webhook${d} ${webhook_endpoint_url} ${event_type}
Switch Window ${webhook_handle}
Delete All Requests
# Quota near threshold
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} ${tag1} ${tag1}
&{quota_near_threshold_property}= Create Dictionary type=QUOTA_WARNING name=nginx namespace=project${d}
Verify Request &{quota_near_threshold_property}
Retry Action Keyword Verify Webhook By Quota Exceed Event project${d} webhook${d} ${image} ${tag2} ${webhook_endpoint_url} ${storage_quota} ${harbor_handle} ${webhook_handle}
Verify Webhook By Quota Exceed Event
[Arguments] ${project_name} ${webhook_name} ${image} ${tag} ${webhook_endpoint_url} ${storage_quota} ${harbor_handle} ${webhook_handle}
# Quota exceed
Switch Window ${harbor_handle}
Delete A Webhook ${webhook_name}
${event_type} Create List Quota exceed
Create A New Webhook ${webhook_name} ${webhook_endpoint_url} ${event_type}
Switch Window ${webhook_handle}
Delete All Requests
Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} ${project_name} ${image}:${tag} err_msg=adding 21.1 MiB of storage resource, which when updated to current usage of 48.5 MiB will exceed the configured upper limit of ${storage_quota}.0 MiB.
&{quota_exceed_property}= Create Dictionary type=QUOTA_EXCEED name=${image} namespace=${project_name}
Verify Request &{quota_exceed_property}

136
tests/robot-cases/Group1-Nightly/Webhook.robot
View File

@ -38,52 +38,32 @@ Test Case - Webhook CRUD
Close Browser
Test Case - Artifact Event Type Webhook Functionality
[Tags] artifact_webhook
[Tags] artifact_webhook need_webhook_endpoint
Init Chrome Driver
${image}= Set Variable busybox
${tag}= Set Variable latest
${digest}= Set Variable sha256:34efe68cca33507682b1673c851700ec66839ecf94d19b928176e20d20e02413
${d}= Get Current Date result_format=%m%s
Go To http://${WEBHOOK_ENDPOINT}
Delete All Requests
${webhook_endpoint_url}= Get Text //p//code
New Tab
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project And Go Into Project project${d}
Switch To Project Webhooks
@{event_type} Create List Artifact deleted Artifact pulled Artifact pushed
Create A New Webhook webhook${d} ${webhook_endpoint_url} @{event_type}
${event_type} Create List Artifact deleted Artifact pulled Artifact pushed
Create A New Webhook webhook${d} ${webhook_endpoint_url} ${event_type}
${handles}= Get Window Handles
${webhook_handle}= Set Variable ${handles}[0]
${harbor_handle}= Set Variable ${handles}[1]
# Artifact pushed
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} ${tag}
Switch Window ${webhook_handle}
&{artifact_pushed_property}= Create Dictionary type=PUSH_ARTIFACT operator=${HARBOR_ADMIN} namespace=project${d} name=${image} tag=${tag}
Verify Request &{artifact_pushed_property}
Delete All Requests
Clean All Local Images
Retry Action Keyword Verify Webhook By Artifact Pushed Event project${d} ${image} ${tag} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} ${webhook_handle}
# Artifact pulled
Docker Login ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Docker Pull ${ip}/project${d}/${image}:${tag}
Docker Logout ${ip}
&{artifact_pulled_property}= Create Dictionary type=PULL_ARTIFACT operator=${HARBOR_ADMIN} namespace=project${d} name=${image} tag=${digest}
Verify Request &{artifact_pulled_property}
Delete All Requests
Retry Action Keyword Verify Webhook By Artifact Pulled Event project${d} ${image} ${tag} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} ${webhook_handle}
# Artifact deleted
Switch Window ${harbor_handle}
Go Into Project project${d}
Go Into Repo project${d}/${image}
@{tag_list} Create List ${tag}
Multi-delete Artifact @{tag_list}
Switch Window ${webhook_handle}
&{artifact_deleted_property}= Create Dictionary type=DELETE_ARTIFACT operator=${HARBOR_ADMIN} namespace=project${d} name=${image} tag=${tag}
Verify Request &{artifact_deleted_property}
Delete All Requests
Retry Action Keyword Verify Webhook By Artifact Deleted Event project${d} ${image} ${tag} ${HARBOR_ADMIN} ${harbor_handle} ${webhook_handle}
Close Browser
Test Case - Scan Event Type Webhook Functionality
[Tags] scan_webhook
[Tags] scan_webhook need_webhook_endpoint
Init Chrome Driver
${image1}= Set Variable busybox
${tag1}= Set Variable latest
@ -91,7 +71,6 @@ Test Case - Scan Event Type Webhook Functionality
${tag2}= Set Variable latest-api
${d}= Get Current Date result_format=%m%s
Go To http://${WEBHOOK_ENDPOINT}
Delete All Requests
${webhook_endpoint_url}= Get Text //p//code
New Tab
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
@ -99,109 +78,58 @@ Test Case - Scan Event Type Webhook Functionality
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image1} ${tag1} ${tag1}
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image2} ${tag2} ${tag2}
Switch To Project Webhooks
@{event_type} Create List Scanning finished Scanning stopped
Create A New Webhook webhook${d} ${webhook_endpoint_url} @{event_type}
${event_type} Create List Scanning finished Scanning stopped
Create A New Webhook webhook${d} ${webhook_endpoint_url} ${event_type}
${handles}= Get Window Handles
${webhook_handle}= Set Variable ${handles}[0]
${harbor_handle}= Set Variable ${handles}[1]
# Scanning finished
Switch To Project Repo
Go Into Repo project${d}/${image1}
Scan Repo ${tag1} Succeed
Switch Window ${webhook_handle}
&{scanning_finished_property}= Create Dictionary type=SCANNING_COMPLETED scan_status=Success namespace=project${d} tag=${tag1} name=${image1}
Verify Request &{scanning_finished_property}
Delete All Requests
Retry Action Keyword Verify Webhook By Scanning Finished Event project${d} ${image1} ${tag1} ${harbor_handle} ${webhook_handle}
# Scanning stopped
Switch Window ${harbor_handle}
Scan Artifact project${d} ${image2}
Stop Scan Artifact
Check Scan Artifact Job Status Is Stopped
Switch Window ${webhook_handle}
&{scanning_stopped_property}= Create Dictionary type=SCANNING_STOPPED scan_status=Stopped namespace=project${d} tag=${tag2} name=${image2}
Verify Request &{scanning_stopped_property}
Delete All Requests
Retry Action Keyword Verify Webhook By Scanning Stopped Event project${d} ${image2} ${tag2} ${harbor_handle} ${webhook_handle}
Close Browser
Test Case - Tag Retention And Replication Event Type Webhook Functionality
[Tags] tag_retention_replication_webhook
[Tags] tag_retention_replication_webhook need_webhook_endpoint
Init Chrome Driver
${image}= Set Variable busybox
${tag1}= Set Variable latest
${tag2}= Set Variable stable
${d}= Get Current Date result_format=%m%s
Go To http://${WEBHOOK_ENDPOINT}
Delete All Requests
${webhook_endpoint_url}= Get Text //p//code
New Tab
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project And Go Into Project project_dest${d}
Create An New Project And Go Into Project project${d}
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} ${tag1} ${tag1}
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} ${tag2} ${tag2}
Switch To Project Webhooks
@{event_type} Create List Tag retention finished Replication finished
Create A New Webhook webhook${d} ${webhook_endpoint_url} @{event_type}
${handles}= Get Window Handles
${webhook_handle}= Set Variable ${handles}[0]
${harbor_handle}= Set Variable ${handles}[1]
# Tag retention finished
Switch To Tag Retention
Add A Tag Retention Rule
Edit A Tag Retention Rule ** ${tag1}
Execute Run ${image}
Switch Window ${webhook_handle}
&{tag_retention_finished_property}= Create Dictionary type=TAG_RETENTION operator=MANUAL project_name=project${d} name_tag=${image}:${tag2} status=SUCCESS
Verify Request &{tag_retention_finished_property}
Delete All Requests
# Replication finished
Switch Window ${harbor_handle}
Switch To Project Webhooks
${event_type} Create List Tag retention finished Replication finished
Create A New Webhook webhook${d} ${webhook_endpoint_url} ${event_type}
Switch To Registries
Create A New Endpoint harbor e${d} https://${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Switch To Replication Manage
Create A Rule With Existing Endpoint rule_push_${d} push project${d}/* image e${d} project_push_dest${d}
Select Rule And Replicate rule_push_${d}
Retry Wait Until Page Contains Succeeded
Switch Window ${webhook_handle}
&{replication_finished_property}= Create Dictionary type=REPLICATION operator=MANUAL registry_type=harbor harbor_hostname=${ip}
Verify Request &{replication_finished_property}
Delete All Requests
Close Browser
Test Case - Tag Quota Event Type Webhook Functionality
[Tags] quota_webhook
Init Chrome Driver
${d}= Get Current Date result_format=%m%s
${image}= Set Variable nginx
${tag1}= Set Variable 1.17.6
${tag2}= Set Variable 1.14.0
${storage_quota}= Set Variable 50
Go To http://${WEBHOOK_ENDPOINT}
Delete All Requests
${webhook_endpoint_url}= Get Text //p//code
New Tab
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Create An New Project And Go Into Project project${d} storage_quota=${storage_quota} storage_quota_unit=MiB
Switch To Project Webhooks
@{event_type} Create List Quota near threshold
Create A New Webhook webhook${d} ${webhook_endpoint_url} @{event_type}
${handles}= Get Window Handles
${webhook_handle}= Set Variable ${handles}[0]
${harbor_handle}= Set Variable ${handles}[1]
# Quota near threshold
Push Image With Tag ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image} ${tag1} ${tag1}
Switch Window ${webhook_handle}
&{quota_near_threshold_property}= Create Dictionary type=QUOTA_WARNING name=nginx namespace=project${d}
Verify Request &{quota_near_threshold_property}
Delete All Requests
# Quota exceed
Switch Window ${harbor_handle}
Delete A Webhook webhook${d}
@{event_type} Create List Quota exceed
Create A New Webhook webhook${d} ${webhook_endpoint_url} @{event_type}
Cannot Push image ${ip} ${HARBOR_ADMIN} ${HARBOR_PASSWORD} project${d} ${image}:${tag2} err_msg=adding 21.1 MiB of storage resource, which when updated to current usage of 48.5 MiB will exceed the configured upper limit of ${storage_quota}.0 MiB.
Switch Window ${webhook_handle}
&{quota_exceed_property}= Create Dictionary type=QUOTA_EXCEED name=nginx namespace=project${d}
Verify Request &{quota_exceed_property}
Delete All Requests
# Tag retention finished
Retry Action Keyword Verify Webhook By Tag Retention Finished Event project${d} ${image} ${tag1} ${tag2} ${harbor_handle} ${webhook_handle}
# Replication finished
Retry Action Keyword Verify Webhook By Replication Finished Event project${d} project_push_dest${d} rule_push_${d} ${harbor_handle} ${webhook_handle}
Close Browser
Test Case - Tag Quota Event Type Webhook Functionality
[Tags] quota_webhook need_webhook_endpoint
Init Chrome Driver
Go To http://${WEBHOOK_ENDPOINT}
${webhook_endpoint_url}= Get Text //p//code
New Tab
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
${handles}= Get Window Handles
${webhook_handle}= Set Variable ${handles}[0]
${harbor_handle}= Set Variable ${handles}[1]
Retry Action Keyword Verify Webhook By Quota Near Threshold Event And Quota Exceed Event ${webhook_endpoint_url} ${harbor_handle} ${webhook_handle}
Close Browser