Commit Graph

258 Commits

Author SHA1 Message Date
Chlins Zhang
62bb56daf6
refactor: refresh the execution status in async mode (#18434)
Refactor the logic for updating the status of execution when receiving
the hook from jobservice, avoid the optimistic lock due to the multiple
tasks update one execution by refreshing the status asynchronously. But
still retain the old way by specifying the flag from ENV.

Fixes: #17584

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-04-04 16:37:37 +08:00
Chlins Zhang
f21b1481bb
refactor: refactor the old goroutine execution sweep (#18361)
refactor: refactor the old goroutine execution sweep to global execution sweep job

1. Delete the old goroutine execution sweeper when create execution.(in the case of high concurrency can cause goroutine backlogs, affect the performance of core)
2. Introduce the new way to sweep executions, a global scheduled job will take the work.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-21 10:10:55 +08:00
Yang Jiao
660e1d1362
Remove chartmuseum test case (#18267)
Harbor deprecates chartmuseum as of v2.8.0
Epic: #17958
Discussion: #15057

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2023-02-24 10:56:28 +08:00
stonezdj(Daojun Zhang)
58054c0ce8
Change queue name for purge audit log and filter the old one (#18182)
Fixes #18121
 Refactor job name with VendorType prefix, make sure job queue name and vendor type in execution and task are identical

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-02-20 16:03:22 +08:00
Yang Jiao
2b82f99986
Add JobService Dashboard API test cases (#18234)
Added test cases for the following APIs:
1. GET /jobservice/pools/{pool_id}/workers Get workers
2. PUT /jobservice/jobs/{job_id} Stop running jc
3. PUT /jobservice/queues/{job_type} stop and clean, pause, resume pending jobs in the queue
4. GET /jobservice/queues list job queues
5. GET /jobservice/pools Get worker pools
6. GET /schedules List schedules
7. GET /schedules/{job_type}/paused Get scheduler paused status

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2023-02-17 14:56:13 +08:00
Wang Yan
738fde7d3b
remove chartmuseum backend (#18191)
Harbor deprecates chartmuseum as of v2.8.0

Epic: https://github.com/goharbor/harbor/issues/17958

Discussion: https://github.com/goharbor/harbor/discussions/15057

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-02-16 18:11:05 +08:00
Yang Jiao
fd5cb73e26
Modify audit log forward test case (#17780)
Modify the audit log forward test case to forward the audit log to elk

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-11-18 18:03:46 +08:00
Chlins Zhang
321a9abfb3
fix: skip to push system artifact for empty CSV file (#17816)
1. Skip to push system artifact to the distribution when the exported CSV file is empty.
2. Add status message for cve export execution.

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-18 14:16:36 +08:00
Yang Jiao
cf036df68b
Add metrics test case (#17795)
Add more metrics validation

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-11-15 11:19:34 +08:00
Yang Jiao
484abd6213
Update e2e engine image (#17793)
1. Update ubuntu version to 20.04
2. Update containerd version to 1.6.9
3. Update docker-compose version to 2.12.2
4. Update helm version to 3.10.1
5. Update ORAS version to 0.16.0
6. Update CNAB version to 0.3.7
7. Update imgpkg version to 0.33.0
8. Update cosign version to 1.13.1
9. Remove singularity

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-11-14 14:02:23 +08:00
Yang Jiao
180a967416
Remove singularity testcase (#17664)
Singularity has been renamed to Apptainer

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-10-20 09:54:51 +08:00
Yang Jiao
7bbefca8c2
Add scan data export API test case (#17603)
1. Add export scan data API test case
2. Add get scan data export execution list test case
3. Add get scan data export execution test case
4. Add download scan data test case

Signed-off-by: Yang Jiao <jiaoya@vmware.com>

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-10-17 13:25:03 +08:00
Yang Jiao
b8b2c2a4ac
Add audit log forword API test case (#17589)
Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-09-27 10:45:08 +08:00
Yang Jiao
a56d927143
Remove the log in testcase (#17500)
Remove logs that do not need to be printed

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-09-06 14:15:45 +08:00
Yang Jiao
1dd2b0bc7c
Add purge audit log API test cases (#17175)
Added test cases for the following APIs:
1. PUT /system/purgeaudit/{purge_id}  Stop the specific purge audit log execution
2. GET /system/purgeaudit/{purge_id}  Get purge job status
3. GET /system/purgeaudit/{purge_id}/log  Get purge job log
4. PUT /system/purgeaudit/schedule   Update purge job's schedule
5. POST /system/purgeaudit/schedule  Create a purge job schedule
6. GET /system/purgeaudit/schedule  Get purge's schedule
7. GET /system/purgeaudit  Get purge job results.

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-08-01 16:52:57 +08:00
Yang Jiao
6dfdda7778 Add retries to test cases
Retry to solve the ecs environment s3aws Path not found issue

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-04-12 07:16:55 +00:00
Yang Jiao
5f7f6c7d59 Add cosign Artifact API test case
Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2022-01-20 02:48:56 +00:00
Yang Jiao
72be9df283 Add test cases helm3.7 CLI push in python test
Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2021-10-27 18:59:08 +08:00
Yang Jiao
60ab3aeb27
Merge pull request #15832 from YangJiao0817/fix-python-testcase-security
Fix issues in python api test
2021-10-26 11:16:26 +08:00
Yang Jiao
9178ba1138 Fix issues in python api test
Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2021-10-23 13:26:48 +08:00
He Weiwei
619d3e818c feat: bump up trivy adater to v0.23.0
Closes #14983

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-10-22 09:07:40 +00:00
Shengwen YU
3f75f0db32
feat: add python testing script for stop scan job (#15779)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>

Co-authored-by: Shengwen Yu <yshengwen@vmware.com>
2021-10-18 13:13:53 +08:00
Yang Jiao
e79dd316cd Update helm cm-push plugin
Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2021-10-08 15:27:44 +08:00
Wang Yan
dd6fb049f5
drop adminjob router (#15326)
As the admin job has already droped from v2.3, remove the request handling from router.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-07-19 17:12:31 +08:00
danfengliu
344e51d2c6 Fix build base workflow issues
Signed-off-by: danfengliu <danfengl@vmware.com>
2021-05-25 18:13:15 +08:00
danfengliu
3ef4dc17fc Fix notary trust directory issue and add login for each base image in makefile
1. Use root instead of ~ in notary parameter;
2. Fix tag immutability issue caused by GUI change;
3. Replace email domain name to harbor test;
4. Add login for each base image in makefile;
5. Add customize look test in nightly.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-05-20 18:23:13 +08:00
stonezdj(Daojun Zhang)
c7852df8d3
Remove unused items in legacy_swagger.yaml (#14785)
Also updated the api test

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-05-06 10:55:33 +08:00
stonezdj(Daojun Zhang)
46dcf7b0cb
Add missing operationId in swagger.yaml (#14722)
Add operationId for systeminfo api
Add operationId for configure api
Update api test after adding operationId

Fixes #14617

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-27 12:35:00 +08:00
danfengliu
f0ebd17994 Add build base image step in build package git action workflow
Build base image step should be in build package workflow, and local base images build by new step should be removed since images have been pushed to docker hub.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-04-17 18:10:44 +08:00
Wang Yan
6e3c9e29df
Api refactor label (#14650)
* Refactor labl api

move to the new program model

Signed-off-by: wang yan <wangyan@vmware.com>

* continue resolve review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-04-15 17:27:58 +08:00
Wenkai Yin
654f4d9202 Refactor health check API
Refactor the health check API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-04-15 14:50:27 +08:00
He Weiwei
dc37c83e11
refactor: use singular as the tag for user APIs (#14654)
Use singular as the tag for user APIs to align with other APIs.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-04-15 10:52:48 +08:00
stonezdj
751d404519 Refactor project member api to new programming model
Add pkg/member/dao
    Add pkg/member/models
    Add pkg/member/manager
    Add controller/member
    Remove the old project member API

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-13 21:28:54 +08:00
Daniel Jiang
d4cd2b87bd API for users to new model
This commit moves the legacy apis related to users to new model.
Some funcs under common/dao are left b/c they are used by other module,
which should also be shifted to leverage managers.
We'll handle them separately.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-04-13 11:11:50 +08:00
stonezdj
ac5e908597 Refactor user group to new programming model
Add context to required methods
Add pkg/usergroup/dao
Add pkg/usergroup/manager
Add controller/usergroup/controller

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-09 14:40:48 +08:00
stonezdj
107e468b60 Refactor configure api to new programming model
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-09 08:10:11 +08:00
Qian Deng
c5d12ce8ee
Merge pull request #14542 from ninjadq/add_task_info_in_exporter
Add task info in exporter
2021-04-07 18:17:26 +08:00
DQ
dcb28d8e30 Add test case for exporter
add e2e test to verify exporter and jobservice metrics exist

Signed-off-by: DQ <dengq@vmware.com>
2021-04-06 15:54:33 +08:00
danfengliu
fbf2409c78
Merge pull request #14472 from danfengliu/add-tests-for-manifest-replication
Add test for manifest replication
2021-04-02 17:39:14 +08:00
danfengliu
1801a3cee3 Add manifest and CNAB replication tests
1. Add manifest and CNAB replication tests;
2. Duplicate ORAS and sigularity tests from API test to nightly common
tests;
3. Optimize get dns code in CI;
4. Optimize E2E dockerfile;
5. Sample image size should be cover requirement for large size like 512M.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-04-02 15:39:30 +08:00
Wenkai Yin(尹文开)
28596c3ffb
Refactor registry API (#14528)
* Refactor registry API

Refactor registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Fix bugs of replications

1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-31 15:49:23 +08:00
Wenkai Yin
3d7fd070c7 Refeactor replication policy APIs
Refeactor replication policy APIs

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-17 10:57:31 +08:00
danfengliu
068d1d46ca Fix API python tests log missing issue
After debugging for issue of missing some http message logs, we found out that swagger client configuration will be re-initiated by calling models in swagger client, so in API python tests, defination for models must be in front of swagger client defination.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-03-15 17:23:13 +08:00
Daniel Jiang
0d4992a41e
API for system CVE allowlist to new model (#14412)
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-03-12 10:23:48 +08:00
danfengliu
b181d4df16
Merge pull request #14321 from danfengliu/fix-quotas-test-in-upgrade-pipeline
Fix quotas test issue in upgrade pipeline
2021-03-10 15:59:33 +08:00
danfengliu
a2fc1bcfaf Fix quotas test issue in upgrade pipeline
1. Change way for quotas verification in upgrade pipeline, prepare specific size of image, then it's an known value for verifcation;
2. Add notary key rotate test;
3. For issue brought by docker 20, clean containerd cache is the only effective way, so both dockerd and containerd should be cache cleard and restarted;
4. Upgrade E2E Dockerfile for importing readable file size package, and other issues;
5. Uncomment project level robot account test in nightly.
6. Get DNS from local setting, and set it into docker deamon config file;

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-03-10 06:27:22 +00:00
Wang Yan
85254ccc22
refactor immutable rule (#14371)
Migrate immutable realted APIs to v2 swagger

Signed-off-by: wang yan <wangyan@vmware.com>
2021-03-08 17:10:12 +08:00
He Weiwei
ef0bdf6954 refactor: generate scanner APIs by go-swagger
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-03-03 04:23:36 +00:00
He Weiwei
4b033c266a refactor: generate quota APIs by go-swagger
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-02-25 08:19:55 +00:00
danfengliu
7fb9dbd0fa Upgrade docker and containerd
1. Fix E2E quotas issue, push the same image but with different name;
2. Add checkpoint for robot account test;
3. Upgraded docker and containerd in E2E image;
4. Package base image sample(busybox) into E2E image, so in E2E
container, all local docker images can be cleaned up, once base image is needed for
building image, it can be loaded locally;
5. Adapt OIDC service of supporting LDAP user, and add OIDC group user
test;
6. Restart docker deamon before content trust test, both in API and UI
test;
7. Add retry for keyword "Add A Tag Immutability Rule";
8. Fix tag retention test issue, missing click angle icon, and enhance
checkpoint of dry run and real run;
9. Fix schedule test issue for wrong cron string;
10. Disable quotas verification, it's not stable for script defect;

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-02-24 15:43:11 +08:00