Commit Graph

437 Commits

Author SHA1 Message Date
Wang Yan
01e4aa61f7
fix gc remove manifest issue (#12748)
fixes #12720

The GC job doesn't remove the manifest of scheme1.MediaTypeSignedManifest as it's recognized by GC job.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-13 11:25:06 +08:00
Yiyang Huang
b98dc97fbd feat: enhanced default processor
Signed-off-by: Yiyang Huang <huangyiyang.huangyy@bytedance.com>
2020-08-11 01:31:02 +08:00
chlins
302e4c6659 fix(preheat): add pagination for execution and task list api
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-08-06 16:59:38 +08:00
Wenkai Yin
d6288a43e8 Do some refine for the scheduler
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-05 17:43:18 +08:00
fanjiankong
4570a46823 Fix bug of update preheat instance default.
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-08-04 18:54:03 +08:00
Chlins Zhang
6f6742894c
Merge pull request #12647 from chlins/refactor/preheat-policy-serialize
refactor(preheat): refactor policy schema serialize funcs
2020-08-03 23:01:01 +08:00
chlins
4446302330 refactor(preheat): refactor policy schema serialize funcs
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-08-02 14:43:30 +08:00
He Weiwei
df1bdc1020 refactor(project): add more methods to project controller and manager
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-31 17:55:35 +00:00
stonezdj(Daojun Zhang)
518a1721a7
Merge pull request #12571 from ywk253100/200723_proxy_cache_secret
Limit the permission of secret used by proxy cache service
2020-07-30 14:04:54 +08:00
Daniel Jiang
0a9219dcd6
Merge pull request #12615 from wy65701436/update-logs
revise the blob logs
2020-07-30 09:10:07 +08:00
Steven Zou
507d792655 fix(preheat): fix npe issues
- fix npe issue in create/update policy
- fix issue of missing schedule job id in the preheat policy

Signed-off-by: Steven Zou <szou@vmware.com>

- increase the client timeout
2020-07-30 00:29:26 +08:00
wang yan
20df844d5a revise the blob logs
1, update typo in the update blob status sql, the typo will not impact the sql result.
2, correct blob status in the middleware & GC job log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-29 12:45:30 +08:00
fanjiankong
3653d3cdef Schedule preheat policy.
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-29 00:36:34 +08:00
Steven Zou
d392e27ef9 fix(preheat):enhance preheat job
- add job stop check points in preheat job
- add missing digest property for the preheat request sent to the provider

Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-28 22:19:00 +08:00
Steven Zou
8f6fb7db5b
Merge branch 'master' into fix/p2p_job_log 2020-07-27 22:47:15 +08:00
Steven Zou
1adaf58ab1 fix(preheat):improve preheat job logs
Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-27 22:43:18 +08:00
Wenkai Yin
ced7b73322 Limit the permission of secret used by proxy cache service
Limit the permission of secret used by proxy cache service, fixes #12257

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-27 10:15:00 +08:00
Steven Zou
522bd7a8ee
Merge pull request #12582 from steven-zou/fix/nothing_filtered_out_issue
fix(p2p-preheat):fix issues of triggering preheat
2020-07-26 22:05:22 +08:00
Steven Zou
716da7f3ff fix(p2p-preheat):fix issues of triggering preheat
- fix invalid data type of vulnerability filter param
- add more debug logs
- add more logs in the preheat job
- fix issue of getting empty list when doing querying artifacts

Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-26 13:47:58 +08:00
Steven Zou
cafb2e819c
Merge pull request #12547 from kofj/fix-transaction
Fix: preheat install update transaction.
2020-07-24 16:33:29 +08:00
Daniel Jiang
2041fd337b
Merge pull request #12448 from ninjadq/update_chart_client
Fix Chart V2 Can't parse
2020-07-22 13:31:05 +08:00
He Weiwei
06dc5bf10a
Merge pull request #12548 from heww/gen-and-check-mocks
chore(mocks): add make targets to generate and check mocks
2020-07-22 11:28:13 +08:00
He Weiwei
e3b1ec775f
refactor(quota): align Get and List methods of quota controller (#12434)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-22 11:18:05 +08:00
Wang Yan
eeb8fca255
add debugging env for GC time window (#12528)
* add debugging env for GC time window

For debugging, the tester/users wants to run GC to delete the removed artifact immediately instead of waitting for two hours, add the env(GC_BLOB_TIME_WINDOW) to meet this.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-22 11:09:01 +08:00
He Weiwei
aa55fcfce7 chore(mocks): add make targets to generate and check mocks
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-22 01:59:32 +00:00
Daniel Jiang
4170b267a1
Merge pull request #12496 from reasonerjt/secret-mgr-proxycache
Provide secret manager for proxy cache project
2020-07-22 09:48:24 +08:00
fanjiankong
adbdaaffe6 Fix: preheat install update transaction.
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-22 08:15:18 +08:00
Wenkai Yin(尹文开)
112e38a080
Merge pull request #12541 from stonezdj/20200720_tag_ret_proxy
Add default retention policy
2020-07-21 19:31:12 +08:00
stonezdj
2f0f3c45d2 Add default retention policy
Remove images that are not visited in last 7 days
Add a build method in policy models.go

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-07-21 16:49:35 +08:00
DQ
f957acf763 Fix Chart V2 Can't parse
Use helm v3 libary to replace v2

Signed-off-by: DQ <dengq@vmware.com>
2020-07-21 10:07:08 +08:00
He Weiwei
6db1a1cb91 perf(db): skip tx for get blob, patch/put blob upload apis
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-20 16:42:16 +00:00
Daniel Jiang
14203169bf Add GC mechanism to secret manager
When Generate is called and the size is larger than cap, GC will be
triggered.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-07-20 15:53:36 +08:00
Steven Zou
46d7434d0b
Merge pull request #12473 from ywk253100/200706_scheduler
Refactor the scheduler with the task manager mechanism
2020-07-20 15:53:14 +08:00
Wenkai Yin
4dc4b6728c Refactor the scheduler with the task manager mechanism
Refactor the scheduler with the task manager mechanism, this will reduce the duplicate code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-20 14:03:15 +08:00
Wang Yan
24ed52112e
fix blob deleting status issue (#12481)
1, The update blob status method should udpate the blob version of the blob object as well, otherwise the GC job cannot handle the blob status transform(none - delete - deleting - deletefailed)
as the method is using version equals as the query condition.
2, For the deleting blob which marked for more than 2 hours, it should be set to delete failed in head blob & put manifest request

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-20 11:44:29 +08:00
Daniel Jiang
840aa86dfa Provide secret manager for proxy cache project
This commit provides the secret manager for proxy cache.
The secret is used for pushing blobs to local when it's proxied from
remote registry.
Each secret can be used only once and has a relatively short expiration
time.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-07-16 19:12:52 +08:00
Wang Yan
d73265d10d
revise gc job to align non blocking gc (#12439)
two phases:
1, mark, select the gc candidates bases on the DB and mark them as status delete.
2, sweep, select the candidate and mark it as status deleting and remove it from backend and database.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-16 14:35:54 +08:00
He Weiwei
a22d803a95 refactor(quota): move pkg/types to pkg/quota/types
Closes #9664

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-14 14:28:53 +00:00
chlins
38d14dff30 fix(preheat): validate instance/policy name, set unique filed and policy
manager adds parsePolicy

Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-10 10:29:47 +08:00
Steven Zou
b56a49efe2
Merge pull request #12411 from kofj/instance_by_name
Get instance by name.
2020-07-07 23:20:01 +08:00
fanjiankong
3c1c799f0d Get instance by name.
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-07 22:06:38 +08:00
Wang Yan
c3baeac5ae
add time windows support in artifact trash (#12400)
support with time window to filter and delete item in artifact trash

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-07 20:13:08 +08:00
chlins
37a00912b7 feat: add p2p preheat swagger yaml and implement preheat api policy handler
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-06 21:05:22 +08:00
Steven Zou
1ee3f00709 fix(p2p):remove the provider manager related to
- use real provider instance manager
- move mock insatnce manager to testing/pkg
- modify kraken deriver implementation to remove digest fetcher
- update related UT cases

Signed-off-by: Steven Zou <szou@vmware.com>
2020-07-03 17:30:11 +08:00
Steven Zou
c7a050b629
Merge pull request #12389 from chlins/fix/p2p-preheat-policy-count-total
fix: add count method of policy manager to replace list method return…
2020-07-03 13:50:34 +08:00
Steven Zou
f3fcb96570
Merge pull request #12335 from kofj/p2p_preheat_api
feat(preheat):add preheat api, controller and manager
2020-07-03 13:47:04 +08:00
chlins
ace21240a4 fix: add count method of policy manager to replace list method return wrong counts
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-03 11:59:32 +08:00
Wang Yan
e8784de5fe
support list blobs by update time (#12385)
Add support list blob with update time.
As introduces the time window in GC, it wants to list the blobs less than specific time.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-03 11:50:53 +08:00
fanjiankong
a0c2d0ac9e feat(preheat):add preheat api, controller and manager
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-03 11:25:42 +08:00
Steven Zou
264bd02892
Merge pull request #12378 from steven-zou/feat/preheat_policy_enforcement
feat(preheat):implement policy enforcer
2020-07-02 14:30:25 +08:00