Commit Graph

329 Commits

Author SHA1 Message Date
stonezdj
5618f6d211 The DB config manager is not init on jobservice to call config.WithNotary
Make func defaultMgr() public
  Set DefaultCfgManager to RestCfgManager and Load it in the jobservice main
  config.WithNotary call DefaultMgr(), it wil get the RestCfgManager
  Fixes #16418

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-03-03 14:42:56 +08:00
chlins
3d7a5064e9 fix: trim linebreaks for log parameters
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-02-16 10:52:58 +08:00
chlins
b37a987073 fix: fix codeql alerts
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-02-15 17:25:02 +08:00
Shengwen Yu
d2ae0165c9 feat: add failure-tolerance for gc
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-01-10 11:34:10 +08:00
chlins
adf866e629 fix: resolve the codeql alerts
Signed-off-by: chlins <chenyuzh@vmware.com>
2021-12-15 13:49:42 +08:00
Wang Yan
6014646bcb
fixes gc dry run issue (#15804)
fixes #15332, for the dry run mode, gc job should not remove the untagged candidates.
To fix it, use the simulate untagged artifact deletion for dry-run.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-10-19 07:42:54 +08:00
yxxhero
63b8e6a9ce
add unittest for jobservice/common/utils package and fix a issue for IsValidURL (#15539)
add unittest for common utils package and fix a issue

Signed-off-by: yxxhero <aiopsclub@163.com>
2021-09-24 19:07:55 +08:00
Qian Deng
354a2bd80d Enhance the trace related code
* Move request id to requestid middleware
* fix span pass to child ctx on orm
* fix typos
* remove unused code
* add operation name to Transaction

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-22 04:14:43 +00:00
Qian Deng
31707dbf25 Using harbor lib config
using config manager to get env configs

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Qian Deng
bad913cf6d Refactor trace code
* use lib trace helper function
* add gracefull shutdown
* Add commens for new added exposed function
* Add licence on top of new created files
* Update trace library
* Update configs
* Add attribute and namespance in config

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Qian Deng
6fec5b2873 Add trace to jobservice
* Add trace init in main
* Add env template
* Add trace for router

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Qian Deng
879eecc926 Refactor and Add trace to http Transport
* Refactor common http GetTransport function signature
* Remove redendent GetHTTPTransport and similar functions
* Update Authorized function signature to meet new HTTPTransport
* Add trace for default Transport

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Ziming Zhang
98cef43ead limit replication bandwidth
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-09-14 11:23:28 +08:00
Steven Zou
6b5cd3a7a5 feat(schedule):sync schedules in db to js datastore
- add a sync worker to sync db schedules when js starting. add missing ones and clear dirty ones.
- update task model to contain status revision info
- update job lifecycle tracker save() method
- update job ACK model
- add UT cases
- update malformat comments

fix #15323

Signed-off-by: Steven Zou <szou@vmware.com>
2021-09-07 08:33:01 +08:00
Wang Yan
e0c4ebd84d
fix gc delete manifest log issue (#15495)
It needs to use the logger to print error log when timeout, otherwise, it will be dropped.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-08-30 13:40:33 +08:00
He Weiwei
d482a0c323
fix: avoid panic in the RetryUntil (#15501)
1. Use jpillora/backoff to get the backoff to avoid the panic in RetryUntil.
2. Return with last err when retry timeout.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-08-27 17:28:33 +08:00
Wang Yan
29ccdff766
add retry for blob & manifest deletion in GC (#15235)
Fixes #15197

To handle the blob & manifest deletion error, add 1 minute retry.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-07-01 18:49:40 +08:00
Wang Yan
1abe978e87 refactor job log
Move job service logger to new program model

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-06-29 13:37:45 +08:00
Wang Yan
8a0cd99473
fixes 13976 (#15047)
Fixes #13976 for the quota exceed case, gc will print the untagged blobs for dry-run

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-06-04 15:56:53 +08:00
Wang Yan
42a9d0d905
remove common project code (#14939)
move project model from common to pkg

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-25 11:01:19 +08:00
Wang Yan
a08709b50d
deprecate read only GC job (#14773)
After refactor GC, the way to trigger read only GC job has been deprecated, remove the code.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-04-29 10:34:13 +08:00
Archer1A
264e2e45f2
fix(jobservice): value out of range (#14634)
value out of range when job count bigger than 256

Signed-off-by: vic <985695123@qq.com>
2021-04-14 14:28:36 +08:00
Steven Zou
39256193ca fix(js):job log not found issue
- return 404 code when the job log is not found

Signed-off-by: Steven Zou <szou@vmware.com>
2021-04-13 12:11:31 +08:00
Steven Zou
3646b263da
Merge pull request #14591 from steven-zou/fix/hook_resent_issue
fix(js): webhook infinitely resending issue
2021-04-09 17:08:08 +08:00
stonezdj
107e468b60 Refactor configure api to new programming model
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-09 08:10:11 +08:00
Steven Zou
bb7f706d78 fix(js): webhook infinitely resending issue
- update the resending logic in the hook agent
- use backoff lib to generate backoff duration
- remove retry cache queue
- leverage reaper to restore the sending if direct retries are all failed
- remove useless funcs/codes

fix #14545

Signed-off-by: Steven Zou <szou@vmware.com>
2021-04-08 19:59:16 +08:00
Wenkai Yin(尹文开)
28596c3ffb
Refactor registry API (#14528)
* Refactor registry API

Refactor registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Fix bugs of replications

1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-31 15:49:23 +08:00
DQ
fbe9cd88f8 Enabled Prometheus for Jobservice
* Add prom server on jobservice
* Enabeld configs in templates
* Enabeld jobservice metrics in nginx

Signed-off-by: DQ <dengq@vmware.com>
2021-03-30 08:52:59 +00:00
Wenkai Yin
506d1ad465 Introduce "sort" in query to provide a general solution for sorting
Introduce "sort" in query to provide a general solution for sorting

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-11 08:25:49 +08:00
Josh Soref
dfe360040b Spelling
* addition
* attribute
* auditing
* availability
* available
* bandwidth
* browser
* business
* cadence
* chartmuseum
* client
* column
* content
* demonstrate
* described
* endpoints
* facilitate
* github
* harbor
* information
* instance
* manual
* meaningful
* operation
* overridden
* password
* possible
* project
* refactor
* replication
* requires
* running
* scanned
* settings
* signup
* those
* unsigned
* vulnerability

--
Also removes trailing space from a filename

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2021-02-19 11:59:15 +08:00
Wang Yan
21d35f9702
update blob list query (#14195)
* update blob list query

Deprecate blob list parameters, and use the query for instead.

Signed-off-by: wang yan <wangyan@vmware.com>

* update per review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-02-09 17:08:26 +08:00
Steven Zou
d61e560890 fix(jobservice):wrong depth of job logging
- use separate std logger for job, not shared with jobservice std logger
- merge and remove useless functions

Signed-off-by: Steven Zou <szou@vmware.com>

fix #14079
2021-02-07 18:07:02 +08:00
Steven Zou
4737a394ee
Merge pull request #13947 from shaobo322/fix_improper_spell
fix improper spelling
2021-02-01 16:50:02 +08:00
Wenkai Yin(尹文开)
63831dfd08
Merge pull request #14027 from ywk253100/210116_status
Don't ignore the NotFoundErr when handling the status hook of tasks to avoid the status out of sync
2021-01-20 17:07:18 +08:00
Wenkai Yin
e55c7d05ff Don't ignore the NotFoundErr when handling the status hook of tasks to avoid the status out of sync
Don't ignore the NotFoundErr when handling the status hook of tasks to avoid the status out of sync
Fixes #14016

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-20 14:53:50 +08:00
Wenkai Yin
7c072e17a6 Fix the legacy scheduled job issue for GC/scan all
Fix the legacy scheduled job issue for GC/scan all

Fixes #13968

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-15 22:02:36 +08:00
shaobo
fba0ff4d83 fix improper spelling
Signed-off-by: shaobo <shaobo@caicloud.io>
2021-01-08 16:25:51 +08:00
shaobo
434758b71e reform job status compare method
Signed-off-by: shaobo <shaobo@caicloud.io>
2021-01-04 21:34:53 +08:00
He Weiwei
8fa03e3739
refactor: remove code of scan all job (#13821)
Remove code of scan all job as it's implemented by execution now.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-22 11:39:18 +08:00
Wenkai Yin(尹文开)
a4f21b1d85
Merge pull request #13694 from kofj/ghcr-provider
GHCR Provider
2020-12-11 09:41:10 +08:00
Steven Zou
4acb708938
Merge pull request #13705 from steven-zou/fix/js_api
fix(api):update stop job api
2020-12-09 13:41:58 +08:00
fanjiankong
2898117411 GHCR Provider
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-12-09 09:54:04 +08:00
Steven Zou
85783f3c0c fix(api):update stop job api
- update StopJob() of basic worker
- update UnSchedule() of basic scheduler
- update the policy store to get more data

fix #13599 , fix #13597

Signed-off-by: Steven Zou <szou@vmware.com>
2020-12-08 23:15:05 +08:00
疯魔慕薇
9f8a743da9 Tecent TCR Provider
1. Docker image registry.
2. Helm chart registry.

Signed-off-by: 疯魔慕薇 <kfanjian@gmail.com>
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-12-08 15:44:18 +08:00
peimingming
186109162b Support replicate from artifact hub
Signed-off-by: peimingming <peimingming@corp.netease.com>
2020-11-26 21:41:24 +08:00
Greg
d1ee94bbc9
DTR replication support (#9512)
Adding DTR replication support

Signed-off-by: Greg Sidelinger <gate@ilive4code.net>
2020-10-29 18:54:44 +08:00
chlins
685d9277c4 fix(webhook): fix the verify cert logic of webhook endpoint
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-09-10 21:01:25 +08:00
Wang Yan
262f22f5ef
fix gc log issue (#12943)
1, Do not log redis url, just log the user input from UI.
2, Format the artifact trash items.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-09-02 17:11:05 +08:00
He Weiwei
f309896f2f refactor(api): generate project apis by go-swagger
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-15 16:10:57 +00:00
He Weiwei
e3fd83aad1 fix(blob): make blob size in redis expired in 24 hours
Closes #11873

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-12 11:45:50 +00:00