Commit Graph

245 Commits

Author SHA1 Message Date
Wang Yan
4594d58ba8
add clean untagged blobs in gc job (#11248)
Fixes #11190, delete all of non-referenced blobs of each project in GC job, thun the quota
can be released.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-01 18:01:46 +08:00
Wang Yan
f6c0608e22
fix GC jobs upgrade issue (#11365)
Fixes #11313
Fixes #11275

1, Add more details log in GC job
2, Add type assertion for the upgrading case, the delete_untagged parameter is introduced from v2.0
3, Add UT

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-01 11:53:41 +08:00
Wang Yan
d6261d9456
Does not throw err in the notification job (#11363)
Fixes #11280, no error return but just log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-31 18:12:12 +08:00
Steven Zou
3d6c65f53b
fix[UT]:improve the UT cases of enqueuer (#11358)
- change the test cron spec
- use cretaed timer for timeout

Signed-off-by: Steven Zou <szou@vmware.com>
2020-03-31 00:09:20 +08:00
Steven Zou
f2beed577f
fix[logger]:update log ID validation logic (#11351)
Signed-off-by: Steven Zou <szou@vmware.com>
2020-03-31 00:08:55 +08:00
Wenkai Yin
9a92b9e725 Fix bugs of replication
1. Bump up the version of API used in replicatoin scheduler job
2. Check the error message to determine whether the job exists or not in jobservice when unschedule a job

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-26 23:12:02 +08:00
Wenkai Yin(尹文开)
8984979bd2
Relocate/rename some packages (#11183)
Fixes #11016
1. src/pkg/q->src/internal/q
2. src/internal->src/lib (internal is a reserved package name of golang)
3. src/api->src/controller

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-24 20:45:45 +08:00
Steven Zou
e8d5ba9491
Merge pull request #11004 from steven-zou/fix/update_js_ut_case
fix[js_ut]:update stop job case of js
2020-03-19 15:49:08 +08:00
DQ
b93092e012 Add tls for trivy
Add trivy tls cert files
Add tivey tls env and config
enhance gencert

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
c954969bcd Add mTLS configs
mTLS only enabled in jobservice and registryctl

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
115185894f Merge internal Transport and Secure Transport
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00
DQ
dcc6950af7 Feat: auto install ca in registry
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00
DQ
e6bb3b0977 Feat: enable tls related thing to jobservice
Add tls related code in jobservice

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00
Steven Zou
b2434945ff fix[js_ut]:update stop job case of js
remove the status checking loop

Signed-off-by: Steven Zou <szou@vmware.com>
2020-03-10 15:54:10 +08:00
Wang Yan
e86d3a728c
Merge pull request #10899 from steven-zou/fix/failure_js_ut_cases
fix[js]:fix ut case faulure
2020-03-07 19:10:02 +08:00
wang yan
2b0b7576b2 Fix gc issue on clean the artifact trash
1, enable dao test for artifact trash
2, set default flush trash table to false
3, hanlder empty parameter in API call
4, add registry auth info into jobservice container

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-06 03:11:31 +08:00
Steven Zou
bd0e401cae fix[js]:fix ut case faulure
- refactor default context creation to avoid data race
- refactor the timer interval in c_worker UT cases to avoid receieving signals at the same time

Signed-off-by: Steven Zou <szou@vmware.com>
2020-03-03 11:47:41 +08:00
Wang Yan
948d45604c Revise the GC job flow,
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-02-24 18:29:55 +08:00
Wenkai Yin
528f598268 Reimplement the registry client
This commit reimplements the registry client under directory src/pkg/registry and removes the useless code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-24 14:36:26 +08:00
Daniel Jiang
2064a1cd6d Switch to basic authentication for registry
1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth.  #10604 is opened
to track the follow up work.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-01-31 21:46:47 +09:00
Steven Zou
e899d659f3 fix[jobservice]:job status is hung after restart
- improve the status hook sending/resending approach
- improve the status compare and set approach
- simplify the relevant flow
- add reaper to fix the out of sync jobs
- fix #10244 , fix #9963

Signed-off-by: Steven Zou <szou@vmware.com>
2020-01-09 17:10:35 +08:00
Steven Zou
ebc5d2482b do improvements to the scan all job
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job

fix #9705
fix #9722
fix #9670

Signed-off-by: Steven Zou <szou@vmware.com>
2019-11-05 15:12:07 +08:00
Steven Zou
35d074e849 fix bug found in the pluggable scanner
- fix #9632
- fix #9633

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-29 15:32:51 +08:00
Wenkai Yin(尹文开)
28ee5f7d43
Merge pull request #9302 from liuxu623/fix-fatal-error
fix fatal error
2019-10-25 13:16:54 +08:00
He Weiwei
cc86664730 fix(gc,quota): clean blob size cache in redis when gc
Closes #9151

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-22 14:20:04 +00:00
Steven Zou
0f16913635 rebase: resolve the code confilcts with master
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-17 17:42:41 +08:00
Steven Zou
f18afc0a3f do changes to let the vul policy check compatiable with new framework
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-16 23:15:26 +08:00
Ziming
3725667448
Merge branch 'master' into feat/gitlab 2019-10-16 14:34:21 +08:00
liuxu
79322f2df2 fix fatal error
Signed-off-by: liuxu <liuxu623@gmail.com>
2019-10-14 21:41:29 +08:00
Wenkai Yin(尹文开)
7d0505593f
Merge pull request #8556 from chlins/feat/image-replication-adapter-for-quay.io
Feat/image replication adapter for quay.io
2019-10-14 09:16:45 +08:00
Steven Zou
58afd8e14b [stage3] support pluggable scanner
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces

Signed-off-by: Steven Zou <szou@vmware.com>

- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases

- update robot account making content
- hidden credential in the job log

Commnet scan related API test cases which will be re-activate later
fix #8985

fix the issues found by codacy

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-11 12:53:02 +08:00
chlins
4ab3b864ae feat: add image replication adapter for quay.io
Signed-off-by: chlins <chlins.zhang@gmail.com>
2019-10-11 10:00:07 +08:00
Wenkai Yin(尹文开)
0a85acac9a
Merge pull request #9329 from chlins/feat/jfrog-docker-registry
feat(replication): support for jfrog artifactory docker image replica…
2019-10-10 09:41:41 +08:00
chlins
0e0c42d3c0 feat(replication): support for jfrog artifactory docker image replication
Signed-off-by: chlins <chlins.zhang@gmail.com>
2019-10-09 21:06:36 +08:00
Qian Deng
d2d0affcae
Merge pull request #9245 from ninjadq/check_logid
Add checking for logid
2019-09-26 10:26:49 +08:00
Wenkai Yin(尹文开)
ac17afac1f
Merge pull request #9243 from ninjadq/change_file_permission_of_joblog
change the permission of job log file
2019-09-25 17:45:21 +08:00
DQ
e078b1c209 Add checking for logid
the logid content need check before use

Signed-off-by: DQ <dengq@vmware.com>
2019-09-25 13:55:32 +08:00
DQ
4333c3ad5c change the permission of job log file
Log file permission from 644 to 600

Signed-off-by: DQ <dengq@vmware.com>
2019-09-24 20:04:17 +08:00
Wenkai Yin
c2e529576a Update the error log format
Update the erro log format in jobservice

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-09-23 15:40:04 +08:00
wang yan
42a5db83b2 refactor selector of retention
extract select from pkg/retention, move it to pkg/artselector to make it usable by immutable tag

Signed-off-by: wang yan <wangyan@vmware.com>
2019-09-18 16:38:41 +08:00
lxShaDoWxl
6937731744 feat(Registries): added gitlab adapter
Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>
2019-09-09 21:21:21 +06:00
Wang Yan
39f78ae768
Merge pull request #7872 from cd1989/config-redis-pool-idletimeout
Config idle timeout for redis pool to avoid jobservice restarting
2019-08-27 14:46:01 +08:00
wang yan
e3155e00d6 fix #8815 :add remove untagged blob record in table project_blob
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-26 15:57:19 +08:00
Steven Zou
718e336302 reduce the expire time of job stats in redis DB
Signed-off-by: Steven Zou <szou@vmware.com>
2019-08-22 18:26:09 +08:00
Wenkai Yin(尹文开)
e3d8fa845d
Merge pull request #8767 from steven-zou/fix/issue-#8525
fix #8525: periodic job retry issue: job stats are not found
2019-08-22 11:06:13 +08:00
wang yan
e91ded65cb fix quota size usage in gc job, issue #https://github.com/goharbor/harbor/issues/8699
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-21 16:51:31 +08:00
Steven Zou
926ac44a67 fix #8525: periodic job retry issue: job stats is not found
Signed-off-by: Steven Zou <szou@vmware.com>
2019-08-21 16:30:47 +08:00
cd1989
db9b52d827 Config idle timeout for redis pool
Signed-off-by: cd1989 <chende@caicloud.io>
2019-08-20 17:23:32 +08:00
Steven Zou
ace07b0c0a fix #8745: job retry issue
Signed-off-by: Steven Zou <szou@vmware.com>
2019-08-20 16:39:44 +08:00
王添
94d4f9c6b6 add webhook job
Signed-off-by: 王添 <wangtian@corp.netease.com>
2019-08-07 20:56:31 +08:00