Commit Graph

226 Commits

Author SHA1 Message Date
Daniel Jiang
2064a1cd6d Switch to basic authentication for registry
1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth.  #10604 is opened
to track the follow up work.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-01-31 21:46:47 +09:00
Steven Zou
e899d659f3 fix[jobservice]:job status is hung after restart
- improve the status hook sending/resending approach
- improve the status compare and set approach
- simplify the relevant flow
- add reaper to fix the out of sync jobs
- fix #10244 , fix #9963

Signed-off-by: Steven Zou <szou@vmware.com>
2020-01-09 17:10:35 +08:00
Steven Zou
ebc5d2482b do improvements to the scan all job
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job

fix #9705
fix #9722
fix #9670

Signed-off-by: Steven Zou <szou@vmware.com>
2019-11-05 15:12:07 +08:00
Steven Zou
35d074e849 fix bug found in the pluggable scanner
- fix #9632
- fix #9633

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-29 15:32:51 +08:00
Wenkai Yin(尹文开)
28ee5f7d43
Merge pull request #9302 from liuxu623/fix-fatal-error
fix fatal error
2019-10-25 13:16:54 +08:00
He Weiwei
cc86664730 fix(gc,quota): clean blob size cache in redis when gc
Closes #9151

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2019-10-22 14:20:04 +00:00
Steven Zou
0f16913635 rebase: resolve the code confilcts with master
Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-17 17:42:41 +08:00
Steven Zou
f18afc0a3f do changes to let the vul policy check compatiable with new framework
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-16 23:15:26 +08:00
Ziming
3725667448
Merge branch 'master' into feat/gitlab 2019-10-16 14:34:21 +08:00
liuxu
79322f2df2 fix fatal error
Signed-off-by: liuxu <liuxu623@gmail.com>
2019-10-14 21:41:29 +08:00
Wenkai Yin(尹文开)
7d0505593f
Merge pull request #8556 from chlins/feat/image-replication-adapter-for-quay.io
Feat/image replication adapter for quay.io
2019-10-14 09:16:45 +08:00
Steven Zou
58afd8e14b [stage3] support pluggable scanner
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces

Signed-off-by: Steven Zou <szou@vmware.com>

- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases

- update robot account making content
- hidden credential in the job log

Commnet scan related API test cases which will be re-activate later
fix #8985

fix the issues found by codacy

Signed-off-by: Steven Zou <szou@vmware.com>
2019-10-11 12:53:02 +08:00
chlins
4ab3b864ae feat: add image replication adapter for quay.io
Signed-off-by: chlins <chlins.zhang@gmail.com>
2019-10-11 10:00:07 +08:00
Wenkai Yin(尹文开)
0a85acac9a
Merge pull request #9329 from chlins/feat/jfrog-docker-registry
feat(replication): support for jfrog artifactory docker image replica…
2019-10-10 09:41:41 +08:00
chlins
0e0c42d3c0 feat(replication): support for jfrog artifactory docker image replication
Signed-off-by: chlins <chlins.zhang@gmail.com>
2019-10-09 21:06:36 +08:00
Qian Deng
d2d0affcae
Merge pull request #9245 from ninjadq/check_logid
Add checking for logid
2019-09-26 10:26:49 +08:00
Wenkai Yin(尹文开)
ac17afac1f
Merge pull request #9243 from ninjadq/change_file_permission_of_joblog
change the permission of job log file
2019-09-25 17:45:21 +08:00
DQ
e078b1c209 Add checking for logid
the logid content need check before use

Signed-off-by: DQ <dengq@vmware.com>
2019-09-25 13:55:32 +08:00
DQ
4333c3ad5c change the permission of job log file
Log file permission from 644 to 600

Signed-off-by: DQ <dengq@vmware.com>
2019-09-24 20:04:17 +08:00
Wenkai Yin
c2e529576a Update the error log format
Update the erro log format in jobservice

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-09-23 15:40:04 +08:00
wang yan
42a5db83b2 refactor selector of retention
extract select from pkg/retention, move it to pkg/artselector to make it usable by immutable tag

Signed-off-by: wang yan <wangyan@vmware.com>
2019-09-18 16:38:41 +08:00
lxShaDoWxl
6937731744 feat(Registries): added gitlab adapter
Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>
2019-09-09 21:21:21 +06:00
Wang Yan
39f78ae768
Merge pull request #7872 from cd1989/config-redis-pool-idletimeout
Config idle timeout for redis pool to avoid jobservice restarting
2019-08-27 14:46:01 +08:00
wang yan
e3155e00d6 fix #8815 :add remove untagged blob record in table project_blob
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-26 15:57:19 +08:00
Steven Zou
718e336302 reduce the expire time of job stats in redis DB
Signed-off-by: Steven Zou <szou@vmware.com>
2019-08-22 18:26:09 +08:00
Wenkai Yin(尹文开)
e3d8fa845d
Merge pull request #8767 from steven-zou/fix/issue-#8525
fix #8525: periodic job retry issue: job stats are not found
2019-08-22 11:06:13 +08:00
wang yan
e91ded65cb fix quota size usage in gc job, issue #https://github.com/goharbor/harbor/issues/8699
Signed-off-by: wang yan <wangyan@vmware.com>
2019-08-21 16:51:31 +08:00
Steven Zou
926ac44a67 fix #8525: periodic job retry issue: job stats is not found
Signed-off-by: Steven Zou <szou@vmware.com>
2019-08-21 16:30:47 +08:00
cd1989
db9b52d827 Config idle timeout for redis pool
Signed-off-by: cd1989 <chende@caicloud.io>
2019-08-20 17:23:32 +08:00
Steven Zou
ace07b0c0a fix #8745: job retry issue
Signed-off-by: Steven Zou <szou@vmware.com>
2019-08-20 16:39:44 +08:00
王添
94d4f9c6b6 add webhook job
Signed-off-by: 王添 <wangtian@corp.netease.com>
2019-08-07 20:56:31 +08:00
Wenkai Yin
8325f0769f Ignore the arguments when logging the job detail
As the arguments of a job may contain sensitive information, ignore them when logging the job detail. Fixes #8404

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-08-06 15:02:14 +08:00
Wenkai Yin(尹文开)
27245021d3
Merge pull request #8500 from steven-zou/fix/pool_run_out
fix bug #8499: redis connection pool is run out
2019-08-02 17:47:26 +08:00
Wenkai Yin
1aa5e631d7 Remove the retention job for deleting repository
Remove the retention job for deleting repository

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-08-02 10:27:19 +08:00
Steven Zou
3a68bd7aa2 fix bug #8499: redis connection pool is run out
Signed-off-by: Steven Zou <szou@vmware.com>
2019-07-31 15:45:03 +08:00
Wenkai Yin
6464dcbd51 Register the replication adapter for Helm Hub
Register the replication adapter for Helm Hub

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-07-30 16:28:06 +08:00
Steven Zou
1e7f51615f fix conflicts in runtime/bootstrap.go 2019-07-30 09:52:22 +08:00
Steven Zou
3ca5116a53 add retention job for deleting repository
Signed-off-by: Steven Zou <szou@vmware.com>
2019-07-30 09:45:19 +08:00
Ziming
a5c31d7160
Merge pull request #8456 from ywk253100/190729_scheduler
Register the scheduler job and fix bugs
2019-07-30 08:57:10 +08:00
Wenkai Yin
94c1cf8ac9 Register the scheduler job and fix bugs
1. Register the scheduler job
2. Fix retention bugs

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-07-30 07:37:47 +08:00
慕薇疯魔
0d71eb363a Aliyun ACR adapter
1. aliyun ACR registry adapter.
2. Unit test case.
3. Update go modules and vendor.

Signed-off-by: 疯魔慕薇 <kfanjian@gmail.com>
2019-07-29 18:24:03 +08:00
Steven Zou
c44747fd3c merge code from master and fix conflicts
Signed-off-by: Steven Zou <szou@vmware.com>
2019-07-24 17:27:37 +08:00
Wenkai Yin
1adf5c976e Create retention client with the configurations got from jobservice config
As the retention client is only used in job which running in jobservice, the configurations that needed to init the client should be read from jobservice

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-07-24 13:10:10 +08:00
Wenkai Yin
7362fae7cc Implement a common scheduler
Implement a common scheduler that can be used globally

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-07-23 17:20:31 +08:00
Steven Zou
ccad9c0a29 support stop in the retention job and register it
Signed-off-by: Steven Zou <szou@vmware.com>
2019-07-23 11:05:22 +08:00
Bernardo Pastorelli
791394a2b1 Make jobservice use the proxy env variables (including no_proxy)
Signed-off-by: Bernardo Pastorelli <berpast@hotmail.com>
2019-07-19 16:32:17 +02:00
Wenkai Yin
d6c6231e08 Implement the retention client
Implement the retention client

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-07-17 14:04:29 +08:00
Steven Zou
63e2ce7606
Merge pull request #8209 from steven-zou/fix/enqueuer_schedule_slot
use separate key for the last periodic enqueue data
2019-07-03 21:48:21 +08:00
Steven Zou
b88159e747 use separate key for the last periodic enqueue data
Signed-off-by: Steven Zou <szou@vmware.com>
2019-07-03 17:08:23 +08:00
Steven Zou
1002e3ce17 fix failure ut case of job service
Signed-off-by: Steven Zou <szou@vmware.com>
2019-07-03 15:10:41 +08:00