Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-09-29 22:07:32 +02:00
Code Issues Projects Releases Wiki Activity
12,062 Commits 44 Branches 311 Tags 339 MiB
67c03ddc4f
Commit Graph

406 Commits

Author SHA1 Message Date
Steven Zou
adb305e4e1
feat(job):enable priority option (#11608) 
- priority option is supported when doing job registration
- the priority is defined by a unique priority sampler
- the default priority is 1000 (max is 10000)

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-04-14 14:54:44 +08:00
DQ
b3db293091 TLS update min version and cipher suits 
min version set to tls 1.2
suit only use ecdhe and strenth above 256

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-13 18:13:30 +08:00
Steven Zou
3ad5b2ba06
fix(job currency):introduce max corrency of job (#11589) 
- update Job interface to introdcue MaxCurrency method for declaring the max currency of the specified job
- change the downstream jobs to implement the new interface method
  - GC and sample jobs are set to 1
  - other jobs are set to 0 that means unlimited
- add max currency optiot when doing job registration
- resolve issue #11586
  - probably resolve issue #11281
  - resolve issue #11570

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-04-13 18:07:54 +08:00
He Weiwei
0b87eaf039
Merge pull request #11505 from heww/revert-registry-authorization-type-support 
feat(scan): revert bearer token support for scanner
 2020-04-13 11:19:02 +08:00
Wang Yan
740b1e46b4
fix artifact trash filter issue (#11575) 
fixes #11533

GC jobs will use the filter results to call registry API to delete manifest.

In the current imple, the filter function in some case does not return the deleted artifact as it's using digest as the filter condition.

Like: If one artifact is deleted, but there is another project/repo has a image with same digest with the deleted one, filter func will
not mark the deleted artifact as candidate. It results in, GC job does not call API to remove the manifest.

To fix it, update the filter to use both digest and repository name to filter candidate.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-12 03:11:49 +08:00
He Weiwei
4623cec1e5 feat(scan): revert bearer token support for scanner 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-11 08:45:29 +00:00
Steven Zou
59f14dff98 fix(js UT cases):fix the cron spec bug 
- eliminate the time of cron spec  overflow issue in the UT case

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-04-08 15:59:05 +08:00
Steven Zou
0fc7629865 fix[jobservice]:enqueue job with UTC 
- schedule the periodical jobs following the UTC timezone
  - e.g: 5 10 10 * * * means run jobs at UTC time 10:10:05 everyday
- fix issue #11466

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-04-07 17:10:19 +08:00
wang yan
a11a70d941 move logger from common to lib 
The logger is the fundamental library, so move it into lib folder
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-02 14:09:03 +08:00
Steven Zou
1f6301267c
Merge pull request #11369 from steven-zou/fix/issue_#11361 
fix[lua_scripts]:add default values for tonumber
 2020-04-01 19:01:02 +08:00
Wang Yan
4594d58ba8
add clean untagged blobs in gc job (#11248) 
Fixes #11190, delete all of non-referenced blobs of each project in GC job, thun the quota
can be released.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-01 18:01:46 +08:00
Wang Yan
f6c0608e22
fix GC jobs upgrade issue (#11365) 
Fixes #11313
Fixes #11275

1, Add more details log in GC job
2, Add type assertion for the upgrading case, the delete_untagged parameter is introduced from v2.0
3, Add UT

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-01 11:53:41 +08:00
Steven Zou
b5fceae734 fix[lua_scripts]:add default values for tonumber 
- add default values for the integer vars converted by tonumber()

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-03-31 18:20:52 +08:00
Wang Yan
d6261d9456
Does not throw err in the notification job (#11363) 
Fixes #11280, no error return but just log.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-03-31 18:12:12 +08:00
Steven Zou
3d6c65f53b
fix[UT]:improve the UT cases of enqueuer (#11358) 
- change the test cron spec
- use cretaed timer for timeout

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-03-31 00:09:20 +08:00
Steven Zou
f2beed577f
fix[logger]:update log ID validation logic (#11351) 
Signed-off-by: Steven Zou <szou@vmware.com>
 2020-03-31 00:08:55 +08:00
Wenkai Yin
9a92b9e725 Fix bugs of replication 
1. Bump up the version of API used in replicatoin scheduler job
2. Check the error message to determine whether the job exists or not in jobservice when unschedule a job

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-03-26 23:12:02 +08:00
Wenkai Yin(尹文开)
8984979bd2
Relocate/rename some packages (#11183) 
Fixes #11016
1. src/pkg/q->src/internal/q
2. src/internal->src/lib (internal is a reserved package name of golang)
3. src/api->src/controller

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-03-24 20:45:45 +08:00
Steven Zou
e8d5ba9491
Merge pull request #11004 from steven-zou/fix/update_js_ut_case 
fix[js_ut]:update stop job case of js
 2020-03-19 15:49:08 +08:00
DQ
b93092e012 Add tls for trivy 
Add trivy tls cert files
Add tivey tls env and config
enhance gencert

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ
c954969bcd Add mTLS configs 
mTLS only enabled in jobservice and registryctl

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ
115185894f Merge internal Transport and Secure Transport 
Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
DQ
dcc6950af7 Feat: auto install ca in registry 
Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
DQ
e6bb3b0977 Feat: enable tls related thing to jobservice 
Add tls related code in jobservice

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
Steven Zou
b2434945ff fix[js_ut]:update stop job case of js 
remove the status checking loop

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-03-10 15:54:10 +08:00
Wang Yan
e86d3a728c
Merge pull request #10899 from steven-zou/fix/failure_js_ut_cases 
fix[js]:fix ut case faulure
 2020-03-07 19:10:02 +08:00
wang yan
2b0b7576b2 Fix gc issue on clean the artifact trash 
1, enable dao test for artifact trash
2, set default flush trash table to false
3, hanlder empty parameter in API call
4, add registry auth info into jobservice container

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-03-06 03:11:31 +08:00
Steven Zou
bd0e401cae fix[js]:fix ut case faulure 
- refactor default context creation to avoid data race
- refactor the timer interval in c_worker UT cases to avoid receieving signals at the same time

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-03-03 11:47:41 +08:00
Wang Yan
948d45604c Revise the GC job flow, 
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-02-24 18:29:55 +08:00
Wenkai Yin
528f598268 Reimplement the registry client 
This commit reimplements the registry client under directory src/pkg/registry and removes the useless code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-02-24 14:36:26 +08:00
Daniel Jiang
2064a1cd6d Switch to basic authentication for registry 
1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth.  #10604 is opened
to track the follow up work.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-01-31 21:46:47 +09:00
Steven Zou
e899d659f3 fix[jobservice]:job status is hung after restart 
- improve the status hook sending/resending approach
- improve the status compare and set approach
- simplify the relevant flow
- add reaper to fix the out of sync jobs
- fix #10244 , fix #9963

Signed-off-by: Steven Zou <szou@vmware.com>
 2020-01-09 17:10:35 +08:00
Steven Zou
ebc5d2482b do improvements to the scan all job 
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job

fix #9705
fix #9722
fix #9670

Signed-off-by: Steven Zou <szou@vmware.com>
 2019-11-05 15:12:07 +08:00
Steven Zou
35d074e849 fix bug found in the pluggable scanner 
- fix #9632
- fix #9633

Signed-off-by: Steven Zou <szou@vmware.com>
 2019-10-29 15:32:51 +08:00
Wenkai Yin(尹文开)
28ee5f7d43
Merge pull request #9302 from liuxu623/fix-fatal-error 
fix fatal error
 2019-10-25 13:16:54 +08:00
He Weiwei
cc86664730 fix(gc,quota): clean blob size cache in redis when gc 
Closes #9151

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-10-22 14:20:04 +00:00
Steven Zou
0f16913635 rebase: resolve the code confilcts with master 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-10-17 17:42:41 +08:00
Steven Zou
f18afc0a3f do changes to let the vul policy check compatiable with new framework 
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema

Signed-off-by: Steven Zou <szou@vmware.com>
 2019-10-16 23:15:26 +08:00
Ziming
3725667448
Merge branch 'master' into feat/gitlab 2019-10-16 14:34:21 +08:00
liuxu
79322f2df2 fix fatal error 
Signed-off-by: liuxu <liuxu623@gmail.com>
 2019-10-14 21:41:29 +08:00
Wenkai Yin(尹文开)
7d0505593f
Merge pull request #8556 from chlins/feat/image-replication-adapter-for-quay.io 
Feat/image replication adapter for quay.io
 2019-10-14 09:16:45 +08:00
Steven Zou
58afd8e14b [stage3] support pluggable scanner 
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces

Signed-off-by: Steven Zou <szou@vmware.com>

- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases

- update robot account making content
- hidden credential in the job log

Commnet scan related API test cases which will be re-activate later
fix #8985

fix the issues found by codacy

Signed-off-by: Steven Zou <szou@vmware.com>
 2019-10-11 12:53:02 +08:00
chlins
4ab3b864ae feat: add image replication adapter for quay.io 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2019-10-11 10:00:07 +08:00
Wenkai Yin(尹文开)
0a85acac9a
Merge pull request #9329 from chlins/feat/jfrog-docker-registry 
feat(replication): support for jfrog artifactory docker image replica…
 2019-10-10 09:41:41 +08:00
chlins
0e0c42d3c0 feat(replication): support for jfrog artifactory docker image replication 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2019-10-09 21:06:36 +08:00
Qian Deng
d2d0affcae
Merge pull request #9245 from ninjadq/check_logid 
Add checking for logid
 2019-09-26 10:26:49 +08:00
Wenkai Yin(尹文开)
ac17afac1f
Merge pull request #9243 from ninjadq/change_file_permission_of_joblog 
change the permission of job log file
 2019-09-25 17:45:21 +08:00
DQ
e078b1c209 Add checking for logid 
the logid content need check before use

Signed-off-by: DQ <dengq@vmware.com>
 2019-09-25 13:55:32 +08:00
DQ
4333c3ad5c change the permission of job log file 
Log file permission from 644 to 600

Signed-off-by: DQ <dengq@vmware.com>
 2019-09-24 20:04:17 +08:00
Wenkai Yin
c2e529576a Update the error log format 
Update the erro log format in jobservice

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-09-23 15:40:04 +08:00
wang yan
42a5db83b2 refactor selector of retention 
extract select from pkg/retention, move it to pkg/artselector to make it usable by immutable tag

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-09-18 16:38:41 +08:00
lxShaDoWxl
6937731744 feat(Registries): added gitlab adapter 
Signed-off-by: lxShaDoWxl <lxshadowxkingxl@gmail.com>
 2019-09-09 21:21:21 +06:00
Wang Yan
39f78ae768
Merge pull request #7872 from cd1989/config-redis-pool-idletimeout 
Config idle timeout for redis pool to avoid jobservice restarting
 2019-08-27 14:46:01 +08:00
wang yan
e3155e00d6 fix #8815 :add remove untagged blob record in table project_blob 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-08-26 15:57:19 +08:00
Steven Zou
718e336302 reduce the expire time of job stats in redis DB 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-08-22 18:26:09 +08:00
Wenkai Yin(尹文开)
e3d8fa845d
Merge pull request #8767 from steven-zou/fix/issue-#8525 
fix #8525: periodic job retry issue: job stats are not found
 2019-08-22 11:06:13 +08:00
wang yan
e91ded65cb fix quota size usage in gc job, issue #https://github.com/goharbor/harbor/issues/8699 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-08-21 16:51:31 +08:00
Steven Zou
926ac44a67 fix #8525: periodic job retry issue: job stats is not found 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-08-21 16:30:47 +08:00
cd1989
db9b52d827 Config idle timeout for redis pool 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-08-20 17:23:32 +08:00
Steven Zou
ace07b0c0a fix #8745: job retry issue 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-08-20 16:39:44 +08:00
王添
94d4f9c6b6 add webhook job 
Signed-off-by: 王添 <wangtian@corp.netease.com>
 2019-08-07 20:56:31 +08:00
Wenkai Yin
8325f0769f Ignore the arguments when logging the job detail 
As the arguments of a job may contain sensitive information, ignore them when logging the job detail. Fixes #8404

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-08-06 15:02:14 +08:00
Wenkai Yin(尹文开)
27245021d3
Merge pull request #8500 from steven-zou/fix/pool_run_out 
fix bug #8499: redis connection pool is run out
 2019-08-02 17:47:26 +08:00
Wenkai Yin
1aa5e631d7 Remove the retention job for deleting repository 
Remove the retention job for deleting repository

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-08-02 10:27:19 +08:00
Steven Zou
3a68bd7aa2 fix bug #8499: redis connection pool is run out 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-07-31 15:45:03 +08:00
Wenkai Yin
6464dcbd51 Register the replication adapter for Helm Hub 
Register the replication adapter for Helm Hub

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-30 16:28:06 +08:00
Steven Zou
1e7f51615f fix conflicts in runtime/bootstrap.go 2019-07-30 09:52:22 +08:00
Steven Zou
3ca5116a53 add retention job for deleting repository 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-07-30 09:45:19 +08:00
Ziming
a5c31d7160
Merge pull request #8456 from ywk253100/190729_scheduler 
Register the scheduler job and fix bugs
 2019-07-30 08:57:10 +08:00
Wenkai Yin
94c1cf8ac9 Register the scheduler job and fix bugs 
1. Register the scheduler job
2. Fix retention bugs

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-30 07:37:47 +08:00
慕薇疯魔
0d71eb363a Aliyun ACR adapter 
1. aliyun ACR registry adapter.
2. Unit test case.
3. Update go modules and vendor.

Signed-off-by: 疯魔慕薇 <kfanjian@gmail.com>
 2019-07-29 18:24:03 +08:00
Steven Zou
c44747fd3c merge code from master and fix conflicts 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-07-24 17:27:37 +08:00
Wenkai Yin
1adf5c976e Create retention client with the configurations got from jobservice config 
As the retention client is only used in job which running in jobservice, the configurations that needed to init the client should be read from jobservice

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-24 13:10:10 +08:00
Wenkai Yin
7362fae7cc Implement a common scheduler 
Implement a common scheduler that can be used globally

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-23 17:20:31 +08:00
Steven Zou
ccad9c0a29 support stop in the retention job and register it 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-07-23 11:05:22 +08:00
Bernardo Pastorelli
791394a2b1 Make jobservice use the proxy env variables (including no_proxy) 
Signed-off-by: Bernardo Pastorelli <berpast@hotmail.com>
 2019-07-19 16:32:17 +02:00
Wenkai Yin
d6c6231e08 Implement the retention client 
Implement the retention client

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-07-17 14:04:29 +08:00
Steven Zou
63e2ce7606
Merge pull request #8209 from steven-zou/fix/enqueuer_schedule_slot 
use separate key for the last periodic enqueue data
 2019-07-03 21:48:21 +08:00
Steven Zou
b88159e747 use separate key for the last periodic enqueue data 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-07-03 17:08:23 +08:00
Steven Zou
1002e3ce17 fix failure ut case of job service 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-07-03 15:10:41 +08:00
cd1989
2097e928d0 Implement azure acr adapter 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-06-29 18:34:48 +08:00
Ziming
af548e915e
Merge branch 'master' into replication_gcr_1.9 2019-06-27 11:27:33 +08:00
Steven Zou
5521b7b7ad
Merge pull request #7915 from bitsf/replication_ecr_1.9 
aws driver for replication
 2019-06-27 11:24:54 +08:00
Ziming Zhang
072bdd101b aws driver for replication 
Change-Id: I8792ffce2eaa5975359bb6159a1ba7b85926a925
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-06-25 19:11:27 +08:00
Ziming Zhang
e387c63242 gcr driver for replication 
Change-Id: I5a6626950d3878bfa9726b332e68bee59159269f
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-06-25 18:08:10 +08:00
Steven Zou
7c86c98687
Merge pull request #8049 from steven-zou/fix_js_ut_cases 
fix failure UT case of job service
 2019-06-21 14:55:50 +08:00
Steven Zou
b9f45cd30a fix failure UT case of job service 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-06-21 13:31:27 +08:00
Steven Zou
2bc1402357 add migration process for rdb as well as UT cases 
fix issue #7988

Signed-off-by: Steven Zou <szou@vmware.com>
 2019-06-12 12:29:51 +08:00
Steven Zou
e091916d54 fix UT case failure (#7734) 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-05-10 13:15:44 +08:00
Steven Zou
b664b90b86
Merge pull request #7585 from steven-zou/add_get_jobs_API 
add get jobs API to provide powerful job stats management
 2019-05-08 17:17:33 +08:00
Steven Zou
2562146faf adjust the order of robot test case of API_DB 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-05-07 16:13:31 +08:00
Steven Zou
f2870272ce add get jobs API to provide powerful job stats management 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-28 22:36:13 +08:00
Wenkai Yin
2a463016a9 Upgrade the distribution and notary library (#7516) 
* Return 404 when the log of task doesn't exist

Return 404 when the log of task doesn't exist

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Upgrade the distribution and notary library

Upgrade the distribution library to 2.7.1, the notary library to 0.6.1

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-28 12:00:26 +08:00
Steven Zou
9bcbe2907b fix go vet issues in the code 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-24 07:31:37 +08:00
Steven Zou
ad68a3f79d update the code of other components influenced by the js code change 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-22 22:21:49 +08:00
Steven Zou
fae5d1304d add more/refacotr current test cases of job service 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-22 19:24:37 +08:00
Steven Zou
6f8a80d21c refine the hook event and status update retry approach 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-20 15:00:32 +08:00
Steven Zou
8e734407c0 Merge branch 'master' into fix_issues_for_jobservice 2019-04-19 21:15:21 +08:00
Steven Zou
ea34bcc950 add retry for status switching 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-19 21:14:09 +08:00
Steven Zou
38e6df4edb fix code style issues: missing comments 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-19 14:16:08 +08:00
Steven Zou
f8feaa192e add get scheduled and periodic executions APIs 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-19 13:54:23 +08:00
Yuan Lei
7ee009f1d2 add huawei adapter to replication module, and adjust some interface after testing 
Signed-off-by: Yuan Lei <371304458@qq.com>
 2019-04-19 10:05:39 +08:00
Steven Zou
608354dd04 refine the returnning errors 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-18 17:15:55 +08:00
Steven Zou
1f481e492c Refactor job servcie primary logic to fix related bugs 
Signed-off-by: Steven Zou <szou@vmware.com>
 2019-04-18 16:02:49 +08:00
Wenkai Yin
547c2337de Update the migration sql 
1. Update the migration sql
2. Rename the ResourceRepository from repository to image

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-17 17:18:03 +08:00
Wenkai Yin
1d16e18dff Remove "ng" from source code 
Remove "ng" from source code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-12 22:42:27 +08:00
慕薇疯魔
a555b09df0 1. Implement native docker registry adapter. 
2. register the native adapter.

Signed-off-by: 慕薇疯魔 <kfanjian@gmail.com>
 2019-04-12 21:08:23 +08:00
cd1989
b876a576a6 Implement dockerhub adapter 
Signed-off-by: cd1989 <chende@caicloud.io>
 2019-04-12 17:31:32 +08:00
Wenkai Yin
b73acde051 Support the migration for scheduled replication rule from previous version of Harbor 
Support the migration for scheduled replication rule from previous version of Harbor

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-04-11 13:14:32 +08:00
Wenkai Yin
4f8e283e8e Add event based trigger and scheduled trigger 
This commit implements the event based trigger and scheduled trigger in replilcation

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-29 13:48:34 +08:00
Wenkai Yin
49cf50adb1 Merge remote-tracking branch 'upstream/master' into 190324_sync 
Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-22 15:55:52 +08:00
Wenkai Yin
258b22a9a5 Fix bug in replication 
This commit fixes bugs found in the implement of replciation NG

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-03-10 20:57:59 +08:00
Wenkai Yin
772367498f Merge remote-tracking branch 'upstream/master' into 190311_sync 2019-03-11 20:34:49 +08:00
Qian Deng
5f80fe7b8a Refacotr the prepare script base on the proposal https://github.com/goharbor/community/pull/22 
Signed-off-by: Qian Deng <dengq@vmware.com>
 2019-03-08 16:46:13 +08:00
He Weiwei
053a18a490 Skip transfer blob for foreign layers 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-02-27 16:14:17 +08:00
Wenkai Yin
95888b3dc2
Merge branch 'replication_ng' into 190130_transfer_repo 2019-02-27 11:00:42 +08:00
stonezdj(Daojun Zhang)
4cb49e5388
Merge pull request #6963 from stonezdj/remove_container 
Remove everything of adminserver container
 2019-02-22 18:27:43 +08:00
stonezdj
0cba36d79f Remove everything of adminserver 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-02-22 16:34:39 +08:00
Nguyen Quang Huy
eda6c47b3e add signoff for DCO gate (#6981) 
Some variable name, function name is colliding with builtin function.

Signed-off-by: Nguyen Quang Huy <huynq0911@gmail.com>
 2019-02-22 15:00:18 +08:00
Wenkai Yin
0c0c833dda Merge remote-tracking branch 'upstream/master' into 190220_sync 2019-02-20 17:21:38 +08:00
stonezdj
1ae5126bb4 Refactor adminserver stage 3: replace config api and change ut settings 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-02-18 14:06:19 +08:00
Wenkai Yin
f0f2e77fb4 Implement the repository transfer 
This commit implements the Transfer interface for resource repository

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-02-15 17:49:35 +08:00
Wenkai Yin
759cb5467b Refine the replication job 
1. Refine the replication job
2. Define the Transfer interface
3. Define the common models

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-01-29 15:48:02 +08:00
Wenkai Yin
5388eb4db6 Read ssl mode param from the adminserver to init the connection with database (#6504) 
This commit adds the ssl mode as a param when establishing the connection with database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2018-12-11 13:58:37 +08:00
peimingming
585d7b2b01 Add unit test for JobService Logger 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2018-12-03 16:20:03 +08:00
Steven Zou
84d864607c Add UT cases for new added code 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-30 16:08:15 +08:00
Steven Zou
ccd486a0ad Support unique job constraints (both unique queue and running jobs) 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-30 14:14:21 +08:00
peimingming
238dbc0347 Add UT and review comments and issue fix (#6144) 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2018-11-28 17:43:14 +08:00
peimingming
c67fdc40f5 Support store job log in DB (#6144) 
Signed-off-by: peimingming <peimingming@corp.netease.com>
 2018-11-28 15:09:29 +08:00
Steven Zou
01bb7f4abb Uncomment the harbor configuration injection code block 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-09 10:19:14 +08:00
Steven Zou
8f0e8d083a Fix typo in the README doc of jobservice 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-08 18:24:41 +08:00
Steven Zou
093bf9fc8a Update the job service config yaml in the installation template 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-08 10:40:39 +08:00
Steven Zou
e6d4c024ee Update README of job service to reflect latest updates 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-08 10:35:12 +08:00
Steven Zou
e9194ca107 Add/update unit test cases for the new implemented job logger framework 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-06 14:43:06 +08:00
Steven Zou
7b106d06c5 Build logger framework to support configurable loggers/sweepers/getters 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-11-06 09:31:31 +08:00
Steven Zou
300977d539 Use separate redis conn for the lock mechanism in the periodic enqueuer 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-31 15:19:11 +08:00
Steven Zou
58eefb9c17 Add TestOnBorrow check func of redis pool to verify the getting conn 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-31 13:30:29 +08:00
Steven Zou
b48748492e Fix the issue of stopping periodic job (#6174) 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-31 11:43:46 +08:00
Steven Zou
8341aa14b0 Support health check free call in job service 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-30 17:35:07 +08:00
Wenkai Yin
af17c2c3fd
Merge pull request #6154 from wy65701436/fix-gc-redis 
Fix gc jobs issues
 2018-10-29 16:29:57 +08:00
Steven Zou
dde6c39e80
Merge pull request #6169 from steven-zou/fix_issue_of_periodic_conflicts 
Return 409 conflict error in liue of 200 with ID of existing one
 2018-10-29 16:18:40 +08:00
Steven Zou
408b751ae6 Return 409 conflict error in liue of 200 with ID of existing one when submmiting periodic job 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-29 15:40:31 +08:00
wang yan
903d3906ab Fix per comments by wk 
Signed-off-by: wang yan <wangyan@vmware.com>
 2018-10-29 15:29:14 +08:00
Steven Zou
d32e51052e Fix the issue of duplicated periodic job scheduling 
Signed-off-by: Steven Zou <szou@vmware.com>

Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-29 14:52:45 +08:00
wang yan
566169f59a Fix gc jobs issues. 
This commit is to fix two problems in the gc job,
1, in order to support redis cluseter, gc job should delete the keys only
for docker reigstry instead of all keys. It because that in the redis cluster,
only db 0 is supported.
2, to restore the real status of read only not just to set it to false.

Signed-off-by: wang yan <wangyan@vmware.com>
 2018-10-26 18:37:20 +08:00
Steven Zou
a2f0da0d7d Fix the web hook mising issue for the periodic job executions 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-26 14:28:04 +08:00
Steven Zou
0c901a475f
Merge pull request #6074 from goharbor/enhance_job_service 
Enhance job service to support multiple executions recording
 2018-10-23 09:50:05 +08:00
Steven Zou
7ca2e9df81 Enhance job service to support multiple executions rectording 
- print more logs
- add execution records for periodic job
- add launch job func in job context to launch sub job
- replace copyright header with new version

Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-19 11:02:09 +08:00
stonezdj
79bac7a64e Change admin server to core in jobservice 
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2018-10-15 14:56:18 +08:00
Steven Zou
326e0d455a Log error if failed to create logger for job 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-11 16:29:00 +08:00
Wenkai Yin
b4a9eae8cc
Merge pull request #6009 from reasonerjt/fix-insecure-transport 
Use secure transport to access HTTP endpoint
 2018-10-11 10:23:12 +08:00
Steven Zou
95d48436b8 Add README doc for job service (#6010) 
Signed-off-by: Steven Zou <szou@vmware.com>
 2018-10-10 19:08:20 +08:00
Daniel Jiang
1188bd89b9 Use secure transport to access HTTP endpoint 
In various parts of the code, we used insecure transport in http Client
when we assume the endpoint is http.  This causes complaints form
security scanner.  We should use secure transport in such cases.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-10-10 17:51:02 +08:00
Daniel Jiang
0699980924 Add Scan All job to job service (#5934) 
This commit adds the job to scan all images on registry.
It also makes necessary change to Secret based security context, to
job service has higher permission to call the API of core service.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-09-22 13:07:32 +08:00
clouderati
587459df15 Replacing copyright notices with "Copyright Project Harbor Authors". 
Signed-off-by: clouderati <35942204+clouderati@users.noreply.github.com>
 2018-09-19 16:59:36 +00:00
Qian Deng
7873a0312a Rename harbor-ui to harbor-core 
1. Update the nginx.conf
2. Update Makefile
3. Update docker-compose
4. Update image name
5. Rename folder ui to core
6. Change the harbor-ui's package name to core
7. Remove unused static file on harbor-core
8. Remove unused code for harbor-portal

Signed-off-by: Qian Deng <dengq@vmware.com>
 2018-09-19 16:35:13 +08:00
陈德
0582db9a82 Apply consistent format for comments 
Signed-off-by: 陈德 <chende@caicloud.io>
 2018-09-05 16:16:31 +08:00
Steven Zou
1636b138f2 Return more data of the job when reporting status info via webhook 
- update the status report func `RedisJobStatsManager.reportStatus`
- update the UT case for the above change

Signed-off-by: Steven Zou <szou@vmware.com>
 2018-08-30 14:28:12 +08:00
wang yan
aab761ac8a Fix gofmt check results 
Signed-off-by: wang yan <wangyan@vmware.com>
 2018-08-29 11:50:00 +08:00
Yan
fca2bb3a6b
Fix misspell checking results (#5749) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2018-08-29 10:25:42 +08:00
Daniel Jiang
dcf4e2ee78 Update import path in go code 
vmware -> goharbor

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-08-23 17:50:53 +08:00
Daniel Jiang
65cf02a1d7 Validate job ID when getting job log 
Add validation to job ID in the API to get job log in job service, to
prevent file path traversal attack.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2018-08-15 15:33:13 +08:00
wangyan
29d0d51403 Signed-off-by: wangyan <wangyan@vmware.com> 
Add clean registry cache to gc job

To workaround the issue: https://github.com/docker/distribution/issues/2094
GC needs to clean cache before to call the docker reigstry api to delete blobs.
Otherwise, the following docker push will not be performed as docker registry
does not clean cache in GC, it thinks the image is still there, and the new
blobs will be uploaded.
 2018-08-13 02:58:27 -07:00
wangyan
9a95f14918 Cherry-pick -- Fix security issue found by gas 2018-08-03 01:16:53 -07:00
Wenkai Yin
2a40068d75 Fix cross slot issue of redis 
This commit upgrdes the github.com/gocraft/work package to 0.5.1 and updates the namespace value
to fix the redis cross slot issue mentioned in #4633
 2018-07-27 14:23:39 +08:00
Yan
efdb57548f
add admin job api (#5344) 
It supports Harbor admin to trigger job either manual or
schedule. The job will be populated to job service to execute. 
The api includes:
1. POST /api/system/gc
2, GET /api/system/gc/:id 
3, GET /api/system/gc/:id/log
4, PUT/GET/POST /api/system/gc/schedule
 2018-07-20 19:22:37 +08:00
Yan
9e65499c10
Add garbage collection job implemention, this job could (#5268) 
be triggered by manual and schedule. It calls registrtctl
to do the GC job, and log the output.
 2018-07-16 18:08:40 +08:00
Steven Zou
4db708096b Add retry for getting configs from admin server when doing job conetxt initialization 2018-05-10 15:26:59 +08:00
Steven Zou
f7bc467c99 Return none zero code when job service exit with error 
replace fmt.Println/logger.Errorf with logger.Fatal(f)
 2018-05-10 14:44:04 +08:00
Yan
ae257433cc
Fully migrate harbor db to postgresql (#4689) 
* Merge harbor db to postgres
 2018-04-27 02:27:12 -07:00
Steven Zou
e1b509e3f3 Fix the issue of intermittent restarting of job service 
github issue: #4712
ping redis server before pool starting

Let worker pool to restart the message server if message server exits with error (controlled by max retries)
 2018-04-25 17:02:16 +08:00
Steven Zou
a17436962f Return the logger.Interface instead of the pointer of logger to avoid nil checking issue 2018-04-20 14:59:28 +08:00
Steven Zou
ca8d3bdcc9
Merge pull request #4638 from vmware/use_redis_url_addr 
Use redis URL address to replace host:port when connecting to redis server
 2018-04-17 10:12:51 +08:00
Steven Zou
adc2f8f124 Use redis URL address to replace host:port when connecting to redis server 
replace tcp host:port with
'redis://arbitrary_usrname:password@ipaddress:port/database_index'

update prepare to generate config yaml file of job service based on harbor.cfg

update harbor.cfg default values
 2018-04-13 19:19:56 +08:00
Steven Zou
b5b728bee3 Fix the vulnerability issues in the job service code 
'Errors unhandled' in 'jobservice/job/impl/logger/job_logger.go'
'TLS InsecureSkipVerify set true' in 'jobservice/opm/hook_client.go'
 2018-04-12 15:12:39 +08:00
Steven Zou
85ae40f5ec Apply auth checking to all the incoming requests 2018-04-08 18:12:13 +08:00
Steven Zou
529ad3e079
Merge pull request #4593 from vmware/add_more_log_2_js 
Fix issue: failed to update the status of job if runtme error occurred
 2018-04-08 14:03:34 +08:00
Steven Zou
233692c127 fix issue: job context may be nil pointer when trying to be closed in defer func 2018-04-08 13:03:23 +08:00
Steven Zou
1cb40368f5 Fix issue: failed to update the status of job if runtme error occurred 2018-04-08 10:38:47 +08:00
Tan Jiang
ff06ec05c3 Store secret in header instead of cookie 2018-04-07 22:02:06 +08:00
Tan Jiang
15580a5e8c Read the system properties from scan job context 2018-04-04 19:58:54 +08:00
Steven Zou
6c69a8cd05 Fix issue of stopping periodic job 
improve op command by using cache
return 404 if no job found to stop
 2018-04-02 18:08:03 +08:00
Steven Zou
fee7f6ddef Merge branch 'job_service' into switch_job_service 2018-04-02 12:06:23 +08:00
Steven Zou
250360307b Modify docker compose file template and make file to enable new job service 
Fix typo in Makefile under photon

Fix version tag issue of redis container

Assign container name for redis container

Update docker compose template to enable network for redis

Remove exposed ports of redis from compose yaml tpl
 2018-03-30 16:52:55 +08:00
Steven Zou
013028ef4a Replace old job service package with V2 (rename packages) 2018-03-30 11:22:29 +08:00
Steven Zou
bfbb949034 Use the new job service(v2) to replace the old one 2018-03-29 23:30:03 +08:00
yixingj
cb64ad96ff Make endpoint configurable 
Move all the endpoint to harbor.cfg
 2018-03-26 10:50:18 +08:00
Wenkai Yin
9022abfc13 Fix code issues found by Gas 2018-01-29 15:17:03 +08:00
Wenkai Yin
4070ed5152 Provide a mechanism to stop pending and retrying jobs 2018-01-12 15:29:20 +08:00
Wenkai Yin
3be1d5a7fd Assign read-only privilege of replication policy to project admin and add stopping replication jobs API. 2017-12-25 23:49:21 +08:00
Wenkai Yin
260ef561c4 Update the HTTP client for easy use by add more util functions 2017-12-16 06:45:59 +08:00
Wenkai Yin
a736cb7b09 Update the HTTP client according to the comments 2017-12-15 09:40:31 +08:00
Wenkai Yin
b5e7de331e Delete enabled and start_time properties of replication rule 2017-12-15 09:40:31 +08:00
Wenkai Yin
fe10c2e7f5 Create replicator to submit replication job to jobservice 2017-12-15 09:40:31 +08:00
Daniel Jiang
cdadc94d0f
Merge pull request #3804 from ywk253100/171215_jobservice 
Print stack trace when recover from panic and print warning message rather than returning an error when updating 0 records
 2017-12-18 16:36:20 +08:00
stonezdj
9393d26fdc Fix ldap ping issue #3653 2017-12-15 14:47:54 +08:00
Wenkai Yin
43489c2b67 Print stack trace when recover from panic and print warning message rather than returning an error when updating 0 records 2017-12-14 13:48:45 +08:00
Tan Jiang
b3e0af2382 Fix permission issue in job_log directory 2017-11-21 19:31:15 +08:00
Daniel Jiang
a2f20801c0
Merge pull request #3597 from ywk253100/171110_bug_fix 
Add content-type header to the request when creating project during replication
 2017-11-12 22:54:15 -06:00
Wenkai Yin
9889896e8a Add content-type header to the request when creating project during replication 2017-11-10 16:08:13 +08:00