Commit Graph

1050 Commits

Author SHA1 Message Date
Daniel Jiang
947eadaa72
Merge pull request #12440 from heww/remove-init-clair-db
refactor: remove initialization of clair db
2020-07-15 00:38:12 +08:00
He Weiwei
c000608d55
Merge pull request #12437 from heww/db-max-connections
chore(db): change max_connections of postgres to 1024
2020-07-14 17:24:16 +08:00
He Weiwei
2a6fe801bc chore(db): change max_connections of postgres to 1024
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-14 07:34:37 +00:00
Daniel Jiang
e96165412d
Merge pull request #12432 from ywk253100/200709_allowlist
Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist"
2020-07-13 16:42:43 +08:00
chlins
38d14dff30 fix(preheat): validate instance/policy name, set unique filed and policy
manager adds parsePolicy

Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-10 10:29:47 +08:00
He Weiwei
039aef5356 refactor: remove initialization of clair db
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-09 15:26:14 +00:00
Wenkai Yin
cd6c1b8c31 Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist"
Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist" in project metadata

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-09 11:04:44 +08:00
Daniel Jiang
1637e6a588 Rename master role to maintainer
This commit rename the var name, text appearance, and swagger of "master" role
to "maintainer" role.
It only covers backend code.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-07-08 09:20:07 +08:00
DQ
4617e0ff38 Enhance: Support multi downversion in migration
1. Change down version to list to accept multi verstion value
2. Update search function use BFS to find migration path
2. Add test case

Signed-off-by: DQ <dengq@vmware.com>
2020-07-07 21:36:58 +08:00
stonezdj(Daojun Zhang)
6f4e8150d5
Merge pull request #12383 from ywk253100/200702_registry_api
Suport filtering registries by type in listing registry API
2020-07-07 14:21:54 +08:00
Wenkai Yin
02690d1d04 Suport filtering registries by type in listing registry API
Suport filtering registries by type in listing registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-07 10:30:46 +08:00
fanjiankong
a0c2d0ac9e feat(preheat):add preheat api, controller and manager
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-03 11:25:42 +08:00
Wenkai Yin(尹文开)
1d03b8727a
Merge pull request #12357 from ninjadq/add_env_for_aws
Fix Amazon S3 storage not work
2020-07-01 11:10:47 +08:00
chlins
15e4361d6e feat: add p2p preheat policy dao and manager(#12286)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-06-30 15:56:50 +08:00
DQ
d0ddd61ad9 Fix Amazon S3 storage not work
The Chartmuseum S3 client need set an Env variable
Ref: https://github.com/helm/chartmuseum/issues/280

Signed-off-by: DQ <dengq@vmware.com>
2020-06-30 15:16:18 +08:00
He Weiwei
0474a2a040
Merge pull request #12322 from heww/install-tls-ca
feat(certs): install internal tls ca from /etc/harbor/ssl dir
2020-06-25 21:03:35 +08:00
He Weiwei
13436b75a6 feat(certs): install internal tls ca from /etc/harbor/ssl dir
Closes #10222

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-06-24 08:58:08 +00:00
AllForNothing
fff6f7529a Replace all whitelist with allowlist
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-06-24 16:17:17 +08:00
Wang Yan
53044da28f
update blob controller & manager (#12101)
* update blob controller & manager

1, add two more attributes, version, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 17:11:54 +08:00
wang yan
c10467eb36 continue refactor
Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
Wang Yan
de504993ad update blob controller & manager
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
Max Rosin
34d5591b1b Fix DOCKERIMASES and SWAAGER_IMAGE_BUILD_CMD typos in Makefiles
Fix #12259

Signed-off-by: Max Rosin <git@hackrid.de>
2020-06-16 12:18:55 +02:00
Wenkai Yin
127988b70c Define the task manager interface and data model
Define the task manager interface and data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-15 18:42:09 +08:00
Wenkai Yin
a79bb127b3 Update creating project API to support proxy cache project
Update creating project API to support proxy cache project

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-10 17:14:12 +08:00
Wang Yan
dec8397c21
Add api to delete blob and manifest (#12006)
* Add api to delete blob and manifest

Enable the capability of registry controller to delete blob and manifest

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-06 01:34:23 +08:00
Qian Deng
9e1302211b
Merge pull request #12072 from ninjadq/add_timeout_in_nginx_config
Add timeout in nginx config
2020-06-02 15:14:42 +08:00
Steven Zou
c7c1742b88
Merge pull request #12106 from heww/clean-clair-url
refactor(configuration): cleanup unneeded CLAIR_URL configuration in core
2020-06-01 19:24:19 +08:00
Daniel Jiang
58894e9d9c
Merge pull request #12071 from ninjadq/upgrade_chartversion
Enhance: Upgrade chartmuseum version
2020-06-01 13:36:54 +08:00
Daniel Jiang
6271da471b
Update health check script for harbor-db (#12103)
This patch remove the trailing space of the hostname introduced by
`hostname -i`.

The trailing space will cause resolution error after this patch is
applied to glibc in photon:
https://github.com/vmware/photon/blob/2.0/SPECS/glibc/glibc-fix-CVE-2019-10739.patch

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-05-30 14:05:39 +08:00
He Weiwei
d97be71234 refactor(configuration): cleanup unneeded CLAIR_URL configuration in core
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-05-29 07:27:50 +00:00
DQ
278338e401 Add timount on nginx configs
set timeout to 900

Signed-off-by: DQ <dengq@vmware.com>
2020-05-26 16:18:35 +08:00
DQ
715685ae51 Remove tls1.1 in notary
Signed-off-by: DQ <dengq@vmware.com>
2020-05-26 16:11:57 +08:00
DQ
f7ffd991cc Enhance: Upgrade chartmuseum version
Upgrade chartmuseum version 0.12.0

Signed-off-by: DQ <dengq@vmware.com>
2020-05-26 15:59:58 +08:00
AllForNothing
90e34e0104 Improve i18n service
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-05-06 14:45:56 +08:00
DQ
b06e19a637 Fix: GCS storage gc issue
Mount gcs key to registryctl

Signed-off-by: DQ <dengq@vmware.com>
2020-04-29 15:04:16 +08:00
Daniel Jiang
f91d7080d1
Merge pull request #11753 from tedgxt/2.0-webhook-event-types-fix
Webhook data fix when updgrding to 2.0
2020-04-28 19:36:44 +08:00
Qian Deng
9469252e85
Merge pull request #11745 from ninjadq/mount_ca_bundle
Enhance: Create shared to store shared ca
2020-04-28 10:19:26 +08:00
Qian Deng
7f1e3a7bb8
Merge pull request #11758 from ninjadq/output_subprocess_stdout
Output subprocess stdout
2020-04-28 09:46:02 +08:00
DQ
f70339870a Enhance: Create shared to store shared ca
this shared ca will mount to all harbor components

Signed-off-by: DQ <dengq@vmware.com>
2020-04-28 02:58:11 +08:00
guanxiatao
f96cfab100 Table notification_policy fix when updgrding to 2.0
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-04-27 20:13:46 +08:00
Wang Yan
add8dedc90
Fix the database upgrade issue (#11766)
It's a workaround for issue https://github.com/goharbor/harbor/issues/11754

The phenomenon is the repository data is gone, but artifacts belong to the repository are still there.
To resolve it, just set the repository_id to a negative, and cannot duplicate.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-27 17:28:36 +08:00
DQ
90faf700f8 Enhance: output the stdout of gen cert script
use popen replace check_all

Signed-off-by: DQ <dengq@vmware.com>
2020-04-27 10:43:22 +08:00
DQ
026e37e777 Fix chart museum absolute url issue
if absolute url is enabled return true else set it to false

Signed-off-by: DQ <dengq@vmware.com>
2020-04-26 13:04:29 +08:00
DQ
599ca98c09 Hidden veriify client cert verfiy option
Remove to avoid replication access core from external_url issue

Signed-off-by: DQ <dengq@vmware.com>
2020-04-23 10:14:36 +08:00
Daniel Jiang
2ecf0425a4 Remove the certs of notary signer
Since `prepare` generates the certs as needed during installation, these
certs should not exist in the repo.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-04-21 13:20:19 +08:00
DQ
b728f04d0a Fix tls min version for registry
cert,key,mintls should in the same context

Signed-off-by: DQ <dengq@vmware.com>
2020-04-20 19:19:15 +08:00
Qian Deng
9c7caddeae
Merge pull request #11635 from hyy0322/set-root-password-never-expire
fix: set root password never expire
2020-04-16 22:05:10 +08:00
Maosheng Ren
89e9ea0145
Merge pull request #11636 from danielpacak/bump_up_trivy_adapter_to_0.9.0
chore(trivy): Bump up trivy adapter to 0.9.0
2020-04-16 16:16:50 +08:00
Wang Yan
790064df2e
fix notification policy ugrade issue (#11627)
Fixes #11624

All of the existing policies created v1.10 has no name, it fails the upgrade process.
When to set the unique constraint for policy name, the empty can be seen as duplicated key.

ERROR:  could not create unique index "notification_policy_name_key"
DETAIL:  Key (name)=() is duplicated.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-16 14:53:58 +08:00
Daniel Pacak
5c3abee135 chore(trivy): Bump up trivy adapter to 0.9.0
- Vendor the latest Trivy release 0.6.0
- Configure TLS 1.2 as min version when TLS is enabled
- Add more tracing to adapter config to facilitate troubleshooting

Resolves: #11544

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-04-16 08:40:27 +02:00
DQ
42c1095216 Fix cert issue of trivy
Trivy can't access harbor from external if https enabled so inject cert to trivy container trust

Signed-off-by: DQ <dengq@vmware.com>
2020-04-16 10:52:03 +08:00
Yiyang Huang
4598f52057 fix: set root password never expire
Signed-off-by: Yiyang Huang <huangyiyang@caicloud.io>
2020-04-16 00:15:28 +08:00
He Weiwei
355c16943c chore(clair): bump up clair adapter version to 1.0.2
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-04-15 14:07:46 +00:00
He Weiwei
385aaac00d
Merge pull request #11620 from heww/fix-issue-11524
feat(scanner): make Clair and Trivy as reserved name for scanners
2020-04-15 15:21:35 +08:00
He Weiwei
f5487479dd feat(scanner): make Clair and Trivy as reserved name for scanners
Closes #11524

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-04-15 02:26:02 +00:00
Qian Deng
95d7c9382b
Merge pull request #11592 from ninjadq/min_version_tls_to_12
Min version tls to 12
2020-04-14 18:12:55 +08:00
wang yan
ff2a6c7a01 add warning to registry binary name
Fixes #11606

As we DO NOT want to user to execute GC in the container, rename it and append the warning message.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-14 15:16:50 +08:00
DQ
75f78b64b2 Set registry tls version to 1.2
when internal tls enabled set min version of registry to 1.2

Signed-off-by: DQ <dengq@vmware.com>
2020-04-13 18:13:30 +08:00
jwangyangls
e28b5811f7
Merge pull request #11176 from jwangyangls/change-helm-version
Separate swagger to get v2.0 swagger and chart swagger
2020-04-10 17:12:00 +08:00
Yogi_Wang
33ed4fb67e Separate swagger to get v2.0 swagger and chart swagger
1. Partial helm api version number clear
2. Separate swagger to get v2.0 swagger and chart swagger
3. router add chart swagger

Signed-off-by: Yogi_Wang <yawang@vmware.com>
2020-04-10 16:25:30 +08:00
DQ
e907cbe2b6 Fix health check for jobservice and regctl
need cert when mTLS is enabled

Signed-off-by: DQ <dengq@vmware.com>
2020-04-09 20:35:46 +08:00
DQ
08ff622310 Remove lines not needed
volume already defined above

Signed-off-by: DQ <dengq@vmware.com>
2020-04-09 20:06:51 +08:00
Ziming Zhang
572ebef685 feat(cicd) parameterize docker base image and external url
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-04-08 00:21:47 +08:00
DQ
6ae1b1dc97 Add missiong entrypoint file for trivy-adapter
Signed-off-by: DQ <dengq@vmware.com>
2020-04-07 10:39:07 +00:00
He Weiwei
3f567514b5
Merge pull request #11468 from wy65701436/remove-count-quota-code
remove the chart handling in quota
2020-04-07 16:51:07 +08:00
Daniel Jiang
5bcd015d6f
Merge pull request #11469 from ninjadq/clean_up_migrator
Remove migrator flags in script
2020-04-07 16:37:24 +08:00
Daniel Jiang
e064bd4c01
Merge pull request #11428 from ninjadq/fix_container_unhealth
Fix container unhealth
2020-04-07 15:57:00 +08:00
wang yan
a96d2f3746 remove the chart handling in quota
1, remove the chartmuseum controller
2, doesn't handle chartrepo url in v2 middleware

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-07 15:26:34 +08:00
DQ
1ae50b8d66 Remove migrator flags in script
Because migrator tool removed

Signed-off-by: DQ <dengq@vmware.com>
2020-04-07 14:57:10 +08:00
DQ
4a836ea975 Fix health check url
health check url should depend on internal https

Signed-off-by: DQ <dengq@vmware.com>
2020-04-07 03:35:52 +00:00
wang yan
44825e819e deprecate quota count on artifact
Fixes #11241

1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-06 16:56:11 +08:00
DQ
cdb675bf3d Add proxy cert file to jobservice when https enabled
jobservice may request via absolute path of url to harbor

Signed-off-by: DQ <dengq@vmware.com>
2020-04-04 17:44:34 +00:00
DQ
23ed189ed4 Add SAN to gencert script
add localhost and 127.0.0.1 to SAN

Signed-off-by: DQ <dengq@vmware.com>
2020-04-04 17:44:34 +00:00
He Weiwei
77a8c3205f fix(prepare): not accpet items of false value in external_redis
Item in yaml without value will be as None in python, which will make
the password of redis as `None` in `get_redis_configs`. This fix will
not accept items of `false value` in `external_redis` configurations.

Closes #11367

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-04-03 04:09:26 +00:00
Qian Deng
a702c32346
Merge pull request #11063 from ninjadq/fix_syslog_dir_in_tpl
Fix: fix logrotate is dir issue
2020-04-02 11:37:29 +08:00
Qian Deng
0319baabcb
Merge pull request #11381 from ninjadq/enhance_migrate_config
Enhance migrate config
2020-04-02 10:00:38 +08:00
DQ
dc271e1a87 Add packaging to pipenv
Signed-off-by: DQ <dengq@vmware.com>
2020-04-01 22:54:47 +08:00
DQ
d636f2ea5c Enhance help message
Provide more info in help message
Add requried opition and they will show missing option if you are not provide them instead of Exception

Signed-off-by: DQ <dengq@vmware.com>
2020-04-01 17:02:59 +08:00
DQ
b2e1905e7a Enhance: Stop upgrade when input version less then 1.9.0
The migration script should failure early when version is not supported

Signed-off-by: DQ <dengq@vmware.com>
2020-04-01 15:35:49 +08:00
Ziming Zhang
ae7834af0b feat(cicd) fix build base image
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-03-26 10:55:40 +08:00
Qian Deng
9e101b73a4
Merge pull request #11156 from ninjadq/migrate_config_to_harbor2
Migrate config to harbor2
2020-03-25 16:02:18 +08:00
DQ
85ec0e7820 Enhance: Refactor the migration structure
1. Refactor structure of migrate file
2. fix some previous bugs

Signed-off-by: DQ <dengq@vmware.com>
2020-03-23 21:26:28 +08:00
DQ
444678fe07 Fix: module path raise exception when it is loop
add test for loop

Signed-off-by: DQ <dengq@vmware.com>
2020-03-23 19:29:59 +08:00
Maosheng Ren
1dbec0c1d7
Fix a typo in the help message of install.sh (#11167)
Signed-off-by: ren maosheng <stevenr@vmware.com>
2020-03-23 10:30:37 +08:00
DQ
e8bb977ae1 Feat: Upgrade configs to harbor 2.0
add migrate files for harbor 2.0

Signed-off-by: DQ <dengq@vmware.com>
2020-03-20 15:20:32 +08:00
DQ
1e0c9f7231 Feat: Add config migrator to prepare
deprecated migrator container and move config migration to prepare

Signed-off-by: DQ <dengq@vmware.com>
2020-03-20 03:04:10 +08:00
Steven Zou
2859cd8b69
Merge pull request #11134 from danielpacak/feat/issue_11090/trivy_skip_update_flag
feat(trivy): Configure Trivy to skip database updates
2020-03-19 18:13:08 +08:00
Wenkai Yin(尹文开)
9ebcf95758
Merge pull request #11122 from ywk253100/200318_replication_task
Increase the length the columns (src_resource, dst_resource)of replication_task
2020-03-19 12:16:27 +08:00
DQ
f18a546429 Fix: return error when internal_tls_not_provided
When iinternal_tls is empty, prepare should works as usual

Signed-off-by: DQ <dengq@vmware.com>
2020-03-19 10:37:58 +08:00
Daniel Pacak
7325105714 feat(trivy): Configure Trivy to skip database updates
Resolves: #11090

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-03-18 17:11:47 +01:00
DQ
6e8d44101f Enhance: User can generate cert by their own ca key pair
User can put their ca key pair on internal cert dir and name them to `harbor_internal_ca.key` and `harbor_internal_ca.crt` we wil use them to generate other certs

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
b93092e012 Add tls for trivy
Add trivy tls cert files
Add tivey tls env and config
enhance gencert

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
c954969bcd Add mTLS configs
mTLS only enabled in jobservice and registryctl

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
c5d73e6a0c Add switch to https
use switch to make decision whether mTLS or server TLS

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
454382149f TLS update for chart, clairadapter, registry
Remove trustca in chartmuseum
Remove trustca in registry
Add tls in clair-adapter

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
03e11c63c7 Fix docker file with secure tls change
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ
dcc6950af7 Feat: auto install ca in registry
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00
DQ
b852605193 Feat: enable mtls in harbor replication
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00
DQ
40e67f3b14 Feat: Enable mtls for registry
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00
DQ
07a1d51693 Feat: enable tls in registryctlAdd tls related code in registryctl
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00