Upstream
/
harbor
mirror of https://github.com/goharbor/harbor.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
12,131 Commits 46 Branches 299 Tags 356 MiB
Commit Graph

1362 Commits

Author SHA1 Message Date
Wenkai Yin(尹文开) 50a1e85095
Make sure the revision of execution isn't null during the upgrade (#14085) 
Make sure the revision of execution isn't null during the upgrade
Fixes #14075

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2021-01-27 10:10:36 +08:00
Qian Deng f013d88efc
Merge pull request #14013 from ninjadq/upgrade_script_for_2_2_0 
Harbor upgrading for 2.2
 2021-01-22 18:10:24 +08:00
Qian Deng 045e1d9abe
Merge pull request #14040 from ninjadq/metric_improvement 
Metric improvement
 2021-01-22 17:13:57 +08:00
DQ 489f31d8fe Add upgrade scirpt for 2.2 
1. add metrics config item in config
2. upgrade version in template

Signed-off-by: DQ <dengq@vmware.com>
 2021-01-22 16:15:06 +08:00
Wang Yan dba229d0df
build third party binaries in CI (#14019) 
Signed-off-by: Wang Yan <wangyan@vmware.com>
 2021-01-22 11:33:42 +08:00
DQ 92cf728371 Add custom cert for exporter 
* injecting custom certs related config to exporter

Signed-off-by: DQ <dengq@vmware.com>
 2021-01-20 10:52:34 +08:00
DQ a61e9b0e2e Add san for notary upgrading 
if san not exists then remove that cert, prepare will regenerate one

Signed-off-by: DQ <dengq@vmware.com>
 2021-01-18 21:00:35 +08:00
Wenkai Yin 7c072e17a6 Fix the legacy scheduled job issue for GC/scan all 
Fix the legacy scheduled job issue for GC/scan all

Fixes #13968

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2021-01-15 22:02:36 +08:00
Daniel Jiang 1b64b9fdc2
Bump up the go-migrate (#13914) 
Bump it up to v4.11.0 to be consistent with harbor-core

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2021-01-11 19:08:17 +08:00
He Weiwei 4a326aa8b0 chore: delete records of scan_report 
The report in previous scan_report records not work well the
vulnerabilities stored in the schema table, so delete the scan_report
records.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2021-01-08 03:39:11 +00:00
Qian Deng 642d56041d
Add san for notary cert (#13928) 
Signed-off-by: DQ <dengq@vmware.com>
 2021-01-08 01:00:34 +08:00
stonezdj 6b8fb8431d Add quay registry to proxy cache 
Update env.jinja to add quay

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2021-01-06 17:22:57 +08:00
Ziming Zhang 8faa76a1b6 feat(retention) refactor task manager 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2021-01-05 12:08:03 +08:00
Wenkai Yin(尹文开) 19ad8ad68d
Merge pull request #13823 from reasonerjt/inst-cert-home-dir 
Replace tilde in install_cert.sh
 2020-12-25 10:25:51 +08:00
prahaladdarkin a890b28e1e
Store vulnerability data from scanner into a relational format (#13616) 
feat: Store vulnerability report from scanner into a relational format

Convert vulnerability report JSON obtained  from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
 2020-12-25 08:47:46 +08:00
Wang Yan 7a8a8fa104
upgrade go version to v1.15.6 (#13836) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-12-23 18:53:09 +08:00
He Weiwei 3831e82b20
refactor: remove code of admin job (#13819) 
Remove code of admin job as it's not needed by scan all/gc now.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-12-22 11:48:16 +08:00
Daniel Jiang 9d99dfa82b Replace tilde in install_cert.sh 
This commit fixes #13287 to remove the usage of tilde as the $HOME is not available in some
cases.  More details see #13287

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-12-21 20:39:34 +08:00
Qian Deng 31138f12b0
Merge pull request #13806 from ninjadq/fix_python_yaml_load 
Fix pythom yaml load to safe_load
 2020-12-21 16:04:12 +08:00
Qian Deng 9197471e70
Add Scan for internal tls (#13810) 
Signed-off-by: DQ <dengq@vmware.com>
 2020-12-21 15:23:11 +08:00
Wang Yan 9bc6f3cee4
fix robot account update issue (#13741) 
* fix robot account update issue

enable the update method to support both v1 & v2 robot update

Signed-off-by: Wang Yan <wangyan@vmware.com>

* resolve review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
 2020-12-18 20:01:26 +08:00
Wang Yan 6bc1047013
migration admin job data (#13766) 
1, migrate gc and scan all schedule to schedule/task/exectuion
2, migrate gc history to task/execution

Signed-off-by: Wang Yan <wangyan@vmware.com>
 2020-12-18 16:35:24 +08:00
Will Sun 4392a626f3
Merge pull request #13804 from AllForNothing/scan-all 
Fix robot account UI issues
 2020-12-18 15:48:26 +08:00
Qian Deng 64fcfeaa2f
Merge pull request #13754 from ninjadq/fix_loglevel_parsing_for_registry 
Fix log level issue in registry
 2020-12-18 14:47:15 +08:00
AllForNothing b20cc474b3 Fix robot account UI issues 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-12-18 14:11:08 +08:00
DQ 234b29e170 Fix pythom yaml load to safe_load 
Signed-off-by: DQ <dengq@vmware.com>
 2020-12-16 14:59:06 +08:00
Daniel Jiang b0c8cadad7
Add default CVE allowlist to project library (#13770) 
fixes #12700

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-12-16 14:20:56 +08:00
Wenkai Yin 69808f033e Tiny fixes for task manager 
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-12-14 17:26:32 +08:00
He Weiwei 08580f9fec
refactor(scan): refactor scan/scan all job to task manager (#13684) 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-12-14 13:34:35 +08:00
DQ 19e8527cc1 Fix log level issue in registry 
1. fix level issue in registry.jinja
2. add log level to registryctl

Signed-off-by: DQ <dengq@vmware.com>
 2020-12-14 11:52:42 +08:00
Wenkai Yin(尹文开) 6569016d35
Merge pull request #13139 from wy65701436/migrate-gc 
Migrate gc to task manager
 2020-12-14 10:43:44 +08:00
Wang Yan 39e1a4f2b4 add extra attributes in the schedule table 
Signed-off-by: Wang Yan <wangyan@vmware.com>
 2020-12-14 02:28:52 +08:00
wang yan 1bb79d402d update code per review comments 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-12-10 16:08:52 +08:00
Wang Yan dba5522d0b Migrate to task manager (#129) 
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-12-10 14:00:33 +08:00
DQ d95f22448c Add cache for exporter 
Add timed cache for exporter
default cache time is 30s, cleanup job run every 4 hours

Signed-off-by: DQ <dengq@vmware.com>
 2020-12-09 21:22:40 +08:00
DQ f0db193895 Add prepare file for exporter 
prepare env for exporter

Signed-off-by: DQ <dengq@vmware.com>
 2020-12-09 21:22:13 +08:00
DQ dc0047c48c Add build script for exporter 
- Add dockerfile
- update makefile

Signed-off-by: DQ <dengq@vmware.com>
 2020-12-09 20:42:21 +08:00
Wang Yan d2fa2e6b84
update robot secret (#13654) 
* update robot secret

1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object

Signed-off-by: Wang Yan <wangyan@vmware.com>

* update robot secret

1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object

Signed-off-by: Wang Yan <wangyan@vmware.com>
 2020-12-03 18:13:06 +08:00
Wang Yan 732e9a21cd
updates on robot accounts (#13623) 
* updates on robot accounts

1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account

Signed-off-by: wang yan <wangyan@vmware.com>

* update code per review comments

1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.

Signed-off-by: Wang Yan <wangyan@vmware.com>
 2020-12-01 18:31:34 +08:00
DQ 907904f480 Add DB Migration code for clair cleanning 
- Delete clair scanner if exist
- Delete report is it is scanned by clair
- Set Trivy to Default if it exist and not default scanner

Signed-off-by: DQ <dengq@vmware.com>
 2020-11-29 16:19:02 +08:00
DQ 590212b485 Remove clair related code 
- clair code in harbor core
- clair code in frontend
- clair code in robotcase

Signed-off-by: DQ <dengq@vmware.com>
 2020-11-27 14:01:04 +08:00
stonezdj(Daojun Zhang) be4e6a5985
Merge pull request #13537 from stonezdj/201118_add_more_registry_type 
Add more registry type to proxy cache
 2020-11-26 11:16:16 +08:00
Ziming Zhang d55f55aeb9 fix(chartmuseum) compatible s3 cache fail 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-11-25 17:00:16 +08:00
He Weiwei eb38180483 fix(quota): ignore the fail when getting reference of quota 
1. Clean the dirty data in quota/quota_usage.
2. Ignore the fail when getting the reference of quota.

Closes #13387

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-11-24 14:50:38 +00:00
Wenkai Yin(尹文开) fe8b628f0c
Merge pull request #13437 from ywk253100/200929_replication 
Refactor the replication execution
 2020-11-24 10:38:22 +08:00
Wenkai Yin 294385c34d Refactor the replication execution 
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-11-23 14:24:10 +08:00
stonezdj e667121a34 Add more registry type to proxy cache 
Includes: azure-acr, aws-ecr, google-gcr
Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-11-18 10:38:07 +08:00
Will Sun eca3de3489
Merge pull request #13494 from dirkmueller/lock_json_include 
Include package.json/package-lock.json in portal image
 2020-11-16 16:38:02 +08:00
stonezdj(Daojun Zhang) fb549b2d9e
Merge pull request #13444 from wy65701436/robot2-self-mgr 
add robot mgr
 2020-11-16 11:33:33 +08:00
He Weiwei 83c07d6680
fix: ensure the role_id of role is correct (#13476) 
Closes #13317

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-11-12 15:34:04 +08:00
Dirk Mueller 12adc63a48 Include package.json/package-lock.json in portal image 
This allows Trivy and other vulnerability scanners to correctly
determine the embedded dependencies in minified harbor-portal image.

Also simplify build process by reducing the number of layers in the
final stage container image

Signed-off-by: Dirk Mueller <dirk@dmllr.de>
 2020-11-11 21:21:28 +01:00
Wang Yan 3550b5e5e9 add robot mgr 
the robot account manager to handle the CRUD

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-11-11 13:47:03 +08:00
Wang Yan 9723655378 update code per review comments 
1, rename table name to permission_policy
2, rename functions name

Signed-off-by: Wang Yan <wangyan@vmware.com>
 2020-11-10 18:11:31 +08:00
Wang Yan ec15e320bf add role permission manager for robot enhancement 
1, add two db tables of role permission and rbac policy
2, add manager of these two tables

Signed-off-by: Wang Yan <wangyan@vmware.com>
 2020-11-10 16:49:29 +08:00
He Weiwei ebc3443da9
Merge pull request #13474 from heww/fix-issue-11892 
fix: compute artifact size from db for schema1 manifest
 2020-11-10 16:20:39 +08:00
DQ c10a6325d8 Add deprecated msg for clair 
Signed-off-by: DQ <dengq@vmware.com>
 2020-11-10 11:39:18 +08:00
DQ 0c9faea294 Clean up Clair in prepare script 
Signed-off-by: DQ <dengq@vmware.com>
 2020-11-10 11:39:18 +08:00
DQ 8a584aff89 Clean up clair and clair-adapter in build scripts 
1. Makefles
  2. Dockerfiles
  3. Installation script
  4. harbor.yml template

Signed-off-by: DQ <dengq@vmware.com>
 2020-11-10 11:39:18 +08:00
He Weiwei 9c8377909b fix: compute artifact size from db for schema1 manifest 
Closes #11892

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-11-09 12:32:07 +00:00
DQ 9152521b11 Fix: log container password expire 
move chage command to base image

Signed-off-by: DQ <dengq@vmware.com>
 2020-11-09 18:29:41 +08:00
DQ eb470501be Add metrics to Harbor Core 
1. Add configs in prepare
 2. Add models and config items in Core
 3. Encapdulate getting metric in commom package
 4. Add a middleware for global request to collect 3 metrics

Signed-off-by: DQ <dengq@vmware.com>
 2020-11-03 14:33:10 +08:00
Daniel Jiang fb687aeef8 Use pkg/token to generate JWT token 
This commit refactors the approach to encode a token in handler of /service/token,
by reusing pkg/token to avoid inconsistency.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-10-15 16:16:44 +08:00
DQ 184e89365b Fix internal tls config upgrade issue 
internal tls config upgrade is not included in template, this pr is to add it.

Signed-off-by: DQ <dengq@vmware.com>
 2020-09-25 09:54:31 +08:00
Wenkai Yin(尹文开) 8b9727f53f
Support store the cron type in the schedule (#13097) 
There is requirement that show the cron type(daily, weekly, etc.) on the UI, this commit adds the support for storing the cron type in the schedule model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-09-24 16:48:56 +08:00
DQ 17f3bfccb4 Fix trivy setting in upgrading script 
Signed-off-by: DQ <dengq@vmware.com>
 2020-09-08 18:15:57 +08:00
Daniel Jiang 1b8bec3994
Merge pull request #12896 from wy65701436/fixes-12889 
fix migration issue
 2020-08-28 14:16:21 +08:00
He Weiwei 687043c298
Merge pull request #12880 from stefannica/use-exit-in-db-entrypoint 
Use exec in harbor database entrypoint
 2020-08-28 10:09:58 +08:00
Daniel Jiang 91e2779822 Fill in the icon of known artifacts in artifact controller 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-08-28 01:33:26 +08:00
wang yan 84094e7a5d fix migration issue 
fixes #12889

Before the migration script to fix the nativate repo_id issue, is has to remove the duplicate tags
from the tag table, which may caused by user in v2.0.2 to retag & repush the missing image.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-27 19:04:39 +08:00
Daniel Jiang 7b42defb9a Make the 2.1.0 migration SQL script idempotent 
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-08-26 16:50:25 +08:00
Daniel Jiang 37e0aa0798
Merge pull request #12873 from wy65701436/fixes-12827 
fix db migration issue
 2020-08-26 14:42:24 +08:00
wang yan 9822e5bb15 fix db migration issue 
fixes #12827
 After user migrates Harbor from v2.0.2, user got 404 when to pull specific images, and no work after push the same images again.

 Fix:
 1, If the issue is caused by missing repository data, this fix can revert the missing repository data and all things should be fine.
 2, If the issue is caused by missing blob data, this fix can revert the missing repository data and still left the media type of artifact
    as 'UNKNOWN', which leads the meta data and build history of the image cannot be shown in UI. User can delete and push the image again to
    resolve it.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-26 12:02:20 +08:00
Ziming Zhang ff19dd499c fix(jobservice) redis sentinel failover hang 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-08-26 10:42:44 +08:00
Stefan Nica 1c768d0bf1 Use exec in harbor database entrypoint 
The harbor-db pod takes a long time to terminate. Using an `exec`
command in the entrypoint ensures that Unix signals reach the
postgres process [1].

[1] https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example

Signed-off-by: Stefan Nica <snica@suse.com>
 2020-08-25 20:24:52 +02:00
Wang Yan ad47d2f444
fix upgrade issue (#12857) 
fixes #12849

1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.

2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-08-24 16:08:15 +08:00
Daniel Jiang c0602b5fb3
Merge pull request #12832 from ywk253100/200820_data 
Add id column to data_migration table
 2020-08-21 19:30:05 +08:00
Daniel Jiang 4f812f7926
Merge pull request #12811 from ninjadq/fix_portal_health_check 
Fix schema of the portal health check
 2020-08-21 13:44:47 +08:00
Ted Guan 645dea36a6
Fix for duplicate webhook policy name (#12729) 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-08-20 18:02:13 +08:00
Wenkai Yin 975ef193dd Add id column to data_migration table 
Add id column to data_migration table and add logic to make sure there is only one data version record

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-20 17:43:15 +08:00
Dirk Mueller 08a4d8efd2
Update to golang 1.14.7 (#12809) 
We should use a golang that isn't having security issues.

This includes:
* go1.14.6 (released 2020/07/16) includes fixes to the go command, the
compiler, the linker, vet, and the database/sql, encoding/json,
net/http, reflect, and testing packages. See the Go 1.14.6 milestone on
our issue tracker for details.

* go1.14.7 (released 2020/08/06) includes security fixes to the
encoding/binary package. See the Go 1.14.7 milestone on our issue
tracker for details (CVE-2020-16845)

Signed-off-by: Dirk Mueller <dirk@dmllr.de>
Signed-off-by: Dirk Mueller <dmueller@suse.com>
 2020-08-20 15:38:35 +08:00
DQ e9323ca268 Fix schema of the portal health check 
it should be https

Signed-off-by: DQ <dengq@vmware.com>
 2020-08-19 15:58:51 +08:00
Wenkai Yin 0fd230c2d6 Refresh the status of execution for every status changing of task 
Refresh the status of execution for every status changing of task to support filtering executions by status directly

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-17 17:38:55 +08:00
Wenkai Yin b1ddb5e2cc Implement the icon API to get the icon of artifact 
Implement the icon API to get the icon of artifact

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-15 08:40:38 +08:00
Wenkai Yin cca1dcca51 Use a separated database table to store the data version 
Use a separated database table to store the data version.
Fixes #12747

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-14 11:38:13 +08:00
Qian Deng 5dbbfa76d3
Merge pull request #12766 from ninjadq/add_log_dependency_to_trivy 
Add log denpendency ti trivy
 2020-08-13 18:23:09 +08:00
Qian Deng 85fa6654ec
Fix: Add privileged for prepare command (#12689) 
Mount `/` dir in container require privilege
And this change will make `z` label useless. So remove them

Signed-off-by: DQ <dengq@vmware.com>
 2020-08-13 14:55:42 +08:00
Qian Deng 78d4b54ddc
Merge pull request #12765 from ninjadq/fix_trivy_append_in_2_1_0_config 
Fix: append trivy every time when run migrate
 2020-08-13 14:47:54 +08:00
DQ a251e90507 Add log denpendency ti trivy 
To void trivy can not start issue

Signed-off-by: DQ <dengq@vmware.com>
 2020-08-13 11:35:21 +08:00
DQ 7ba498be5b Fix: append trivy every time run migrate 
Signed-off-by: DQ <dengq@vmware.com>
 2020-08-11 17:43:25 +08:00
Yiyang Huang b98dc97fbd feat: enhanced default processor 
Signed-off-by: Yiyang Huang <huangyiyang.huangyy@bytedance.com>
 2020-08-11 01:31:02 +08:00
He Weiwei 8f036c765a chore(images): install shadow package in base images 
The latest `photon:2.0` does not include `groupadd` and `useradd`
we need to install `shadow` package which includes these commands.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-08-10 10:23:48 +00:00
Wenkai Yin(尹文开) d599cd98bf
Merge pull request #10455 from chlins/fix/quay-replication-adapter-refactor 
fix(replication): refactor quay adapter to fix authorization and supp…
 2020-08-10 16:37:19 +08:00
Wenkai Yin(尹文开) e8f9fb63c0
Merge pull request #12674 from reasonerjt/standalone-db-migrator 
Provide a standalone migrator to migrate DB schema.
 2020-08-10 15:11:52 +08:00
Daniel Pacak 9397dff093
docs: Explain how to use Trivy in offline mode (#12102) 
Resolves: #11985

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-08-10 08:48:04 +02:00
chlins b765cfe0ce fix(replication): refactor quay adapter to fix authorization and support quay.io and enterprise quay (#10317) 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-08-08 13:17:01 +08:00
Tianon Gravi 4752cac051 Remove unused "sudo" package from most images 
Notably missing is the "log" image, which still uses sudo.

Signed-off-by: Tianon Gravi <tianon@infosiftr.com>
 2020-08-06 12:44:06 -07:00
Daniel Jiang 4f94f59d2a Provide a standalone migrator to migrate DB schema. 
Fixes #11885
This part will not by default be packaged into release.
A README.md will be added in another commit.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-08-06 18:57:55 +08:00
Qian Deng 26dc5d1b15
Merge pull request #12557 from ninjadq/rm_expose_port 
Remove expose port in dockerfiles
 2020-08-06 14:29:51 +08:00
Wenkai Yin d6288a43e8 Do some refine for the scheduler 
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-08-05 17:43:18 +08:00
DQ b015440074 Remove expose port in dockerfiles 
The export is dynamical now because of introduce of internal TLS

Signed-off-by: DQ <dengq@vmware.com>
 2020-08-05 10:42:46 +08:00
Qian Deng fbef7fd088
Merge pull request #12651 from ninjadq/add_migration_2_1_0 
Add migration 2.1.0
 2020-08-03 15:59:28 +08:00
DQ 1e32792dc5 Add migration 2.1.0 
db_max_open_comms should be 1000 if its value between 100 and 1000

Signed-off-by: DQ <dengq@vmware.com>
 2020-08-03 15:17:41 +08:00
DQ d3ab9d7c6b Add internal tls configs for portal 
add related file, config, command to enabled https for portal

Signed-off-by: DQ <dengq@vmware.com>
 2020-07-31 12:10:47 +08:00
Qian Deng a2112bfa40
Merge pull request #12539 from ninjadq/core_config_port 
Fix: beego app config port hardcode
 2020-07-27 17:21:18 +08:00
DQ d7618a6274 Fix: beego app config port hardcode 
the port should be flexible depend on the internal tls

Signed-off-by: DQ <dengq@vmware.com>
 2020-07-27 15:35:43 +08:00
Steven Zou ee35e1ecc6
Merge pull request #12507 from chlins/fix/preheat-update-instance 
fix(preheat): fix preheat handler PingInstance and UpdateInstance
 2020-07-20 17:45:24 +08:00
Steven Zou 46d7434d0b
Merge pull request #12473 from ywk253100/200706_scheduler 
Refactor the scheduler with the task manager mechanism
 2020-07-20 15:53:14 +08:00
chlins 78927af032 fix(preheat): fix preheat handler PingInstance and UpdateInstance 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-20 15:42:24 +08:00
Wenkai Yin 4dc4b6728c Refactor the scheduler with the task manager mechanism 
Refactor the scheduler with the task manager mechanism, this will reduce the duplicate code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-20 14:03:15 +08:00
Ziming Zhang 8857e89e40 feature(redis) support redis sentinel 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-07-19 21:19:03 +08:00
Wang Yan bad8f026fc
upgrade golang to v1.14.5 (#12489) 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-07-16 16:20:54 +08:00
Qian Deng bd26c294e8
Merge pull request #12341 from ninjadq/support_multi_down_version 
Enhance: Support multi downversion in migration
 2020-07-15 23:39:11 +08:00
Daniel Jiang 947eadaa72
Merge pull request #12440 from heww/remove-init-clair-db 
refactor: remove initialization of clair db
 2020-07-15 00:38:12 +08:00
He Weiwei c000608d55
Merge pull request #12437 from heww/db-max-connections 
chore(db): change max_connections of postgres to 1024
 2020-07-14 17:24:16 +08:00
He Weiwei 2a6fe801bc chore(db): change max_connections of postgres to 1024 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-14 07:34:37 +00:00
Daniel Jiang e96165412d
Merge pull request #12432 from ywk253100/200709_allowlist 
Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist"
 2020-07-13 16:42:43 +08:00
chlins 38d14dff30 fix(preheat): validate instance/policy name, set unique filed and policy 
manager adds parsePolicy

Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-07-10 10:29:47 +08:00
He Weiwei 039aef5356 refactor: remove initialization of clair db 
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-07-09 15:26:14 +00:00
Wenkai Yin cd6c1b8c31 Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist" 
Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist" in project metadata

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-09 11:04:44 +08:00
Daniel Jiang 1637e6a588 Rename master role to maintainer 
This commit rename the var name, text appearance, and swagger of "master" role
to "maintainer" role.
It only covers backend code.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-07-08 09:20:07 +08:00
DQ 4617e0ff38 Enhance: Support multi downversion in migration 
1. Change down version to list to accept multi verstion value
2. Update search function use BFS to find migration path
2. Add test case

Signed-off-by: DQ <dengq@vmware.com>
 2020-07-07 21:36:58 +08:00
stonezdj(Daojun Zhang) 6f4e8150d5
Merge pull request #12383 from ywk253100/200702_registry_api 
Suport filtering registries by type in listing registry API
 2020-07-07 14:21:54 +08:00
Wenkai Yin 02690d1d04 Suport filtering registries by type in listing registry API 
Suport filtering registries by type in listing registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-07-07 10:30:46 +08:00
fanjiankong a0c2d0ac9e feat(preheat):add preheat api, controller and manager 
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch

Signed-off-by: fanjiankong <fanjiankong@tencent.com>
 2020-07-03 11:25:42 +08:00
Wenkai Yin(尹文开) 1d03b8727a
Merge pull request #12357 from ninjadq/add_env_for_aws 
Fix Amazon S3 storage not work
 2020-07-01 11:10:47 +08:00
chlins 15e4361d6e feat: add p2p preheat policy dao and manager(#12286) 
Signed-off-by: chlins <chlins.zhang@gmail.com>
 2020-06-30 15:56:50 +08:00
DQ d0ddd61ad9 Fix Amazon S3 storage not work 
The Chartmuseum S3 client need set an Env variable
Ref: https://github.com/helm/chartmuseum/issues/280

Signed-off-by: DQ <dengq@vmware.com>
 2020-06-30 15:16:18 +08:00
He Weiwei 0474a2a040
Merge pull request #12322 from heww/install-tls-ca 
feat(certs): install internal tls ca from /etc/harbor/ssl dir
 2020-06-25 21:03:35 +08:00
He Weiwei 13436b75a6 feat(certs): install internal tls ca from /etc/harbor/ssl dir 
Closes #10222

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-06-24 08:58:08 +00:00
AllForNothing fff6f7529a Replace all whitelist with allowlist 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-06-24 16:17:17 +08:00
Wang Yan 53044da28f
update blob controller & manager (#12101) 
* update blob controller & manager

1, add two more attributes, version, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-23 17:11:54 +08:00
wang yan c10467eb36 continue refactor 
Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-23 13:10:57 +08:00
Wang Yan de504993ad update blob controller & manager 
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-23 13:10:57 +08:00
Max Rosin 34d5591b1b Fix DOCKERIMASES and SWAAGER_IMAGE_BUILD_CMD typos in Makefiles 
Fix #12259

Signed-off-by: Max Rosin <git@hackrid.de>
 2020-06-16 12:18:55 +02:00
Wenkai Yin 127988b70c Define the task manager interface and data model 
Define the task manager interface and data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-06-15 18:42:09 +08:00
Wenkai Yin a79bb127b3 Update creating project API to support proxy cache project 
Update creating project API to support proxy cache project

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-06-10 17:14:12 +08:00
Wang Yan dec8397c21
Add api to delete blob and manifest (#12006) 
* Add api to delete blob and manifest

Enable the capability of registry controller to delete blob and manifest

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-06-06 01:34:23 +08:00
Qian Deng 9e1302211b
Merge pull request #12072 from ninjadq/add_timeout_in_nginx_config 
Add timeout in nginx config
 2020-06-02 15:14:42 +08:00
Steven Zou c7c1742b88
Merge pull request #12106 from heww/clean-clair-url 
refactor(configuration): cleanup unneeded CLAIR_URL configuration in core
 2020-06-01 19:24:19 +08:00
Daniel Jiang 58894e9d9c
Merge pull request #12071 from ninjadq/upgrade_chartversion 
Enhance: Upgrade chartmuseum version
 2020-06-01 13:36:54 +08:00
Daniel Jiang 6271da471b
Update health check script for harbor-db (#12103) 
This patch remove the trailing space of the hostname introduced by
`hostname -i`.

The trailing space will cause resolution error after this patch is
applied to glibc in photon:
https://github.com/vmware/photon/blob/2.0/SPECS/glibc/glibc-fix-CVE-2019-10739.patch

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-05-30 14:05:39 +08:00
He Weiwei d97be71234 refactor(configuration): cleanup unneeded CLAIR_URL configuration in core 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-05-29 07:27:50 +00:00
DQ 278338e401 Add timount on nginx configs 
set timeout to 900

Signed-off-by: DQ <dengq@vmware.com>
 2020-05-26 16:18:35 +08:00
DQ 715685ae51 Remove tls1.1 in notary 
Signed-off-by: DQ <dengq@vmware.com>
 2020-05-26 16:11:57 +08:00
DQ f7ffd991cc Enhance: Upgrade chartmuseum version 
Upgrade chartmuseum version 0.12.0

Signed-off-by: DQ <dengq@vmware.com>
 2020-05-26 15:59:58 +08:00
AllForNothing 90e34e0104 Improve i18n service 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-05-06 14:45:56 +08:00
DQ b06e19a637 Fix: GCS storage gc issue 
Mount gcs key to registryctl

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-29 15:04:16 +08:00
Daniel Jiang f91d7080d1
Merge pull request #11753 from tedgxt/2.0-webhook-event-types-fix 
Webhook data fix when updgrding to 2.0
 2020-04-28 19:36:44 +08:00
Qian Deng 9469252e85
Merge pull request #11745 from ninjadq/mount_ca_bundle 
Enhance: Create shared to store shared ca
 2020-04-28 10:19:26 +08:00
Qian Deng 7f1e3a7bb8
Merge pull request #11758 from ninjadq/output_subprocess_stdout 
Output subprocess stdout
 2020-04-28 09:46:02 +08:00
DQ f70339870a Enhance: Create shared to store shared ca 
this shared ca will mount to all harbor components

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-28 02:58:11 +08:00
guanxiatao f96cfab100 Table notification_policy fix when updgrding to 2.0 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-04-27 20:13:46 +08:00
Wang Yan add8dedc90
Fix the database upgrade issue (#11766) 
It's a workaround for issue https://github.com/goharbor/harbor/issues/11754

The phenomenon is the repository data is gone, but artifacts belong to the repository are still there.
To resolve it, just set the repository_id to a negative, and cannot duplicate.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-27 17:28:36 +08:00
DQ 90faf700f8 Enhance: output the stdout of gen cert script 
use popen replace check_all

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-27 10:43:22 +08:00
DQ 026e37e777 Fix chart museum absolute url issue 
if absolute url is enabled return true else set it to false

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-26 13:04:29 +08:00
DQ 599ca98c09 Hidden veriify client cert verfiy option 
Remove to avoid replication access core from external_url issue

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-23 10:14:36 +08:00
Daniel Jiang 2ecf0425a4 Remove the certs of notary signer 
Since `prepare` generates the certs as needed during installation, these
certs should not exist in the repo.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-04-21 13:20:19 +08:00
DQ b728f04d0a Fix tls min version for registry 
cert,key,mintls should in the same context

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-20 19:19:15 +08:00
Qian Deng 9c7caddeae
Merge pull request #11635 from hyy0322/set-root-password-never-expire 
fix: set root password never expire
 2020-04-16 22:05:10 +08:00
Maosheng Ren 89e9ea0145
Merge pull request #11636 from danielpacak/bump_up_trivy_adapter_to_0.9.0 
chore(trivy): Bump up trivy adapter to 0.9.0
 2020-04-16 16:16:50 +08:00
Wang Yan 790064df2e
fix notification policy ugrade issue (#11627) 
Fixes #11624

All of the existing policies created v1.10 has no name, it fails the upgrade process.
When to set the unique constraint for policy name, the empty can be seen as duplicated key.

ERROR:  could not create unique index "notification_policy_name_key"
DETAIL:  Key (name)=() is duplicated.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-16 14:53:58 +08:00
Daniel Pacak 5c3abee135 chore(trivy): Bump up trivy adapter to 0.9.0 
- Vendor the latest Trivy release 0.6.0
- Configure TLS 1.2 as min version when TLS is enabled
- Add more tracing to adapter config to facilitate troubleshooting

Resolves: #11544

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-04-16 08:40:27 +02:00
DQ 42c1095216 Fix cert issue of trivy 
Trivy can't access harbor from external if https enabled so inject cert to trivy container trust

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-16 10:52:03 +08:00
Yiyang Huang 4598f52057 fix: set root password never expire 
Signed-off-by: Yiyang Huang <huangyiyang@caicloud.io>
 2020-04-16 00:15:28 +08:00
He Weiwei 355c16943c chore(clair): bump up clair adapter version to 1.0.2 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-15 14:07:46 +00:00
He Weiwei 385aaac00d
Merge pull request #11620 from heww/fix-issue-11524 
feat(scanner): make Clair and Trivy as reserved name for scanners
 2020-04-15 15:21:35 +08:00
He Weiwei f5487479dd feat(scanner): make Clair and Trivy as reserved name for scanners 
Closes #11524

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-15 02:26:02 +00:00
Qian Deng 95d7c9382b
Merge pull request #11592 from ninjadq/min_version_tls_to_12 
Min version tls to 12
 2020-04-14 18:12:55 +08:00
wang yan ff2a6c7a01 add warning to registry binary name 
Fixes #11606

As we DO NOT want to user to execute GC in the container, rename it and append the warning message.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-14 15:16:50 +08:00
DQ 75f78b64b2 Set registry tls version to 1.2 
when internal tls enabled set min version of registry to 1.2

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-13 18:13:30 +08:00
jwangyangls e28b5811f7
Merge pull request #11176 from jwangyangls/change-helm-version 
Separate swagger to get v2.0 swagger and chart swagger
 2020-04-10 17:12:00 +08:00
Yogi_Wang 33ed4fb67e Separate swagger to get v2.0 swagger and chart swagger 
1. Partial helm api version number clear
2. Separate swagger to get v2.0 swagger and chart swagger
3. router add chart swagger

Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2020-04-10 16:25:30 +08:00
DQ e907cbe2b6 Fix health check for jobservice and regctl 
need cert when mTLS is enabled

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-09 20:35:46 +08:00
DQ 08ff622310 Remove lines not needed 
volume already defined above

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-09 20:06:51 +08:00
Ziming Zhang 572ebef685 feat(cicd) parameterize docker base image and external url 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-04-08 00:21:47 +08:00
DQ 6ae1b1dc97 Add missiong entrypoint file for trivy-adapter 
Signed-off-by: DQ <dengq@vmware.com>
 2020-04-07 10:39:07 +00:00
He Weiwei 3f567514b5
Merge pull request #11468 from wy65701436/remove-count-quota-code 
remove the chart handling in quota
 2020-04-07 16:51:07 +08:00
Daniel Jiang 5bcd015d6f
Merge pull request #11469 from ninjadq/clean_up_migrator 
Remove migrator flags in script
 2020-04-07 16:37:24 +08:00
Daniel Jiang e064bd4c01
Merge pull request #11428 from ninjadq/fix_container_unhealth 
Fix container unhealth
 2020-04-07 15:57:00 +08:00
wang yan a96d2f3746 remove the chart handling in quota 
1, remove the chartmuseum controller
2, doesn't handle chartrepo url in v2 middleware

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-07 15:26:34 +08:00
DQ 1ae50b8d66 Remove migrator flags in script 
Because migrator tool removed

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-07 14:57:10 +08:00
DQ 4a836ea975 Fix health check url 
health check url should depend on internal https

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-07 03:35:52 +00:00
wang yan 44825e819e deprecate quota count on artifact 
Fixes #11241

1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-04-06 16:56:11 +08:00
DQ cdb675bf3d Add proxy cert file to jobservice when https enabled 
jobservice may request via absolute path of url to harbor

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-04 17:44:34 +00:00
DQ 23ed189ed4 Add SAN to gencert script 
add localhost and 127.0.0.1 to SAN

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-04 17:44:34 +00:00
He Weiwei 77a8c3205f fix(prepare): not accpet items of false value in external_redis 
Item in yaml without value will be as None in python, which will make
the password of redis as `None` in `get_redis_configs`. This fix will
not accept items of `false value` in `external_redis` configurations.

Closes #11367

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-04-03 04:09:26 +00:00
Qian Deng a702c32346
Merge pull request #11063 from ninjadq/fix_syslog_dir_in_tpl 
Fix: fix logrotate is dir issue
 2020-04-02 11:37:29 +08:00
Qian Deng 0319baabcb
Merge pull request #11381 from ninjadq/enhance_migrate_config 
Enhance migrate config
 2020-04-02 10:00:38 +08:00
DQ dc271e1a87 Add packaging to pipenv 
Signed-off-by: DQ <dengq@vmware.com>
 2020-04-01 22:54:47 +08:00
DQ d636f2ea5c Enhance help message 
Provide more info in help message
Add requried opition and they will show missing option if you are not provide them instead of Exception

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-01 17:02:59 +08:00
DQ b2e1905e7a Enhance: Stop upgrade when input version less then 1.9.0 
The migration script should failure early when version is not supported

Signed-off-by: DQ <dengq@vmware.com>
 2020-04-01 15:35:49 +08:00
Ziming Zhang ae7834af0b feat(cicd) fix build base image 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-03-26 10:55:40 +08:00
Qian Deng 9e101b73a4
Merge pull request #11156 from ninjadq/migrate_config_to_harbor2 
Migrate config to harbor2
 2020-03-25 16:02:18 +08:00
DQ 85ec0e7820 Enhance: Refactor the migration structure 
1. Refactor structure of migrate file
2. fix some previous bugs

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-23 21:26:28 +08:00
DQ 444678fe07 Fix: module path raise exception when it is loop 
add test for loop

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-23 19:29:59 +08:00
Maosheng Ren 1dbec0c1d7
Fix a typo in the help message of install.sh (#11167) 
Signed-off-by: ren maosheng <stevenr@vmware.com>
 2020-03-23 10:30:37 +08:00
DQ e8bb977ae1 Feat: Upgrade configs to harbor 2.0 
add migrate files for harbor 2.0

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-20 15:20:32 +08:00
DQ 1e0c9f7231 Feat: Add config migrator to prepare 
deprecated migrator container and move config migration to prepare

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-20 03:04:10 +08:00
Steven Zou 2859cd8b69
Merge pull request #11134 from danielpacak/feat/issue_11090/trivy_skip_update_flag 
feat(trivy): Configure Trivy to skip database updates
 2020-03-19 18:13:08 +08:00
Wenkai Yin(尹文开) 9ebcf95758
Merge pull request #11122 from ywk253100/200318_replication_task 
Increase the length the columns (src_resource, dst_resource)of replication_task
 2020-03-19 12:16:27 +08:00
DQ f18a546429 Fix: return error when internal_tls_not_provided 
When iinternal_tls is empty, prepare should works as usual

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-19 10:37:58 +08:00
Daniel Pacak 7325105714 feat(trivy): Configure Trivy to skip database updates 
Resolves: #11090

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-03-18 17:11:47 +01:00
DQ 6e8d44101f Enhance: User can generate cert by their own ca key pair 
User can put their ca key pair on internal cert dir and name them to `harbor_internal_ca.key` and `harbor_internal_ca.crt` we wil use them to generate other certs

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ b93092e012 Add tls for trivy 
Add trivy tls cert files
Add tivey tls env and config
enhance gencert

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ c954969bcd Add mTLS configs 
mTLS only enabled in jobservice and registryctl

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ c5d73e6a0c Add switch to https 
use switch to make decision whether mTLS or server TLS

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ 454382149f TLS update for chart, clairadapter, registry 
Remove trustca in chartmuseum
Remove trustca in registry
Add tls in clair-adapter

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ 03e11c63c7 Fix docker file with secure tls change 
Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:10 +08:00
DQ dcc6950af7 Feat: auto install ca in registry 
Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
DQ b852605193 Feat: enable mtls in harbor replication 
Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
DQ 40e67f3b14 Feat: Enable mtls for registry 
Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
DQ 07a1d51693 Feat: enable tls in registryctlAdd tls related code in registryctl 
Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
DQ da359f609f Feat: enable mtls in core 
add mtls related code in core

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
DQ a4855cca36 Feat: update prepare to support tls 
update makefile
add model for prepare
update jinja template for prepare

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-18 19:22:09 +08:00
Wang Yan b4e941e961
drop table access log in migration (#11118) 
Use the audit log instead, the access log table should be dropped after migration

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-03-18 19:04:38 +08:00
Wenkai Yin ac9658bc1e Increase the length the columns (src_resource, dst_resource)of replication_task 
Fixes #10786 by increaseing the length of src_resource and dst_resource to 256

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-03-18 17:05:32 +08:00
He Weiwei 7d20154db5
fix: remove old artifact model (#11112) 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2020-03-18 14:20:06 +08:00
Daniel Pacak 9c13116963 chore(trivy): Allow configuring HTTP(S) proxy 
Resolves: #11032

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-03-16 12:26:49 +01:00
Daniel Pacak 46fb43bc25 chore: Bump up Trivy adapter to v0.4.0 
Allows configuring SCANNER_TRIVY_GITHUB_TOKEN environment variable,
which is passed to trivy executable binary when it starts scanning
a given artifact.

This is to increase GitHub requests rate limit from 60 per hours
(for anonymous requests) to 5000 when Trivy download its
vulnerabilities database.

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-03-16 09:53:16 +01:00
DQ 1eeea6b888 Fix: fix logrotate is dir issue 
Change it to bind command

Signed-off-by: DQ <dengq@vmware.com>
 2020-03-13 14:58:45 +08:00
Wenkai Yin a4a1913598 Repair the count usage during the upgrading 
As the count quota is against artifact rather than tag in 2.0, the count usage should be recalculated

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-03-13 13:59:48 +08:00
Ted Guan 4ac31c6d46
Add API for query supported event types and notify types; Return policy name in last trigger info; Remove project_id unique constraint in table notification_policy (#11029) 
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
 2020-03-11 18:06:58 +08:00
Wenkai Yin(尹文开) 8452100148
Merge pull request #10942 from ywk253100/200305_reference 
Persistent the URLs and annotations of artifact references in database
 2020-03-11 16:20:18 +08:00
Wang Yan bd7940217a
upgrade golang version to v1.13.8 (#11006) 
The vesrion contains two security bug fix - CVE-2020-0601, CVE-2020-7919

More details, see the golang milestone:

https://github.com/golang/go/issues?q=milestone%3AGo1.13.8+label%3ACherryPickApproved

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-03-11 12:20:06 +08:00
Ziming Zhang 695a2559be feat(cicd) use unified version as tag name, clean more 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-03-09 17:13:28 +08:00
Ziming Zhang 200c352c35 feat(cicd) use unified version as tag name 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-03-09 15:30:03 +08:00
Daniel Jiang 6d89553c4d
Merge pull request #10937 from reasonerjt/csrf-2.0 
Update CSRF mechanism
 2020-03-09 12:31:08 +08:00
Wenkai Yin(尹文开) 75eb7a8c5a
Merge pull request #10955 from wy65701436/migrate-access 
add sql for migrating access log
 2020-03-09 10:00:08 +08:00
Daniel Jiang ae5ffce83a Update CSRF mechanism 
This commit replaces beego's CSRF mechanism with gorilla's csrf library.
The criteria for requests to skip the csrf check remain the same.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-03-09 01:15:54 +08:00
wang yan 288c7790d0 add sql for migrating access log 
1, loop each access log, change to resource/resource_type, and insert into audit log
2, loop each first push operation, change it to create repository and insert into audit log.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-03-06 12:06:12 +08:00
wang yan 2b0b7576b2 Fix gc issue on clean the artifact trash 
1, enable dao test for artifact trash
2, set default flush trash table to false
3, hanlder empty parameter in API call
4, add registry auth info into jobservice container

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-03-06 03:11:31 +08:00
stonezdj(Daojun Zhang) 49619e1907
Merge pull request #10939 from wy65701436/access-log-mgr 
add audit logs API
 2020-03-05 16:24:21 +08:00
wang yan df237a5b17 add audit logs API 
1, add API entry for get audit logs
2. add audit log manager to hanlder CRUD

Use the new format of audit log to cover differernt resource, artifact/tag/repostory/project

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-03-05 11:40:51 +08:00
Wenkai Yin 76c04b0219 Persistent the URLs and annotations of artifact references in database 
Persistent the URLs and annotations of artifact references in database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-03-05 10:54:45 +08:00
Will Sun a5d9a3b65d
Merge pull request #10863 from AllForNothing/api-center 
Fix Api cennter
 2020-03-05 10:00:15 +08:00
Wenkai Yin(尹文开) 4fa4c4e74c
Merge pull request #10815 from cd1989/redis-idle-timeout 
Set redis idle timeout for core
 2020-03-03 14:10:22 +08:00
Wenkai Yin 4c9b59c904 Migrate artifact data in 2.0 
Abstract extra attributes and annotations for artifacts stored in database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-02-28 18:09:02 +08:00
jwangyangls 572510c82d
Merge pull request #10755 from Snaacker/master 
Fix typo
 2020-02-28 11:59:51 +08:00
Daniel Jiang 1823c984f7
Merge branch 'master' into redis-idle-timeout 2020-02-27 22:01:22 +08:00
AllForNothing d41c5496a2 Fix Api cennter 
Signed-off-by: AllForNothing <sshijun@vmware.com>
 2020-02-27 15:55:20 +08:00
Wenkai Yin 02c2647e1e Use the repository name of artifact model 
As we store the repository name in the artifact table, we can use it direclty in the code to reduce the database query

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-02-26 13:37:09 +08:00
stonezdj(Daojun Zhang) a7e5873f46
Merge pull request #10821 from stonezdj/20200224_remove_notification 
Remove registry notification and change core health check url
 2020-02-25 13:34:37 +08:00
Ziming Zhang 94230b5e19 feat(cicd) fix some build problem 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-02-25 12:05:39 +08:00
stonezdj 6005101c95 Remove registry notification and change /api/ping 
Update config.yaml.jinja to remove notification
Change api/ping in core/Dockerfile

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2020-02-25 11:24:21 +08:00
Wang Yan 948d45604c Revise the GC job flow, 
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-02-24 18:29:55 +08:00
Wenkai Yin bd204464f3 Remove dead code 
Remove dead code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-02-23 17:11:46 +08:00
dechen e642a73280 Set redis idle timeout for core 
Signed-off-by: dechen <xxyydream@gmail.com>
 2020-02-23 12:31:56 +08:00
Wenkai Yin 9312b788dc Upgrade the artifact table 
Split the table artifact into artifact and tags, and populate related data

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-02-21 20:37:31 +08:00
Wang Yan 22021a988e
Merge pull request #10753 from wy65701436/artifact-trash 
add artifact trash manager
 2020-02-19 19:14:48 +08:00
Steven Zou f1374737f6
Merge pull request #10694 from danielpacak/feature/install_with_trivy 
chore(install): Add --with-trivy arg to the installation script
 2020-02-19 16:27:57 +08:00
Wang Yan f160505686 add artifact trash manager 
1, move the deleted artifact into trash
2, disable GC to delete the untagged manifest

Signed-off-by: wang yan <wangyan@vmware.com>
 2020-02-19 14:52:58 +08:00
Tran Huy 2aed96851e Fix typo 
Signed-off-by: Tran Huy <Tran.Huy@ericsson.com>
 2020-02-18 13:26:59 +01:00
Wenkai Yin(尹文开) 341cb88cba
Merge pull request #10727 from ywk253100/200214_bump_up_legacy_api_version 
Bump up legacy api version to v2.0
 2020-02-18 10:37:25 +08:00
Wenkai Yin(尹文开) d7903fcf1b
Merge pull request #10682 from ywk253100/200126_label 
Support add/remove label to/from artifact
 2020-02-17 11:33:45 +08:00
Daniel Pacak 1b60bb255c refactor(Makefile): Add variables for download URLs 
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-02-14 11:58:59 +01:00
Wenkai Yin 94787ea60d Bump up the version of legacy APIs to v2.0 
Bump up the version of legacy APIs to v2.0

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-02-14 13:16:30 +08:00
Daniel Pacak 70dda1387a chore: Configure Redis URL for Trivy adapter 
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-02-13 17:57:02 +01:00
Daniel Pacak 4755439b75 chore: Build Trivy adapter from sources 
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-02-13 15:32:57 +01:00
Wenkai Yin 93731eeb2e Support add/remove label to/from artifact 
This commit add supporting for adding/removing label to/from artifacts and populates labels when listing artifacts

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-02-13 10:46:23 +08:00
Daniel Pacak a642667ffc chore(install): Add --with-trivy arg to the installation script 
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
 2020-02-12 23:47:56 +01:00
Ziming Zhang 6047a8102a fix(feat) fix prepare version in installer 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2020-02-06 10:58:29 +08:00
Will Sun acfcd2d175
Merge pull request #10489 from AllForNothing/postinstall 
Fix postinstall script in Docker
 2020-02-03 14:13:12 +08:00
Daniel Jiang 2064a1cd6d Switch to basic authentication for registry 
1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth.  #10604 is opened
to track the follow up work.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2020-01-31 21:46:47 +09:00
Wenkai Yin 7dc28bcff9 Add foreign key to avoid the concurrent issue 
Add foreign key to avoid the concurrent issue when operating the artifacts, tags and references

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2020-01-21 16:03:51 +08:00
sshijun 3175d5f646 Fix postinstall script in Docker 
Signed-off-by: sshijun <sshijun@vmware.com>
 2020-01-15 16:28:57 +08:00
Daniel Jiang a087ba02e3 Populate basic auth information for registry 
This commit updates `prepare` and templates to populate the credential
for registry for basic authentication.

A temporary flag `registry_use_basic_auth` was added to avoid breakage.
It MUST be removed before the release.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-12-31 14:50:46 +08:00
Wenkai Yin(尹文开) 5f11ca4df6
Merge pull request #10248 from ywk253100/191213_controller 
Define the interface for artifact manager
 2019-12-24 17:44:18 +08:00
Wenkai Yin ac605db5da Define the controller/manager interface for artifact and tag 
1. Define the controller/manager interface for artifact and tag
    2. Provide a null implementation for artifact manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-12-24 17:12:55 +08:00
Ziming e32649adb4 enhance[cicd] introduce github action for CICD 
In order to replace travis.
Implement 5 CI jobs
- UTTEST
- APITEST_DB
- APITEST_LDAP
- OFFLINE
- UI_UT

Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-12-17 18:36:33 +08:00
Wenkai Yin 7b41c900a7 Create the models for OCI supporting 
This commits does some basic init work for supporting OCI:
1. Create the artifact and tag model
2. Create database tables

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-12-12 16:05:38 +08:00
Steven Zou 7bf9372f32 chore[api]:rename API folder to api 
- update swagger yaml file reference in `Makefile`
- update swagger yaml file reference in `README`
- update swagger yaml file reference in `docs/configure_swagger.md`
- update swagger yaml file reference in `make/photon/portal/Dockerfile`
- update swagger yaml file reference in `tests/swaggerchecker.sh`

Signed-off-by: Steven Zou <szou@vmware.com>
 2019-12-11 17:41:27 +08:00
Wang Yan 550d690997
Merge pull request #10135 from bitsf/upgrade_clair 
upgrade clair to v2.1.1
 2019-12-06 11:52:10 +08:00
Ziming 9cad403762 fix(build): npm install with special endpoint (#10168) 
Signed-off-by: Ziming Zhang <zziming@vmware.com>
Change-Id: Iaaf33310a2621d58cdc3b9d3359607a961fef05e
 2019-12-06 11:45:48 +08:00
Wang Yan 2a63382236
Merge pull request #10047 from bitsf/makefile_clean 
optimize the makefile process
 2019-12-05 19:03:19 +08:00
He Weiwei 4ea5c41553
chore(scanner): upgrade clair scanner to 1.0.1 (#10147) 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-12-05 17:52:37 +08:00
Ziming Zhang 332f88ec8c add make clean 
Change-Id: Ibe806972a19cd69bfd90be051cdc340c4d7c6afb
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-12-05 14:44:07 +08:00
Ziming Zhang 744ae62831 upgrade clair to v2.1.1 
Change-Id: Idb2ad0470a51666d75895d8c5e68d80a67e05276
Signed-off-by: Ziming Zhang <zziming@vmware.com>
 2019-12-05 14:27:26 +08:00
Wenkai Yin(尹文开) d145f4baf4
Merge pull request #10034 from ywk253100/191128_clean 
Clean up admiral-related code
 2019-12-04 17:33:31 +08:00
stonezdj(Daojun Zhang) 339c1d4cab
Merge pull request #10088 from reasonerjt/authproxy-cert-setting 
Support pinning to authproxy server's cert
 2019-12-04 14:03:27 +08:00
Daniel Jiang 7bb71db478
Merge pull request #10003 from ninjadq/migrator_miss_component_no_proxy 
Add default domainname for no_proxy
 2019-12-03 10:50:32 +08:00
Daniel Jiang 902598fabd Support pinning to authproxy server's cert 
This commit add an attribute to configurations, whose value is the
certificate of authproxy server.  When this attribute is set Harbor will
pin to this cert when connecting authproxy.
This value will also be part of the response of systemInfo API.

This commit will be cherrypicked to 1.10 and 1.9 branch.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-12-03 07:31:26 +08:00
Qian Deng e5f8c2d779
Merge pull request #10022 from ninjadq/fix_ca_bundle_path_join 
Fix ca bundle path join issue
 2019-12-02 11:31:23 +08:00
Wenkai Yin dd2bc0ecef Clean up admiral-related code 
Clean up admiral-related code as it's useless

Signed-off-by: Wenkai Yin <yinw@vmware.com>
 2019-11-28 17:28:54 +08:00
DQ 79344887b9 Fix ca bundle path join issue 
CA bundle name start with '/' will break the os path join

Signed-off-by: DQ <dengq@vmware.com>
 2019-11-27 18:37:45 +08:00
Will Sun a52b99e180
Merge branch 'master' into remove-lib 2019-11-27 17:44:30 +08:00
DQ ed6438cf69 Add default domainname for no_proxy 
All internal service and known internal hostname shuold add to no_proxy by default

Signed-off-by: DQ <dengq@vmware.com>
 2019-11-27 15:10:42 +08:00
sshijun c692f5c67e Move lib into src for better UI building 
Signed-off-by: sshijun <sshijun@vmware.com>
 2019-11-27 09:59:06 +08:00
wang yan 7b664f64f1 Bump up golang version to v1.13.4 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-11-26 19:18:45 +08:00
Wang Yan 60101c4ea1
Merge pull request #9964 from heww/fix-clair-updaters-disable 
fix(prepaire,clair): disable clair updaters when its interval is 0
 2019-11-22 13:38:34 +08:00
He Weiwei b8308f41a0 fix(prepaire,clair): disable clair updaters when its interval is 0 
Closes #9961

Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-11-22 03:31:20 +00:00
stonezdj 938168b8ad Failed to start harbor when proxy is set 
Fix #9614, all communication between internal components should bypass the proxy
Add chartmuseum, notary-server,clair-adapter to the no_proxy list in harbor.yml

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-11-22 11:22:51 +08:00
stonezdj(Daojun Zhang) 2b0ede5341
Merge pull request #9829 from reasonerjt/rm-k8s-install 
Remove scripts to deploy Harbor on k8s
 2019-11-19 15:53:49 +08:00
Wang Yan eab974419c
Merge pull request #9825 from stonezdj/bug_9681 
Avoid to create duplicated immutable tag rules in the same project
 2019-11-18 17:26:22 +08:00
stonezdj 15898f2069 Avoid to create duplicated immutable tag rules in the same project 
Fix #9681, add constraint on immutable_tag_rule and catch the error

Signed-off-by: stonezdj <stonezdj@gmail.com>
 2019-11-15 14:46:23 +08:00
Daniel Jiang 2fcd174e4b
Merge pull request #9828 from wy65701436/cii-docker-base 
add base images when to build harbor assets
 2019-11-15 14:24:11 +08:00
He Weiwei fe69a5df99 build(scanner-adapter): bump up clair adapter to v1.0.1-rc2 
Signed-off-by: He Weiwei <hweiwei@vmware.com>
 2019-11-13 02:35:21 +00:00
wang yan 47793e77e3 update base file name ane pass base version to build file 
Signed-off-by: wang yan <wangyan@vmware.com>
 2019-11-12 19:12:49 +08:00
Wang Yan 544cc98971 add base images when to build harbor assets 
* add base images when to build harbor assets

Signed-off-by: wang yan <wangyan@vmware.com>
 2019-11-12 15:38:51 +08:00
Daniel Jiang 39a22d4470 Remove scripts to deploy Harbor on k8s 
This commit removes scripts have been deprecated for a while and users have to use
helm chart to deploy Harbor on top of k8s cluster.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
 2019-11-11 19:23:22 +08:00
Yogi_Wang cddc1149f1 Modify the memory of nodejs used from 8192MB to 2048MB 
Signed-off-by: Yogi_Wang <yawang@vmware.com>
 2019-11-11 17:05:42 +08:00