Commit Graph

1144 Commits

Author SHA1 Message Date
Qian Deng
31138f12b0
Merge pull request #13806 from ninjadq/fix_python_yaml_load
Fix pythom yaml load to safe_load
2020-12-21 16:04:12 +08:00
Qian Deng
9197471e70
Add Scan for internal tls (#13810)
Signed-off-by: DQ <dengq@vmware.com>
2020-12-21 15:23:11 +08:00
Wang Yan
9bc6f3cee4
fix robot account update issue (#13741)
* fix robot account update issue

enable the update method to support both v1 & v2 robot update

Signed-off-by: Wang Yan <wangyan@vmware.com>

* resolve review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 20:01:26 +08:00
Wang Yan
6bc1047013
migration admin job data (#13766)
1, migrate gc and scan all schedule to schedule/task/exectuion
2, migrate gc history to task/execution

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 16:35:24 +08:00
Will Sun
4392a626f3
Merge pull request #13804 from AllForNothing/scan-all
Fix robot account UI issues
2020-12-18 15:48:26 +08:00
Qian Deng
64fcfeaa2f
Merge pull request #13754 from ninjadq/fix_loglevel_parsing_for_registry
Fix log level issue in registry
2020-12-18 14:47:15 +08:00
AllForNothing
b20cc474b3 Fix robot account UI issues
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-12-18 14:11:08 +08:00
DQ
234b29e170 Fix pythom yaml load to safe_load
Signed-off-by: DQ <dengq@vmware.com>
2020-12-16 14:59:06 +08:00
Daniel Jiang
b0c8cadad7
Add default CVE allowlist to project library (#13770)
fixes #12700

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-16 14:20:56 +08:00
Wenkai Yin
69808f033e Tiny fixes for task manager
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-12-14 17:26:32 +08:00
He Weiwei
08580f9fec
refactor(scan): refactor scan/scan all job to task manager (#13684)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 13:34:35 +08:00
DQ
19e8527cc1 Fix log level issue in registry
1. fix level issue in registry.jinja
2. add log level to registryctl

Signed-off-by: DQ <dengq@vmware.com>
2020-12-14 11:52:42 +08:00
Wenkai Yin(尹文开)
6569016d35
Merge pull request #13139 from wy65701436/migrate-gc
Migrate gc to task manager
2020-12-14 10:43:44 +08:00
Wang Yan
39e1a4f2b4 add extra attributes in the schedule table
Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-14 02:28:52 +08:00
wang yan
1bb79d402d update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 16:08:52 +08:00
Wang Yan
dba5522d0b Migrate to task manager (#129)
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 14:00:33 +08:00
DQ
d95f22448c Add cache for exporter
Add timed cache for exporter
default cache time is 30s, cleanup job run every 4 hours

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 21:22:40 +08:00
DQ
f0db193895 Add prepare file for exporter
prepare env for exporter

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 21:22:13 +08:00
DQ
dc0047c48c Add build script for exporter
- Add dockerfile
- update makefile

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 20:42:21 +08:00
Wang Yan
d2fa2e6b84
update robot secret (#13654)
* update robot secret

1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object

Signed-off-by: Wang Yan <wangyan@vmware.com>

* update robot secret

1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-03 18:13:06 +08:00
Wang Yan
732e9a21cd
updates on robot accounts (#13623)
* updates on robot accounts

1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account

Signed-off-by: wang yan <wangyan@vmware.com>

* update code per review comments

1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-01 18:31:34 +08:00
DQ
907904f480 Add DB Migration code for clair cleanning
- Delete clair scanner if exist
- Delete report is it is scanned by clair
- Set Trivy to Default if it exist and not default scanner

Signed-off-by: DQ <dengq@vmware.com>
2020-11-29 16:19:02 +08:00
DQ
590212b485 Remove clair related code
- clair code in harbor core
- clair code in frontend
- clair code in robotcase

Signed-off-by: DQ <dengq@vmware.com>
2020-11-27 14:01:04 +08:00
stonezdj(Daojun Zhang)
be4e6a5985
Merge pull request #13537 from stonezdj/201118_add_more_registry_type
Add more registry type to proxy cache
2020-11-26 11:16:16 +08:00
Ziming Zhang
d55f55aeb9 fix(chartmuseum) compatible s3 cache fail
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-11-25 17:00:16 +08:00
He Weiwei
eb38180483 fix(quota): ignore the fail when getting reference of quota
1. Clean the dirty data in quota/quota_usage.
2. Ignore the fail when getting the reference of quota.

Closes #13387

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-11-24 14:50:38 +00:00
Wenkai Yin(尹文开)
fe8b628f0c
Merge pull request #13437 from ywk253100/200929_replication
Refactor the replication execution
2020-11-24 10:38:22 +08:00
Wenkai Yin
294385c34d Refactor the replication execution
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-11-23 14:24:10 +08:00
stonezdj
e667121a34 Add more registry type to proxy cache
Includes: azure-acr, aws-ecr, google-gcr
Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-11-18 10:38:07 +08:00
Will Sun
eca3de3489
Merge pull request #13494 from dirkmueller/lock_json_include
Include package.json/package-lock.json in portal image
2020-11-16 16:38:02 +08:00
stonezdj(Daojun Zhang)
fb549b2d9e
Merge pull request #13444 from wy65701436/robot2-self-mgr
add robot mgr
2020-11-16 11:33:33 +08:00
He Weiwei
83c07d6680
fix: ensure the role_id of role is correct (#13476)
Closes #13317

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-11-12 15:34:04 +08:00
Dirk Mueller
12adc63a48 Include package.json/package-lock.json in portal image
This allows Trivy and other vulnerability scanners to correctly
determine the embedded dependencies in minified harbor-portal image.

Also simplify build process by reducing the number of layers in the
final stage container image

Signed-off-by: Dirk Mueller <dirk@dmllr.de>
2020-11-11 21:21:28 +01:00
Wang Yan
3550b5e5e9 add robot mgr
the robot account manager to handle the CRUD

Signed-off-by: wang yan <wangyan@vmware.com>
2020-11-11 13:47:03 +08:00
Wang Yan
9723655378 update code per review comments
1, rename table name to permission_policy
2, rename functions name

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-11-10 18:11:31 +08:00
Wang Yan
ec15e320bf add role permission manager for robot enhancement
1, add two db tables of role permission and rbac policy
2, add manager of these two tables

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-11-10 16:49:29 +08:00
He Weiwei
ebc3443da9
Merge pull request #13474 from heww/fix-issue-11892
fix: compute artifact size from db for schema1 manifest
2020-11-10 16:20:39 +08:00
DQ
c10a6325d8 Add deprecated msg for clair
Signed-off-by: DQ <dengq@vmware.com>
2020-11-10 11:39:18 +08:00
DQ
0c9faea294 Clean up Clair in prepare script
Signed-off-by: DQ <dengq@vmware.com>
2020-11-10 11:39:18 +08:00
DQ
8a584aff89 Clean up clair and clair-adapter in build scripts
1. Makefles
  2. Dockerfiles
  3. Installation script
  4. harbor.yml template

Signed-off-by: DQ <dengq@vmware.com>
2020-11-10 11:39:18 +08:00
He Weiwei
9c8377909b fix: compute artifact size from db for schema1 manifest
Closes #11892

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-11-09 12:32:07 +00:00
DQ
9152521b11 Fix: log container password expire
move chage command to base image

Signed-off-by: DQ <dengq@vmware.com>
2020-11-09 18:29:41 +08:00
DQ
eb470501be Add metrics to Harbor Core
1. Add configs in prepare
 2. Add models and config items in Core
 3. Encapdulate getting metric in commom package
 4. Add a middleware for global request to collect 3 metrics

Signed-off-by: DQ <dengq@vmware.com>
2020-11-03 14:33:10 +08:00
Daniel Jiang
fb687aeef8 Use pkg/token to generate JWT token
This commit refactors the approach to encode a token in handler of /service/token,
by reusing pkg/token to avoid inconsistency.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-10-15 16:16:44 +08:00
DQ
184e89365b Fix internal tls config upgrade issue
internal tls config upgrade is not included in template, this pr is to add it.

Signed-off-by: DQ <dengq@vmware.com>
2020-09-25 09:54:31 +08:00
Wenkai Yin(尹文开)
8b9727f53f
Support store the cron type in the schedule (#13097)
There is requirement that show the cron type(daily, weekly, etc.) on the UI, this commit adds the support for storing the cron type in the schedule model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-09-24 16:48:56 +08:00
DQ
17f3bfccb4 Fix trivy setting in upgrading script
Signed-off-by: DQ <dengq@vmware.com>
2020-09-08 18:15:57 +08:00
Daniel Jiang
1b8bec3994
Merge pull request #12896 from wy65701436/fixes-12889
fix migration issue
2020-08-28 14:16:21 +08:00
He Weiwei
687043c298
Merge pull request #12880 from stefannica/use-exit-in-db-entrypoint
Use exec in harbor database entrypoint
2020-08-28 10:09:58 +08:00
Daniel Jiang
91e2779822 Fill in the icon of known artifacts in artifact controller
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-28 01:33:26 +08:00
wang yan
84094e7a5d fix migration issue
fixes #12889

Before the migration script to fix the nativate repo_id issue, is has to remove the duplicate tags
from the tag table, which may caused by user in v2.0.2 to retag & repush the missing image.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-27 19:04:39 +08:00
Daniel Jiang
7b42defb9a Make the 2.1.0 migration SQL script idempotent
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-26 16:50:25 +08:00
Daniel Jiang
37e0aa0798
Merge pull request #12873 from wy65701436/fixes-12827
fix db migration issue
2020-08-26 14:42:24 +08:00
wang yan
9822e5bb15 fix db migration issue
fixes #12827
 After user migrates Harbor from v2.0.2, user got 404 when to pull specific images, and no work after push the same images again.

 Fix:
 1, If the issue is caused by missing repository data, this fix can revert the missing repository data and all things should be fine.
 2, If the issue is caused by missing blob data, this fix can revert the missing repository data and still left the media type of artifact
    as 'UNKNOWN', which leads the meta data and build history of the image cannot be shown in UI. User can delete and push the image again to
    resolve it.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-26 12:02:20 +08:00
Ziming Zhang
ff19dd499c fix(jobservice) redis sentinel failover hang
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-08-26 10:42:44 +08:00
Stefan Nica
1c768d0bf1 Use exec in harbor database entrypoint
The harbor-db pod takes a long time to terminate. Using an `exec`
command in the entrypoint ensures that Unix signals reach the
postgres process [1].

[1] https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example

Signed-off-by: Stefan Nica <snica@suse.com>
2020-08-25 20:24:52 +02:00
Wang Yan
ad47d2f444
fix upgrade issue (#12857)
fixes #12849

1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.

2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-24 16:08:15 +08:00
Daniel Jiang
c0602b5fb3
Merge pull request #12832 from ywk253100/200820_data
Add id column to data_migration table
2020-08-21 19:30:05 +08:00
Daniel Jiang
4f812f7926
Merge pull request #12811 from ninjadq/fix_portal_health_check
Fix schema of the portal health check
2020-08-21 13:44:47 +08:00
Ted Guan
645dea36a6
Fix for duplicate webhook policy name (#12729)
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-08-20 18:02:13 +08:00
Wenkai Yin
975ef193dd Add id column to data_migration table
Add id column to data_migration table and add logic to make sure there is only one data version record

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-20 17:43:15 +08:00
Dirk Mueller
08a4d8efd2
Update to golang 1.14.7 (#12809)
We should use a golang that isn't having security issues.

This includes:
* go1.14.6 (released 2020/07/16) includes fixes to the go command, the
compiler, the linker, vet, and the database/sql, encoding/json,
net/http, reflect, and testing packages. See the Go 1.14.6 milestone on
our issue tracker for details.

* go1.14.7 (released 2020/08/06) includes security fixes to the
encoding/binary package. See the Go 1.14.7 milestone on our issue
tracker for details (CVE-2020-16845)

Signed-off-by: Dirk Mueller <dirk@dmllr.de>
Signed-off-by: Dirk Mueller <dmueller@suse.com>
2020-08-20 15:38:35 +08:00
DQ
e9323ca268 Fix schema of the portal health check
it should be https

Signed-off-by: DQ <dengq@vmware.com>
2020-08-19 15:58:51 +08:00
Wenkai Yin
0fd230c2d6 Refresh the status of execution for every status changing of task
Refresh the status of execution for every status changing of task to support filtering executions by status directly

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-17 17:38:55 +08:00
Wenkai Yin
b1ddb5e2cc Implement the icon API to get the icon of artifact
Implement the icon API to get the icon of artifact

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-15 08:40:38 +08:00
Wenkai Yin
cca1dcca51 Use a separated database table to store the data version
Use a separated database table to store the data version.
Fixes #12747

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-14 11:38:13 +08:00
Qian Deng
5dbbfa76d3
Merge pull request #12766 from ninjadq/add_log_dependency_to_trivy
Add log denpendency ti trivy
2020-08-13 18:23:09 +08:00
Qian Deng
85fa6654ec
Fix: Add privileged for prepare command (#12689)
Mount `/` dir in container require privilege
And this change will make `z` label useless. So remove them

Signed-off-by: DQ <dengq@vmware.com>
2020-08-13 14:55:42 +08:00
Qian Deng
78d4b54ddc
Merge pull request #12765 from ninjadq/fix_trivy_append_in_2_1_0_config
Fix: append trivy every time when run migrate
2020-08-13 14:47:54 +08:00
DQ
a251e90507 Add log denpendency ti trivy
To void trivy can not start issue

Signed-off-by: DQ <dengq@vmware.com>
2020-08-13 11:35:21 +08:00
DQ
7ba498be5b Fix: append trivy every time run migrate
Signed-off-by: DQ <dengq@vmware.com>
2020-08-11 17:43:25 +08:00
Yiyang Huang
b98dc97fbd feat: enhanced default processor
Signed-off-by: Yiyang Huang <huangyiyang.huangyy@bytedance.com>
2020-08-11 01:31:02 +08:00
He Weiwei
8f036c765a chore(images): install shadow package in base images
The latest `photon:2.0` does not include `groupadd` and `useradd`
we need to install `shadow` package which includes these commands.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-10 10:23:48 +00:00
Wenkai Yin(尹文开)
d599cd98bf
Merge pull request #10455 from chlins/fix/quay-replication-adapter-refactor
fix(replication): refactor quay adapter to fix authorization and supp…
2020-08-10 16:37:19 +08:00
Wenkai Yin(尹文开)
e8f9fb63c0
Merge pull request #12674 from reasonerjt/standalone-db-migrator
Provide a standalone migrator to migrate DB schema.
2020-08-10 15:11:52 +08:00
Daniel Pacak
9397dff093
docs: Explain how to use Trivy in offline mode (#12102)
Resolves: #11985

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-08-10 08:48:04 +02:00
chlins
b765cfe0ce fix(replication): refactor quay adapter to fix authorization and support quay.io and enterprise quay (#10317)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-08-08 13:17:01 +08:00
Tianon Gravi
4752cac051 Remove unused "sudo" package from most images
Notably missing is the "log" image, which still uses sudo.

Signed-off-by: Tianon Gravi <tianon@infosiftr.com>
2020-08-06 12:44:06 -07:00
Daniel Jiang
4f94f59d2a Provide a standalone migrator to migrate DB schema.
Fixes #11885
This part will not by default be packaged into release.
A README.md will be added in another commit.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-06 18:57:55 +08:00
Qian Deng
26dc5d1b15
Merge pull request #12557 from ninjadq/rm_expose_port
Remove expose port in dockerfiles
2020-08-06 14:29:51 +08:00
Wenkai Yin
d6288a43e8 Do some refine for the scheduler
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-05 17:43:18 +08:00
DQ
b015440074 Remove expose port in dockerfiles
The export is dynamical now because of introduce of internal TLS

Signed-off-by: DQ <dengq@vmware.com>
2020-08-05 10:42:46 +08:00
Qian Deng
fbef7fd088
Merge pull request #12651 from ninjadq/add_migration_2_1_0
Add migration 2.1.0
2020-08-03 15:59:28 +08:00
DQ
1e32792dc5 Add migration 2.1.0
db_max_open_comms should be 1000 if its value between 100 and 1000

Signed-off-by: DQ <dengq@vmware.com>
2020-08-03 15:17:41 +08:00
DQ
d3ab9d7c6b Add internal tls configs for portal
add related file, config, command to enabled https for portal

Signed-off-by: DQ <dengq@vmware.com>
2020-07-31 12:10:47 +08:00
Qian Deng
a2112bfa40
Merge pull request #12539 from ninjadq/core_config_port
Fix: beego app config port hardcode
2020-07-27 17:21:18 +08:00
DQ
d7618a6274 Fix: beego app config port hardcode
the port should be flexible depend on the internal tls

Signed-off-by: DQ <dengq@vmware.com>
2020-07-27 15:35:43 +08:00
Steven Zou
ee35e1ecc6
Merge pull request #12507 from chlins/fix/preheat-update-instance
fix(preheat): fix preheat handler PingInstance and UpdateInstance
2020-07-20 17:45:24 +08:00
Steven Zou
46d7434d0b
Merge pull request #12473 from ywk253100/200706_scheduler
Refactor the scheduler with the task manager mechanism
2020-07-20 15:53:14 +08:00
chlins
78927af032 fix(preheat): fix preheat handler PingInstance and UpdateInstance
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-20 15:42:24 +08:00
Wenkai Yin
4dc4b6728c Refactor the scheduler with the task manager mechanism
Refactor the scheduler with the task manager mechanism, this will reduce the duplicate code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-20 14:03:15 +08:00
Ziming Zhang
8857e89e40 feature(redis) support redis sentinel
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-07-19 21:19:03 +08:00
Wang Yan
bad8f026fc
upgrade golang to v1.14.5 (#12489)
Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-16 16:20:54 +08:00
Qian Deng
bd26c294e8
Merge pull request #12341 from ninjadq/support_multi_down_version
Enhance: Support multi downversion in migration
2020-07-15 23:39:11 +08:00
Daniel Jiang
947eadaa72
Merge pull request #12440 from heww/remove-init-clair-db
refactor: remove initialization of clair db
2020-07-15 00:38:12 +08:00
He Weiwei
c000608d55
Merge pull request #12437 from heww/db-max-connections
chore(db): change max_connections of postgres to 1024
2020-07-14 17:24:16 +08:00
He Weiwei
2a6fe801bc chore(db): change max_connections of postgres to 1024
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-14 07:34:37 +00:00
Daniel Jiang
e96165412d
Merge pull request #12432 from ywk253100/200709_allowlist
Rename "reuse_sys_cve_whitelist" to "reuse_sys_cve_allowlist"
2020-07-13 16:42:43 +08:00
chlins
38d14dff30 fix(preheat): validate instance/policy name, set unique filed and policy
manager adds parsePolicy

Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-07-10 10:29:47 +08:00
He Weiwei
039aef5356 refactor: remove initialization of clair db
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-09 15:26:14 +00:00