Commit Graph

1062 Commits

Author SHA1 Message Date
Wenkai Yin
294385c34d Refactor the replication execution
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-11-23 14:24:10 +08:00
He Weiwei
83c07d6680
fix: ensure the role_id of role is correct (#13476)
Closes #13317

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-11-12 15:34:04 +08:00
Wang Yan
9723655378 update code per review comments
1, rename table name to permission_policy
2, rename functions name

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-11-10 18:11:31 +08:00
Wang Yan
ec15e320bf add role permission manager for robot enhancement
1, add two db tables of role permission and rbac policy
2, add manager of these two tables

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-11-10 16:49:29 +08:00
He Weiwei
ebc3443da9
Merge pull request #13474 from heww/fix-issue-11892
fix: compute artifact size from db for schema1 manifest
2020-11-10 16:20:39 +08:00
DQ
c10a6325d8 Add deprecated msg for clair
Signed-off-by: DQ <dengq@vmware.com>
2020-11-10 11:39:18 +08:00
DQ
0c9faea294 Clean up Clair in prepare script
Signed-off-by: DQ <dengq@vmware.com>
2020-11-10 11:39:18 +08:00
DQ
8a584aff89 Clean up clair and clair-adapter in build scripts
1. Makefles
  2. Dockerfiles
  3. Installation script
  4. harbor.yml template

Signed-off-by: DQ <dengq@vmware.com>
2020-11-10 11:39:18 +08:00
He Weiwei
9c8377909b fix: compute artifact size from db for schema1 manifest
Closes #11892

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-11-09 12:32:07 +00:00
DQ
9152521b11 Fix: log container password expire
move chage command to base image

Signed-off-by: DQ <dengq@vmware.com>
2020-11-09 18:29:41 +08:00
DQ
eb470501be Add metrics to Harbor Core
1. Add configs in prepare
 2. Add models and config items in Core
 3. Encapdulate getting metric in commom package
 4. Add a middleware for global request to collect 3 metrics

Signed-off-by: DQ <dengq@vmware.com>
2020-11-03 14:33:10 +08:00
Daniel Jiang
fb687aeef8 Use pkg/token to generate JWT token
This commit refactors the approach to encode a token in handler of /service/token,
by reusing pkg/token to avoid inconsistency.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-10-15 16:16:44 +08:00
DQ
184e89365b Fix internal tls config upgrade issue
internal tls config upgrade is not included in template, this pr is to add it.

Signed-off-by: DQ <dengq@vmware.com>
2020-09-25 09:54:31 +08:00
Wenkai Yin(尹文开)
8b9727f53f
Support store the cron type in the schedule (#13097)
There is requirement that show the cron type(daily, weekly, etc.) on the UI, this commit adds the support for storing the cron type in the schedule model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-09-24 16:48:56 +08:00
DQ
17f3bfccb4 Fix trivy setting in upgrading script
Signed-off-by: DQ <dengq@vmware.com>
2020-09-08 18:15:57 +08:00
Daniel Jiang
1b8bec3994
Merge pull request #12896 from wy65701436/fixes-12889
fix migration issue
2020-08-28 14:16:21 +08:00
He Weiwei
687043c298
Merge pull request #12880 from stefannica/use-exit-in-db-entrypoint
Use exec in harbor database entrypoint
2020-08-28 10:09:58 +08:00
Daniel Jiang
91e2779822 Fill in the icon of known artifacts in artifact controller
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-28 01:33:26 +08:00
wang yan
84094e7a5d fix migration issue
fixes #12889

Before the migration script to fix the nativate repo_id issue, is has to remove the duplicate tags
from the tag table, which may caused by user in v2.0.2 to retag & repush the missing image.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-27 19:04:39 +08:00
Daniel Jiang
7b42defb9a Make the 2.1.0 migration SQL script idempotent
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-26 16:50:25 +08:00
Daniel Jiang
37e0aa0798
Merge pull request #12873 from wy65701436/fixes-12827
fix db migration issue
2020-08-26 14:42:24 +08:00
wang yan
9822e5bb15 fix db migration issue
fixes #12827
 After user migrates Harbor from v2.0.2, user got 404 when to pull specific images, and no work after push the same images again.

 Fix:
 1, If the issue is caused by missing repository data, this fix can revert the missing repository data and all things should be fine.
 2, If the issue is caused by missing blob data, this fix can revert the missing repository data and still left the media type of artifact
    as 'UNKNOWN', which leads the meta data and build history of the image cannot be shown in UI. User can delete and push the image again to
    resolve it.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-26 12:02:20 +08:00
Ziming Zhang
ff19dd499c fix(jobservice) redis sentinel failover hang
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-08-26 10:42:44 +08:00
Stefan Nica
1c768d0bf1 Use exec in harbor database entrypoint
The harbor-db pod takes a long time to terminate. Using an `exec`
command in the entrypoint ensures that Unix signals reach the
postgres process [1].

[1] https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example

Signed-off-by: Stefan Nica <snica@suse.com>
2020-08-25 20:24:52 +02:00
Wang Yan
ad47d2f444
fix upgrade issue (#12857)
fixes #12849

1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.

2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-24 16:08:15 +08:00
Daniel Jiang
c0602b5fb3
Merge pull request #12832 from ywk253100/200820_data
Add id column to data_migration table
2020-08-21 19:30:05 +08:00
Daniel Jiang
4f812f7926
Merge pull request #12811 from ninjadq/fix_portal_health_check
Fix schema of the portal health check
2020-08-21 13:44:47 +08:00
Ted Guan
645dea36a6
Fix for duplicate webhook policy name (#12729)
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-08-20 18:02:13 +08:00
Wenkai Yin
975ef193dd Add id column to data_migration table
Add id column to data_migration table and add logic to make sure there is only one data version record

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-20 17:43:15 +08:00
Dirk Mueller
08a4d8efd2
Update to golang 1.14.7 (#12809)
We should use a golang that isn't having security issues.

This includes:
* go1.14.6 (released 2020/07/16) includes fixes to the go command, the
compiler, the linker, vet, and the database/sql, encoding/json,
net/http, reflect, and testing packages. See the Go 1.14.6 milestone on
our issue tracker for details.

* go1.14.7 (released 2020/08/06) includes security fixes to the
encoding/binary package. See the Go 1.14.7 milestone on our issue
tracker for details (CVE-2020-16845)

Signed-off-by: Dirk Mueller <dirk@dmllr.de>
Signed-off-by: Dirk Mueller <dmueller@suse.com>
2020-08-20 15:38:35 +08:00
DQ
e9323ca268 Fix schema of the portal health check
it should be https

Signed-off-by: DQ <dengq@vmware.com>
2020-08-19 15:58:51 +08:00
Wenkai Yin
0fd230c2d6 Refresh the status of execution for every status changing of task
Refresh the status of execution for every status changing of task to support filtering executions by status directly

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-17 17:38:55 +08:00
Wenkai Yin
b1ddb5e2cc Implement the icon API to get the icon of artifact
Implement the icon API to get the icon of artifact

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-15 08:40:38 +08:00
Wenkai Yin
cca1dcca51 Use a separated database table to store the data version
Use a separated database table to store the data version.
Fixes #12747

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-14 11:38:13 +08:00
Qian Deng
5dbbfa76d3
Merge pull request #12766 from ninjadq/add_log_dependency_to_trivy
Add log denpendency ti trivy
2020-08-13 18:23:09 +08:00
Qian Deng
85fa6654ec
Fix: Add privileged for prepare command (#12689)
Mount `/` dir in container require privilege
And this change will make `z` label useless. So remove them

Signed-off-by: DQ <dengq@vmware.com>
2020-08-13 14:55:42 +08:00
Qian Deng
78d4b54ddc
Merge pull request #12765 from ninjadq/fix_trivy_append_in_2_1_0_config
Fix: append trivy every time when run migrate
2020-08-13 14:47:54 +08:00
DQ
a251e90507 Add log denpendency ti trivy
To void trivy can not start issue

Signed-off-by: DQ <dengq@vmware.com>
2020-08-13 11:35:21 +08:00
DQ
7ba498be5b Fix: append trivy every time run migrate
Signed-off-by: DQ <dengq@vmware.com>
2020-08-11 17:43:25 +08:00
Yiyang Huang
b98dc97fbd feat: enhanced default processor
Signed-off-by: Yiyang Huang <huangyiyang.huangyy@bytedance.com>
2020-08-11 01:31:02 +08:00
He Weiwei
8f036c765a chore(images): install shadow package in base images
The latest `photon:2.0` does not include `groupadd` and `useradd`
we need to install `shadow` package which includes these commands.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-10 10:23:48 +00:00
Wenkai Yin(尹文开)
d599cd98bf
Merge pull request #10455 from chlins/fix/quay-replication-adapter-refactor
fix(replication): refactor quay adapter to fix authorization and supp…
2020-08-10 16:37:19 +08:00
Wenkai Yin(尹文开)
e8f9fb63c0
Merge pull request #12674 from reasonerjt/standalone-db-migrator
Provide a standalone migrator to migrate DB schema.
2020-08-10 15:11:52 +08:00
Daniel Pacak
9397dff093
docs: Explain how to use Trivy in offline mode (#12102)
Resolves: #11985

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-08-10 08:48:04 +02:00
chlins
b765cfe0ce fix(replication): refactor quay adapter to fix authorization and support quay.io and enterprise quay (#10317)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2020-08-08 13:17:01 +08:00
Tianon Gravi
4752cac051 Remove unused "sudo" package from most images
Notably missing is the "log" image, which still uses sudo.

Signed-off-by: Tianon Gravi <tianon@infosiftr.com>
2020-08-06 12:44:06 -07:00
Daniel Jiang
4f94f59d2a Provide a standalone migrator to migrate DB schema.
Fixes #11885
This part will not by default be packaged into release.
A README.md will be added in another commit.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-06 18:57:55 +08:00
Qian Deng
26dc5d1b15
Merge pull request #12557 from ninjadq/rm_expose_port
Remove expose port in dockerfiles
2020-08-06 14:29:51 +08:00
Wenkai Yin
d6288a43e8 Do some refine for the scheduler
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-05 17:43:18 +08:00
DQ
b015440074 Remove expose port in dockerfiles
The export is dynamical now because of introduce of internal TLS

Signed-off-by: DQ <dengq@vmware.com>
2020-08-05 10:42:46 +08:00