Commit Graph

1225 Commits

Author SHA1 Message Date
danfengliu
c39345da96
Merge pull request #15148 from danfengliu/missing-db-base-login-in-build-base-workflow
Fix issue of missing db base build process in build base workflow
2021-06-16 16:39:56 +08:00
Will Sun
410c5bcd78
Undate Angular to the latest version (#15129)
Signed-off-by: AllForNothing <sshijun@vmware.com>
2021-06-16 16:39:10 +08:00
danfengliu
f367aad760 Fix issue of missing db base build process in build base workflow
There is extra build step for db base image building since v2.3, so this
step should be added back.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-06-16 15:22:32 +08:00
He Weiwei
72f1afe2cc
perf: configurations for perf improvement (#15142)
Closes #15041

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-06-15 18:39:15 +08:00
danfengliu
254c4a3497
Merge pull request #15046 from danfengliu/refine-base-build-process-in-makefile
Refind build base process in Makefile
2021-06-09 11:21:48 +08:00
He Weiwei
28921e2997 fix: listen to 5443 when clean unexpected status of db
Closes #15081

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-06-08 06:41:23 +00:00
DQ
68ac772726 Fix: Remove v6 format of harbor
Signed-off-by: DQ <dengq@vmware.com>
2021-06-07 16:26:53 +08:00
Qian Deng
0867a6bfd6
Merge pull request #15055 from ninjadq/health_check_url_2_dn_instead_of_ip
Add IPv6 support
2021-06-03 17:57:55 +08:00
DQ
1b6b47f860 Add IPv6 support
* 127.0.0.1 to localhost
* listening net addr add ipv6 format

Signed-off-by: DQ <dengq@vmware.com>
2021-06-03 09:04:49 +00:00
danfengliu
d66ea07c2f Refind build base process in Makefile
Remove build base executable in Makefile by replacing it as an input parameter.
Add add more input parameters for controlling docker pull/push to make
build base process flexible for users.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-06-03 14:48:34 +08:00
Wang Yan
eec9893918 set shm size of postgres
Fixed #15034, as for postgres 13, the default shm size is 64MB, set to 1gb to avoid could not resize shared memory segment error.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-06-03 11:05:33 +08:00
Wang Yan
ae06ac2fae
fix db issue on helm upgrade (#15028)
fixes #15000

It needs to double confirm that old pg is stopped before migration

Signed-off-by: wang yan <wangyan@vmware.com>
2021-06-01 15:59:41 +08:00
ChenYu Zhang
a038ba672f
perf: add 2.2.3 schema up sql to master (#15027)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2021-06-01 14:12:25 +08:00
Wang Yan
66b8a8f8dd
add build arch parameter in Makefile (#14995)
* add build arch parameter in Makefile

Add parameter BUILDARCH for make file. DB base builds pg96 for x86_64 only

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-06-01 10:38:05 +08:00
DQ
5d02acd043 Add upgrade script for harbor 2.3
no new config item added. harbor.yml keep the same as last version

Signed-off-by: DQ <dengq@vmware.com>
2021-05-28 20:30:12 +08:00
Wenkai Yin
c585f92034 Fix bug of migration sql for replication policy
Fix bug of migration sql for replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-26 18:57:23 +08:00
Qian Deng
9ea8aade01
Upgrade prepare to consistent with photon 4 (#14698)
* requires version to 3.9.1
* upgrade packages

Signed-off-by: DQ <dengq@vmware.com>
2021-05-26 16:39:04 +08:00
Wang Yan
39bdd7b506
pg upgrade failure handling (#14934)
To ensure the upgrade execution idempotence, it needs to clean the $PGDATANEW on pg_upgrade failure.
Otherwise, the upgrade will skip the upgrade process from the second time launch as the exist of $PGDATANEW.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-21 13:53:39 +08:00
danfengliu
6c14e699b1
Merge pull request #14844 from danfengliu/fix-notary-trust-directory-issue-in-nightly
Fix notary trust directory issue in nightly
2021-05-20 21:56:27 +08:00
Wang Yan
73bd373a75
create index for audit (#14930)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-20 19:50:13 +08:00
danfengliu
3ef4dc17fc Fix notary trust directory issue and add login for each base image in makefile
1. Use root instead of ~ in notary parameter;
2. Fix tag immutability issue caused by GUI change;
3. Replace email domain name to harbor test;
4. Add login for each base image in makefile;
5. Add customize look test in nightly.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-05-20 18:23:13 +08:00
Wang Yan
86185989cf
support pg upgrade (#14846)
1, use the pg source and photon spec to build postgres 9.6
2, install 9.6 on the photon 4.0
3, then leverage pg_upgrade to handle the pg major version migration

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-20 16:25:50 +08:00
Wenkai Yin
dc059a9a8f Improve the performance of artifact related APIs
Improve the performance of artifact related APIs by adding indexes and refactoring sql logic

Closes #13890 #14813 #14814

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-20 11:25:43 +08:00
Wang Yan
0fb520a33b bump up go to v1.15.12
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-14 15:36:23 +08:00
Qian Deng
17dd48e5a3
Merge pull request #14854 from ninjadq/upgrade_helm_chart
Upgrade helm chart
2021-05-12 19:33:03 +08:00
Wenkai Yin
c04f3a2aac Fix duplicate execution record issue
When the core service cannot response the checkin request in time, duplicated execution records may be created, this commit introduces the revision column to make sure there is only one record for one schedule trigger

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-12 15:40:52 +08:00
DQ
04ba4a4033 Upgrade chartmuseum version
from 1.12.0 to 1.13.1

Signed-off-by: DQ <dengq@vmware.com>
2021-05-11 13:51:55 +00:00
He Weiwei
476732df89 fix: change art_size to bigint in migrations
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-05-11 04:01:02 +00:00
Wenkai Yin(尹文开)
9bbffa06db
Merge pull request #14662 from ywk253100/210413_dest_ns
Refactor the replication policy destination namespace logic
2021-04-29 11:18:43 +08:00
Wenkai Yin
710c80078b Refactor the replication policy destination namespace logic
Support specifying what part of the repository will be replaced by the provided namespace

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-04-28 10:16:20 +08:00
Daniel Jiang
c701ce09fa
Merge pull request #14681 from bitsf/fix_typo_NOTARYURL
Fixed typo in NOTARYURL variable name
2021-04-21 17:38:01 +08:00
Wenkai Yin(尹文开)
0f6692f68f
Merge pull request #14682 from ninjadq/hostname_hardcode_to_localhost
Fix: Use local host on db's healthcheck
2021-04-19 21:02:31 +08:00
danfengliu
f0ebd17994 Add build base image step in build package git action workflow
Build base image step should be in build package workflow, and local base images build by new step should be removed since images have been pushed to docker hub.

Signed-off-by: danfengliu <danfengl@vmware.com>
2021-04-17 18:10:44 +08:00
Ziming Zhang
39f70287b4 Fixed typo in NOTARYURL variable name
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-04-17 15:55:02 +08:00
DQ
ffed6459c7 Fix: Use local host on db's healthcheck
hostname -i will malfunction in some cases like the `nsswitch.conf` file does'nt exist

Signed-off-by: DQ <dengq@vmware.com>
2021-04-16 18:37:24 +08:00
Pushkar Joglekar
3947c5faff Add --no-cache and --pull flag to image builds to ensure latest security fixes are pulled from base image
Signed-off-by: Pushkar Joglekar <pjoglekar@vmware.com>
2021-04-12 09:49:27 -07:00
Qian Deng
c5d12ce8ee
Merge pull request #14542 from ninjadq/add_task_info_in_exporter
Add task info in exporter
2021-04-07 18:17:26 +08:00
Alexis
06fa88cfb7 Fix typo
Signed-off-by: Alexis <60alexis@gmail.com>
2021-04-07 15:58:17 +08:00
Alexis
e33f7aa9dd Add redis port to 2.1.0 jinja template
Signed-off-by: Alexis <60alexis@gmail.com>
2021-04-07 15:58:17 +08:00
Alexis
7742aec4af Add port to 2.0.0 jinja file
Signed-off-by: Alexis <60alexis@gmail.com>
2021-04-07 15:58:17 +08:00
Alexis
d28845af51 Remove external_redis.port since not used since v1.10.0
Signed-off-by: Alexis <60alexis@gmail.com>
2021-04-07 15:58:17 +08:00
stonezdj(Daojun Zhang)
448f0b6e28
Merge pull request #14579 from stonezdj/21apr_add_docker_registry_proxy
Support proxy cache for docker-registry type
2021-04-07 10:59:24 +08:00
Steven Zou
e2148f9eea
Merge pull request #14514 from goharbor/dependabot/pip/make/photon/prepare/pyyaml-5.4
Bump pyyaml from 4.2b1 to 5.4 in /make/photon/prepare
2021-04-07 09:57:07 +08:00
Steven Zou
10711b7de1
Merge pull request #14482 from goharbor/dependabot/pip/make/photon/prepare/jinja2-2.11.3
Bump jinja2 from 2.11.1 to 2.11.3 in /make/photon/prepare
2021-04-07 09:56:23 +08:00
stonezdj
ccd9ee8c56 Support proxy cache for docker-registry type
Add proxy cache for docker registry type
Fixes #14477, #14547
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-06 16:47:12 +08:00
Wang Yan
d03a29e531 bump up photon to 4.0
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-03-31 13:25:29 +08:00
DQ
7eebbeebdf Add jobservice task queue related task
add jobservice metrics
add redis client

Signed-off-by: DQ <dengq@vmware.com>
2021-03-30 10:29:36 +00:00
DQ
fbe9cd88f8 Enabled Prometheus for Jobservice
* Add prom server on jobservice
* Enabeld configs in templates
* Enabeld jobservice metrics in nginx

Signed-off-by: DQ <dengq@vmware.com>
2021-03-30 08:52:59 +00:00
dependabot[bot]
f20f4215c3
Bump pyyaml from 4.2b1 to 5.4 in /make/photon/prepare
Bumps [pyyaml](https://github.com/yaml/pyyaml) from 4.2b1 to 5.4.
- [Release notes](https://github.com/yaml/pyyaml/releases)
- [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES)
- [Commits](https://github.com/yaml/pyyaml/commits/5.4)

Signed-off-by: dependabot[bot] <support@github.com>
2021-03-25 22:46:56 +00:00
DQ
f5fcc7bd31 Add base image for exporter
* Add base
* update Makefile

Signed-off-by: DQ <dengq@vmware.com>
2021-03-25 16:35:57 +08:00
dependabot[bot]
0ec667c4d8
Bump jinja2 from 2.11.1 to 2.11.3 in /make/photon/prepare
Bumps [jinja2](https://github.com/pallets/jinja) from 2.11.1 to 2.11.3.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/master/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/2.11.1...2.11.3)

Signed-off-by: dependabot[bot] <support@github.com>
2021-03-20 00:43:21 +00:00
Wenkai Yin
8b1817be0f Fix the consume too much CPU issue
1. Update execution status during the upgrade
2. Refine the execution sweeper

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-17 13:34:24 +08:00
Wenkai Yin
43df3bf8a4 Add upgrade sql file introduced in 2.1.4
1. Add upgrade sql file introduced in 2.1.4
2. Minor improvement for task/execution to cover corner cases

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-15 16:48:51 +08:00
Xavier Duthil
280c8272f8
Use exec in all components' entrypoints
Use the exec Bash command so that the final running application becomes
the container’s PID 1. This allows the application to receive any Unix
signals sent to the container, in accordance with
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#entrypoint

Currently, SIGTERM signals sent by kubernetes are not passed to the
executed binary.

Signed-off-by: Xavier Duthil <xavier.duthil@corp.ovh.com>
2021-03-05 15:00:25 +01:00
Wang Yan
3dfddfdf4e
patch upstream fix for io reader (#14356)
Fixes #12850
This patch can fix the GC failure in the NFS v3 env, see https://github.com/distribution/distribution/pull/3309#issuecomment-783606968

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-03-04 15:33:09 +08:00
Daniel Jiang
387be3686a Refine the way to set X-Forwarded-Proto in nginx
Refine the way to set the header so user won't need to comment it if
Harbor is sitting behind a reverse proxy.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-02-25 17:43:55 +08:00
danfengliu
9e3f0de12c
Merge pull request #14124 from danfengliu/reschedule-docker-login-policy-in-build-base-image-in-master
Reschedule docker login policy in base image build process
2021-02-23 10:10:59 +08:00
danfengliu
7d05c8e513 Reschedule docker login policy in base image build process
Signed-off-by: danfengliu <danfengl@vmware.com>
2021-02-22 10:05:25 +08:00
Josh Soref
dfe360040b Spelling
* addition
* attribute
* auditing
* availability
* available
* bandwidth
* browser
* business
* cadence
* chartmuseum
* client
* column
* content
* demonstrate
* described
* endpoints
* facilitate
* github
* harbor
* information
* instance
* manual
* meaningful
* operation
* overridden
* password
* possible
* project
* refactor
* replication
* requires
* running
* scanned
* settings
* signup
* those
* unsigned
* vulnerability

--
Also removes trailing space from a filename

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2021-02-19 11:59:15 +08:00
DQ
307c5a8ed4 Fix metrics template for http mode
the port shouldn't be hardcode

Signed-off-by: DQ <dengq@vmware.com>
2021-02-05 18:44:28 +00:00
Ziming Zhang
ec83f49a1a fix(retention) migrate sql error
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-02-05 09:09:26 +00:00
Wang Yan
24ec772978
fix gc migration issue (#14174)
For the upgrade path v1.10 - v2.1.* - v2.2.0, if user doesn't reset the GC schdule that was created in 1.10 in the v2.1,
the job parameters will keep empty in the database. The fix gives a default value for the schedule record.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-02-05 12:25:24 +08:00
DQ
051b5f289d Add sen existed check for internal cert
fali ealier when there is no san

Signed-off-by: DQ <dengq@vmware.com>
2021-01-28 08:22:07 +00:00
Wenkai Yin(尹文开)
50a1e85095
Make sure the revision of execution isn't null during the upgrade (#14085)
Make sure the revision of execution isn't null during the upgrade
Fixes #14075

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-27 10:10:36 +08:00
Qian Deng
f013d88efc
Merge pull request #14013 from ninjadq/upgrade_script_for_2_2_0
Harbor upgrading for 2.2
2021-01-22 18:10:24 +08:00
Qian Deng
045e1d9abe
Merge pull request #14040 from ninjadq/metric_improvement
Metric improvement
2021-01-22 17:13:57 +08:00
DQ
489f31d8fe Add upgrade scirpt for 2.2
1. add metrics config item in config
2. upgrade version in template

Signed-off-by: DQ <dengq@vmware.com>
2021-01-22 16:15:06 +08:00
Wang Yan
dba229d0df
build third party binaries in CI (#14019)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-22 11:33:42 +08:00
DQ
92cf728371 Add custom cert for exporter
* injecting custom certs related config to exporter

Signed-off-by: DQ <dengq@vmware.com>
2021-01-20 10:52:34 +08:00
DQ
a61e9b0e2e Add san for notary upgrading
if san not exists then remove that cert, prepare will regenerate one

Signed-off-by: DQ <dengq@vmware.com>
2021-01-18 21:00:35 +08:00
Wenkai Yin
7c072e17a6 Fix the legacy scheduled job issue for GC/scan all
Fix the legacy scheduled job issue for GC/scan all

Fixes #13968

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-15 22:02:36 +08:00
Daniel Jiang
1b64b9fdc2
Bump up the go-migrate (#13914)
Bump it up to v4.11.0 to be consistent with harbor-core

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-11 19:08:17 +08:00
He Weiwei
4a326aa8b0 chore: delete records of scan_report
The report in previous scan_report records not work well the
vulnerabilities stored in the schema table, so delete the scan_report
records.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-01-08 03:39:11 +00:00
Qian Deng
642d56041d
Add san for notary cert (#13928)
Signed-off-by: DQ <dengq@vmware.com>
2021-01-08 01:00:34 +08:00
stonezdj
6b8fb8431d Add quay registry to proxy cache
Update env.jinja to add quay

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-01-06 17:22:57 +08:00
Ziming Zhang
8faa76a1b6 feat(retention) refactor task manager
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-01-05 12:08:03 +08:00
Wenkai Yin(尹文开)
19ad8ad68d
Merge pull request #13823 from reasonerjt/inst-cert-home-dir
Replace tilde in install_cert.sh
2020-12-25 10:25:51 +08:00
prahaladdarkin
a890b28e1e
Store vulnerability data from scanner into a relational format (#13616)
feat: Store vulnerability report from scanner into a relational format

Convert vulnerability report JSON obtained  from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2020-12-25 08:47:46 +08:00
Wang Yan
7a8a8fa104
upgrade go version to v1.15.6 (#13836)
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-23 18:53:09 +08:00
He Weiwei
3831e82b20
refactor: remove code of admin job (#13819)
Remove code of admin job as it's not needed by scan all/gc now.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-22 11:48:16 +08:00
Daniel Jiang
9d99dfa82b Replace tilde in install_cert.sh
This commit fixes #13287 to remove the usage of tilde as the $HOME is not available in some
cases.  More details see #13287

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-21 20:39:34 +08:00
Qian Deng
31138f12b0
Merge pull request #13806 from ninjadq/fix_python_yaml_load
Fix pythom yaml load to safe_load
2020-12-21 16:04:12 +08:00
Qian Deng
9197471e70
Add Scan for internal tls (#13810)
Signed-off-by: DQ <dengq@vmware.com>
2020-12-21 15:23:11 +08:00
Wang Yan
9bc6f3cee4
fix robot account update issue (#13741)
* fix robot account update issue

enable the update method to support both v1 & v2 robot update

Signed-off-by: Wang Yan <wangyan@vmware.com>

* resolve review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 20:01:26 +08:00
Wang Yan
6bc1047013
migration admin job data (#13766)
1, migrate gc and scan all schedule to schedule/task/exectuion
2, migrate gc history to task/execution

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 16:35:24 +08:00
Will Sun
4392a626f3
Merge pull request #13804 from AllForNothing/scan-all
Fix robot account UI issues
2020-12-18 15:48:26 +08:00
Qian Deng
64fcfeaa2f
Merge pull request #13754 from ninjadq/fix_loglevel_parsing_for_registry
Fix log level issue in registry
2020-12-18 14:47:15 +08:00
AllForNothing
b20cc474b3 Fix robot account UI issues
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-12-18 14:11:08 +08:00
DQ
234b29e170 Fix pythom yaml load to safe_load
Signed-off-by: DQ <dengq@vmware.com>
2020-12-16 14:59:06 +08:00
Daniel Jiang
b0c8cadad7
Add default CVE allowlist to project library (#13770)
fixes #12700

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-16 14:20:56 +08:00
Wenkai Yin
69808f033e Tiny fixes for task manager
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-12-14 17:26:32 +08:00
He Weiwei
08580f9fec
refactor(scan): refactor scan/scan all job to task manager (#13684)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 13:34:35 +08:00
DQ
19e8527cc1 Fix log level issue in registry
1. fix level issue in registry.jinja
2. add log level to registryctl

Signed-off-by: DQ <dengq@vmware.com>
2020-12-14 11:52:42 +08:00
Wenkai Yin(尹文开)
6569016d35
Merge pull request #13139 from wy65701436/migrate-gc
Migrate gc to task manager
2020-12-14 10:43:44 +08:00
Wang Yan
39e1a4f2b4 add extra attributes in the schedule table
Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-14 02:28:52 +08:00
wang yan
1bb79d402d update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 16:08:52 +08:00
Wang Yan
dba5522d0b Migrate to task manager (#129)
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 14:00:33 +08:00
DQ
d95f22448c Add cache for exporter
Add timed cache for exporter
default cache time is 30s, cleanup job run every 4 hours

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 21:22:40 +08:00
DQ
f0db193895 Add prepare file for exporter
prepare env for exporter

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 21:22:13 +08:00
DQ
dc0047c48c Add build script for exporter
- Add dockerfile
- update makefile

Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 20:42:21 +08:00