From harbor 2.4, the trivy does not support scan report v1.0, we need to remove the old version scan reports from the DB.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
The openssl 3.0.0 using newer `PKCS#8` format.
But it's not compatitable with harbor core
So using tradictional format instead
Signed-off-by: Qian Deng <dengq@vmware.com>
Steps to reproduce:
Clone repo from master branch
Run: make install COMPILETAG=compile_golangimage
Wait until script will check that bzip2 is installed
Signed-off-by: YuriiPaneiko <yurapaneyko@gmail.com>
Define user.Email as sql.NullString to avoid unique constraint when email is empty in LDAP/OIDC
Separate the common/models/User with the pkg/user/dao/User
Fixes#10400
Signed-off-by: stonezdj <stonezdj@gmail.com>
* Move request id to requestid middleware
* fix span pass to child ctx on orm
* fix typos
* remove unused code
* add operation name to Transaction
Signed-off-by: Qian Deng <dengq@vmware.com>
* use lib trace helper function
* add gracefull shutdown
* Add commens for new added exposed function
* Add licence on top of new created files
* Update trace library
* Update configs
* Add attribute and namespance in config
Signed-off-by: Qian Deng <dengq@vmware.com>
Logrotate is run with sudo as the syslog user by cron.hourly
The current working directory is `/root` which is inaccessible to the syslog
user so the logrotate command fails. Currently the following stderr is being
thrown away by the cron script:
```
error: cannot open current directory: Permission denied
```
Fixes#15468
Signed-off-by: Christopher Jenkins <christj@gmail.com>
The following information should cleanup before delete user:
Delete project member of this user.
Delete oidc_user when auth_mode is oidc_auth.
Fixes#8424
It also removes the deleted user from project member and the deleted condition in the project member query for consistency
Signed-off-by: stonezdj <stonezdj@gmail.com>
Fixes#14932
Harbor recompiles the notary v0.6.1 with go 1.15 from v2.2.0, which introduces an break change that leads to notary key not found after migration.
[Root cause]
Notary v0.6.1 consumed an old version dvsekhvalnov/jose2, which is not compatible with go 1.15.
[References]
https://github.com/dvsekhvalnov/jose2go/issues/26https://github.com/golang/go/issues/41089
[Resolve]
To resolve this issue, we have to roll back go vesrion to v1.14 for notary v0.6.1 binary and keep it until upstream have a patch release to support go 1.15 or above.
[Break change]
If you pushed and signed image using Harbor v2.2.0 ~ v2.2.2 and created new repository key in notary, you will encouter the same issue after migrate to v2.2.3(or above) or v2.3.1(or above) because of the go version downgrade. We will have a FAQ to help you to resovle this particular scenario.
The influence path of the particular case:
Harbor v2.1.0(or lower) --> [v2.2.0 ~ v2.2.2] --> v2.2.3(or above)
Harbor v2.1.0(or lower) --> v2.3.0 --> v2.3.1(or above)
The non influence path of the paticular case:
Harbor v2.1.0(or lower) --> v2.2.3(or above)
Harbor v2.1.0(or lower) --> v2.3.1(or above)
[Fix in Version]
Harbor v2.2.3 or above
Harbor v2.3.1 or above
[Note]
If you're a heavy user of notary, avoid using v2.2.0, v2.2.1, v2.2.2 and v2.3.0, and use the fixed version for instead.
Signed-off-by: Wang Yan <wangyan@vmware.com>
Remove build base executable in Makefile by replacing it as an input parameter.
Add add more input parameters for controlling docker pull/push to make
build base process flexible for users.
Signed-off-by: danfengliu <danfengl@vmware.com>
Fixed#15034, as for postgres 13, the default shm size is 64MB, set to 1gb to avoid could not resize shared memory segment error.
Signed-off-by: Wang Yan <wangyan@vmware.com>
* add build arch parameter in Makefile
Add parameter BUILDARCH for make file. DB base builds pg96 for x86_64 only
Signed-off-by: Wang Yan <wangyan@vmware.com>
To ensure the upgrade execution idempotence, it needs to clean the $PGDATANEW on pg_upgrade failure.
Otherwise, the upgrade will skip the upgrade process from the second time launch as the exist of $PGDATANEW.
Signed-off-by: Wang Yan <wangyan@vmware.com>
1. Use root instead of ~ in notary parameter;
2. Fix tag immutability issue caused by GUI change;
3. Replace email domain name to harbor test;
4. Add login for each base image in makefile;
5. Add customize look test in nightly.
Signed-off-by: danfengliu <danfengl@vmware.com>
1, use the pg source and photon spec to build postgres 9.6
2, install 9.6 on the photon 4.0
3, then leverage pg_upgrade to handle the pg major version migration
Signed-off-by: Wang Yan <wangyan@vmware.com>
Improve the performance of artifact related APIs by adding indexes and refactoring sql logic
Closes#13890#14813#14814
Signed-off-by: Wenkai Yin <yinw@vmware.com>
When the core service cannot response the checkin request in time, duplicated execution records may be created, this commit introduces the revision column to make sure there is only one record for one schedule trigger
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Build base image step should be in build package workflow, and local base images build by new step should be removed since images have been pushed to docker hub.
Signed-off-by: danfengliu <danfengl@vmware.com>
Use the exec Bash command so that the final running application becomes
the container’s PID 1. This allows the application to receive any Unix
signals sent to the container, in accordance with
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#entrypoint
Currently, SIGTERM signals sent by kubernetes are not passed to the
executed binary.
Signed-off-by: Xavier Duthil <xavier.duthil@corp.ovh.com>
Refine the way to set the header so user won't need to comment it if
Harbor is sitting behind a reverse proxy.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
For the upgrade path v1.10 - v2.1.* - v2.2.0, if user doesn't reset the GC schdule that was created in 1.10 in the v2.1,
the job parameters will keep empty in the database. The fix gives a default value for the schedule record.
Signed-off-by: Wang Yan <wangyan@vmware.com>
The report in previous scan_report records not work well the
vulnerabilities stored in the schema table, so delete the scan_report
records.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
feat: Store vulnerability report from scanner into a relational format
Convert vulnerability report JSON obtained from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145
Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
This commit fixes#13287 to remove the usage of tilde as the $HOME is not available in some
cases. More details see #13287
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* fix robot account update issue
enable the update method to support both v1 & v2 robot update
Signed-off-by: Wang Yan <wangyan@vmware.com>
* resolve review comments
Signed-off-by: Wang Yan <wangyan@vmware.com>
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.
Signed-off-by: wang yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* update robot secret
1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object
Signed-off-by: Wang Yan <wangyan@vmware.com>
* updates on robot accounts
1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account
Signed-off-by: wang yan <wangyan@vmware.com>
* update code per review comments
1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.
Signed-off-by: Wang Yan <wangyan@vmware.com>
- Delete clair scanner if exist
- Delete report is it is scanned by clair
- Set Trivy to Default if it exist and not default scanner
Signed-off-by: DQ <dengq@vmware.com>
1. Clean the dirty data in quota/quota_usage.
2. Ignore the fail when getting the reference of quota.
Closes#13387
Signed-off-by: He Weiwei <hweiwei@vmware.com>
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This allows Trivy and other vulnerability scanners to correctly
determine the embedded dependencies in minified harbor-portal image.
Also simplify build process by reducing the number of layers in the
final stage container image
Signed-off-by: Dirk Mueller <dirk@dmllr.de>
1. Add configs in prepare
2. Add models and config items in Core
3. Encapdulate getting metric in commom package
4. Add a middleware for global request to collect 3 metrics
Signed-off-by: DQ <dengq@vmware.com>
This commit refactors the approach to encode a token in handler of /service/token,
by reusing pkg/token to avoid inconsistency.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
There is requirement that show the cron type(daily, weekly, etc.) on the UI, this commit adds the support for storing the cron type in the schedule model
Signed-off-by: Wenkai Yin <yinw@vmware.com>
fixes#12889
Before the migration script to fix the nativate repo_id issue, is has to remove the duplicate tags
from the tag table, which may caused by user in v2.0.2 to retag & repush the missing image.
Signed-off-by: wang yan <wangyan@vmware.com>
fixes#12827
After user migrates Harbor from v2.0.2, user got 404 when to pull specific images, and no work after push the same images again.
Fix:
1, If the issue is caused by missing repository data, this fix can revert the missing repository data and all things should be fine.
2, If the issue is caused by missing blob data, this fix can revert the missing repository data and still left the media type of artifact
as 'UNKNOWN', which leads the meta data and build history of the image cannot be shown in UI. User can delete and push the image again to
resolve it.
Signed-off-by: wang yan <wangyan@vmware.com>
fixes#12849
1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.
2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.
Signed-off-by: wang yan <wangyan@vmware.com>
We should use a golang that isn't having security issues.
This includes:
* go1.14.6 (released 2020/07/16) includes fixes to the go command, the
compiler, the linker, vet, and the database/sql, encoding/json,
net/http, reflect, and testing packages. See the Go 1.14.6 milestone on
our issue tracker for details.
* go1.14.7 (released 2020/08/06) includes security fixes to the
encoding/binary package. See the Go 1.14.7 milestone on our issue
tracker for details (CVE-2020-16845)
Signed-off-by: Dirk Mueller <dirk@dmllr.de>
Signed-off-by: Dirk Mueller <dmueller@suse.com>
Refresh the status of execution for every status changing of task to support filtering executions by status directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
The latest `photon:2.0` does not include `groupadd` and `useradd`
we need to install `shadow` package which includes these commands.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Fixes#11885
This part will not by default be packaged into release.
A README.md will be added in another commit.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule
Signed-off-by: Wenkai Yin <yinw@vmware.com>
