Fix#9614, all communication between internal components should bypass the proxy
Add chartmuseum, notary-server,clair-adapter to the no_proxy list in harbor.yml
Signed-off-by: stonezdj <stonezdj@gmail.com>
This commit removes scripts have been deprecated for a while and users have to use
helm chart to deploy Harbor on top of k8s cluster.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job
fix#9705fix#9722fix#9670
Signed-off-by: Steven Zou <szou@vmware.com>
As the quota sync is default called by harbor-core on every launch, and it will break the launch process if any failure throwed.
1, The commit is to provide an switcher for the system admin to bypass the quota sync.
2, In case Harbor goes into the restarting cycle.
Harbor already provides an internal API to sync quota data, in the failure case,
system admin can launch harbor and call the /api/internal/syncquota to sync quota.
Signed-off-by: wang yan <wangyan@vmware.com>
- create API folder
- move harbor API swagger file to API/harbor
- add scanner adapter open API swagger file to API/scanner
- update protal build Dockerfile
- update swagger explorer build command in Makefile
Signed-off-by: Steven Zou <szou@vmware.com>
1, replace the UIVERSION file with ldflags, which is generarted by make to inject into the UI core.
2, inject additional ldflags for harbor compiler
Signed-off-by: wang yan <wangyan@vmware.com>
1. Upgrade clair adapter to v1.0.0.
2. Make the clair adapter which installed by harbor immutable and using internal registry address.
3. Add support to build clair adapter image from binary.
4. Switch to ScannerPull action when make authorization for the scan request.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
- do changes to the sql schema
- add `UseInternalAddr` and `Immutable` properties to scanner registration
- support multiple authentication type
- basic
- bearer token
Signed-off-by: Steven Zou <szou@vmware.com>
This commit modify the log message from upstream notary DB migrator, to
make sure the DB URL is not displayed.
Fixes#7510
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Bump up the golang for compiling the binaries to 1.12.12
This commit also includes some minor changes to Makefile to fix issue in
building the binary files.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema
Signed-off-by: Steven Zou <szou@vmware.com>
- refactor the db schema \
- refactor permission checking in API handlers \
to follow the latest code/interface changes
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases
- update robot account making content
- hidden credential in the job log
Commnet scan related API test cases which will be re-activate later
fix#8985
fix the issues found by codacy
Signed-off-by: Steven Zou <szou@vmware.com>
1. Introduce NPM_REGISTRY in Makefile to support npm registry
configuration when build portal image.
2. Install npm pkgs before copy portal src so that build cache works for
npm install in portal image.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit is target to fix harbor issue #9186, which root cause is mentioned by
https://github.com/docker/distribution/issues/2553, and fixed by https://github.com/docker/distribution/pull/2879.
As the latest distribution release(v2.7.1) does not contain this fix, but it will break the quota migraion process on S3 storage, we have to path this fix into Harbor regsitry binary.
[Tag Version]
It uses the issue number(2553) as the tag naming convention, like v2.7.1-patch-2553, means that we patch the fix of issue 2553 into v2.7.1.
[Note]
So far, this fix is only targets on docker regsitry v2.7.1. If the registry has this fix in new release, we'll move on.
Signed-off-by: wang yan <wangyan@vmware.com>
- change error collection in scan job
- add dead client checking in client pool
- change key word type to interface{} for q.Query
- update bearer authorizer
- add required UT cases
Signed-off-by: Steven Zou <szou@vmware.com>
- add scanner rest API v1 spec
- implement v1 client which is used to talk to scanner adapter
- adjust data/orm models
- adjust code package structure
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan client which is used to talk to scanner adapter
- implement scan job which take the work of communicating with scanner
- update scanner mgmt API routes
- add corresponding UT cases
This commit modifies nginx configuration file to make sure the secure
flag is added to "Set-Cookie" header when Harbor is serving https
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- add DAO layer for scanner registration
- add CURD manager for scanner registration
- add API controller for plug scanner
- add REST APIs for CURD of plug scanner
- add migration sql:0011_1.10.0
- add scan interface definition (no implementations)
- add related UT cases with testify
fix#8979#8990
Signed-off-by: Steven Zou <szou@vmware.com>
The way Harbor handles notification is problematic.
It currently triggers rescan, which will cause problem when there are
lot of images in the registry.
Such as #7316
This commit removes the notifier and we need to revisit the notification
to figure out how to map the notification to a particular image if need
the notification mechanism in future.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
this is for internal registry api call, the request should be intercpeted by quota middlerwares, like retag and delete.
Note: The api developer has to know that if the internal registry call in your api, please consider to use
NewRepositoryClientForLocal() to init the repository client, which can handle quota change.
Signed-off-by: wang yan <wangyan@vmware.com>
Signed-off-by: wang yan <wangyan@vmware.com>
Add dao for quota
Signed-off-by: He Weiwei <hweiwei@vmware.com>
fix govet
Signed-off-by: wang yan <wangyan@vmware.com>
Add "MaxMessageSize" to the config of rsyslogd to avoid the mess of log file when the size of one log line > 8k
Signed-off-by: Wenkai Yin <yinw@vmware.com>
In v1.7.0, it creates an record for scan all but without cron string, just update the record with the cron in properties.
Signed-off-by: wang yan <wangyan@vmware.com>
* Fix migration script
1. port is string when parsed from configparser
2. remove index and db_user in if condition
Signed-off-by: Qian Deng <dengq@vmware.com>
* Add port to public_url
Add port to public_url
Signed-off-by: Qian Deng <dengq@vmware.com>
* Customized value for notary and clair
db config in notary and clair is hardcoded
Signed-off-by: Qian Deng <dengq@vmware.com>
* Add notary and clair db config in harbor.yml
Add notary clair config to harbor.yml and fix related regression
Signed-off-by: Qian Deng <dengq@vmware.com>
* add scan all and gc schedule migration
Signed-off-by: wang yan <wangyan@vmware.com>
* Fix gofmt errors
Signed-off-by: wang yan <wangyan@vmware.com>
* Update code according to review comments
Signed-off-by: wang yan <wangyan@vmware.com>
* remove convertschedule return name just return value
Signed-off-by: wang yan <wangyan@vmware.com>
1. handle the public/private property when creating the projects
2. extend the length of access_secret
3. update the task status by using orm functions
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Mount the ca bunlder into registry controller, and add them into os
trust store that resolves the problem of garabe collection on ca
enabled registry.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit introduces a solution to workaround the restriction of project creation API: only normal users can create projects
Signed-off-by: Wenkai Yin <yinw@vmware.com>
As CLI does not support oauth flow, we'll use secret for help OIDC user
to authenticate via CLI.
Add column to store secret and token, and add code to support
verify/refresh token associates with secret. Such that when the user is
removed from OIDC provider the secret will no longer work.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit is to build a regsitry bases on v2.7.1 code and introduces
an fix on issue #2819, this is a P0 bug on v2.7.1 which causes GCS doesn't
work well on v2.7.1
For more details, refer to https://github.com/docker/distribution/pull/2821
Signed-off-by: wang yan <wangyan@vmware.com>
1. Add operation property for tasks
2. Add trigger property for executions
3. Update the getting registry info API to allow passing 0 as ID to get the info of local Harbor registry
Signed-off-by: Wenkai Yin <yinw@vmware.com>
mount a temp dir input for all input files and configs
generated secrets file stored in data volumns keys dir
certs file stored in data volumns nginx dir
Signed-off-by: Qian Deng <dengq@vmware.com>
This commit updates the upgrade sql script to fix#6698: cannot recreate the same name replication policy after it is deleted
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit is to make the expiration of robot account configurable
1, The expiration could be set by system admin in the configuation page or
by /api/config with robot_token_expiration=60, the default value is 30 days.
2, The expiration could be shown in the robot account infor both on UI and API.
Signed-off-by: wang yan <wangyan@vmware.com>
fixes#5863
The migrate binary that we include in notary is quite out dated.
Additionally it introduced a breaking change, more details see #5863
In this commit a go program was added to workaround this issue to ensure the
migration process works, and refined bootstrap scripts and make process accordingly.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* Support master role for project member create and update apis
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* Fix description for role_id in swagger.yaml
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Rename the "name" colume in table "replication_policy" and "replication_target" before adding the "UNIQUE" constraint to avoid the upgrade failure
Signed-off-by: Wenkai Yin <yinw@vmware.com>
I guess that the purpose of this check is to verify that the container is listening on port 10514. Healthcheck default timeout is 30 sec. In places where the DNS resolver is not working properly, this check could take more than 30 sec, which leads to decide that the container health is unhealthy. I advise you to add to your check the option n, which prevents netstat trying to determine the symbolic host.
Signed-off-by: overdogwatch <overdogwatch@gmail.com>
1. Fix#5102 by checking the existence of name when creating/editing replication rule
2. Add unique constraint to the name of replication policy and target
3. Fix bugs of testing library
Signed-off-by: Wenkai Yin <yinw@vmware.com>
resolve build error 'invalid from flag value 0: repository sha256 not found: does not exist or no pull access'
Signed-off-by: Brett Johnson <brett@sdbrett.com>
Some configuration files and env files contain sensitive information,
they should not be readable by any user by default.
This commit updates the `prepare` script to update the mask of the
rendered files.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Update the entrypoint to allow the image accept other parms,
to help debug in the future.
If replace "$*" with "$@" only one parm will be passed to dumbinit
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit is to add the clair_version into the harbor images, then clair
will use it in the user-agent, and helpful for the debugging.
Signed-off-by: wang yan <wangyan@vmware.com>
Fixes#6115
As for the change in migration sql file, in 1.7 we'll switch to
jobservice for scheduling "scan all" job. To avoid inconsistency,
this item will be reset and user will need to configure the policy again.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Modify nginx configuration to use 308 instead of 301 on the http to
https redirect.
Fix problems with some clients on POST requests that are transformed to
GET on 301 redirect (per HTTP 1.1 standard).
See [RFC7538](https://tools.ietf.org/html/rfc7538).
Signed-off-by: Stéphane Albert <sheeprine@oh.its.fake.nullplace.com>
Update the `grep` filter to anchor at the start of the line
and allow for whitespace characters, in order to correctly
determine the hostname being set if the default is merely
commented out and the custom one added, instead of overridden.
Fixes#6117
Signed-off-by: Christian Witts <cwitts@gmail.com>
Make necessary change to make things work with photon 2.0 docker image.
Remove distro-sync to mitigate the build issue and add `--pull` to docker build
command to make sure the latest photon:2.0 will be pulled during build process.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This env is the workaroud of dns resolver on golang 1.7.3.
Remove it is bacause of harbor is using golang 1.9.2, the bug
has already been fixed.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit is to upgrade the golang version to 1.9.4, it because a
bug of golang 17.3 could introduce one dns resolver issue for harbor
mentioned by #6031.
The bug of golang is https://github.com/golang/go/issues/15419, it makes
harbor containers to lookup 'endpoint.' firstly which may cause network
issue.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit is to set dns search to null in the harbor containers,
that means the dns search domains of docker host doesn't impact
the network IO in the containers.
If do not set this, Harbor notary-server and notary-signer are resolving
the "mysql" alias to the resolv.conf search path instead of to "mysql."
for the notary-db bridge IP, see #6031.
Signed-off-by: wang yan <wangyan@vmware.com>
The migrate tool will try to create table schema_migration upon opening
the connection to DB. This will cause error when there are multiple
instance of adminserver trying to access the migrator upon start.
This commit move the creation of the table during the initialization of
the DB container.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit revoke part of the change introduced in commit #1fc4142, by
calling chown to job log directory within the container when the job
service bootstraps. The reason is we are seeing permission issue in
helm-chart deployment, and we want to reduce effort to handle the
permission on different deployment approaches.
There are some code in `prepare` script to change the ownership of the
JOB_LOG directory, it will be left for now to avoid regression in VIC
integration.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Update the nginx.conf
2. Update Makefile
3. Update docker-compose
4. Update image name
5. Rename folder ui to core
6. Change the harbor-ui's package name to core
7. Remove unused static file on harbor-core
8. Remove unused code for harbor-portal
Signed-off-by: Qian Deng <dengq@vmware.com>
The sslmode of the connection with postgresql is hardcoded as "disable" currently, this commit expose it as an environment variable so that users can configure it
Signed-off-by: Wenkai Yin <yinw@vmware.com>
The PR to fix the Alpine issue has been merged to Clair's release-2.0
branch, and released v2.0.5.
This commit updates Harbor to include that change and re-enable
Clair's updaters by default.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Move the notary-server and notary signer into ./notary/release-${notaryversion} as this will not impact the
release branches, the binaries in ./notary are v0.5.1.
Signed-off-by: wang yan <wangyan@vmware.com>
Set the updater interval to "0" to mitigate the impact of Apline URL
change that cause clair keep polling vuln data.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
The VOLUME definition in Dockerfile of chart museum will mount a volume automatically by docker if no specific volume is provided.
Signed-off-by: Wenkai Yin <yinw@vmware.com>
In some user's environment, there's local object storage hosted with
self-signed certificate.
Because registry process runs in a photon container, it has to trust
the certificate in the photon level such that the registry can access
the storage service.
This commit updates the registry image to append custom cert to the root
bundle when the container is started. And make the customer cert
configurable in `harbor.cfg`
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Add clean registry cache to gc job
To workaround the issue: https://github.com/docker/distribution/issues/2094
GC needs to clean cache before to call the docker reigstry api to delete blobs.
Otherwise, the following docker push will not be performed as docker registry
does not clean cache in GC, it thinks the image is still there, and the new
blobs will be uploaded.
After UI compilling should unlink harbor-ui.
Because it create a link with root user in container that will cause permission issues next time you want to aceess this file.
Root cause: Use default 'now'::timestamp will not generate timestamp for each transaction,
PG will convert now to a timestamp as soon as the constant is parsed. To fix it, update it
to defult CURRENT_TIMESTAMP, thie setting is the same as default now(), which returns the
start time of current transaction because ther are fuction calls, hey will give the desired
behavior of defaulting to the time of row insertion.
Reference: https://www.postgresql.org/docs/9.6/static/functions-datetime.html#FUNCTIONS-DATETIME-CURRENT
PG version: 9.6.9
In 1.6, there will be only one DB process in the default deployment.
The migrator will try to handle the setting by "guessing" whether Harbor
was pointed to external DB.
Verified 1.5->1.6 and 1.4->1.6 migration.
When proxy is set for Clair, there may be issue when Clair pulls image
from the registryif the `no_proxy` attribute is not updated. This
commit adds `registry` to the default setting.
append chart server related config options to the supporting list of adminserver
provide chart server related config access method in the API layer
update prepare script and ui env template file to enable cache driver config for chart server API
append flag info in the systeminfo API to indicate if chart server is deployed with Harbor
refactor the response rewriting logic to return structual error object
add api init method to initilizing objects required in API handlers
chage owner of the storage folder
update offline/online package scripts in Harbor-Util.robot
It supports Harbor admin to trigger job either manual or
schedule. The job will be populated to job service to execute.
The api includes:
1. POST /api/system/gc
2, GET /api/system/gc/:id
3, GET /api/system/gc/:id/log
4, PUT/GET/POST /api/system/gc/schedule
add env file template for chart repo server in make/common/config/chartserver
update the Makefiles to support build chart repo server
add docker file and related build scripts for upstream chart server - chartmuseum
update prepare to support generating chart server related configs
add docker compose file for the chart server
add build/install command options to install with/without chart repo server
update install.sh to support chart repo server installation
docker regsitry. This version has the API to call regsitry GC with jobservice
secret. Seprates it into a standalone container as do not want to invoke two
processes in one container.
It needs to mount the registry storage into this container in order to do GC,
and needs to copy the registry binary into it.
This commit make update to remove the code from ui container to init the
DB schema. As UI has dependency on admin server, so it's safe to assume
adminserver has to be ready first. Regardless the setting of the config
store of admin server, it will try to access and intialize the schema of
database.
By default Harbor will call catalog API of registry and sync the result to DB, this becomes problematic when registry is configured to custom storage service, there maybe inconsistent result and the whole process may be very time consuming.
So in this commit a env var SYNC_REGISTRY is introduced if user want Harbor to sync the repo when it starts up, by default it's false.
This commit bump up clair to v2.0.4. The current build process is
download the binary from google storage, the update of the binary in
google storage is not reflected in this commit.
It gives Harbor the capability to controll the cache of docker
registry, and the workaround for cache invalidation bug caused
by garbage collection, that is clean cache in GC job.
For more details, see Harbor issue #5078.
The init sql script name nad path was changed by PR #5197, this
commit is to update these and log the package command to console,
make it more easy to debug in future. Also remove the action to
pull migrator as it will built each time locally.
This commit is a temp fix to workaround coreos/clair#562
Recompiled the code at the tip of release-2.0 branch of clair and
updated Makefile.
Once clair provides a new release, we'll need to make update in
Makefiles and Dockerfiles again to consume it.
This commit fixes#5072
Due to an issue in bzr, Clair container may consume a lot of CPU
resource while updating the vuln data. This commit mitigates the impact
by setting the cpu_quota of clair container. (default value of
cpu_period is 100000 in v2 docker-compose template)
This commit fixes#5040, the harbor-db image will only contain empty
databases, and harbor ui container will use migrate tool to run initial
SQL scripts to do initialization. This is helpful for the case to
configure Harbor against external DB or DBaaS like RDS for HA deployment
However, this change will results some confusion as there are two tables
to track schema versions have been using alembic for migration, for this
release we'll try to use alembic to mock a `migration` table during
upgrade so the migrator will be bypassed, in future we'll consider to
consolidate to the golang based migrator.
Another issue is that the UI and adminserver containers will access DB
after start up in different congurations, can't ensure the sequence, so
both of them will try to update the schema when started up.
The secretkey file will be loaded by adminserver which is run by non-root
user (uid:10000) previously the entrypoint script will run `chown` to a
lot files, and there's a breakage in upgrade when we skip running
`chown` inside container.
This commit will fix the issue during upgrade by changing the owner of
the secretkey file.
This commit is to enable data migrator to support migrates data
from mysql to pgsql, this is a specific step for user to upgrade
harbor across v1.5.0, as we have move harbor DB to pgsql from
1.5.0. It supports both harbor and notary db data migration,
and be split into two steps with dependency.
It also fix issue #4847, add build DB migrator in make process.
Narrow down the scope of `chown` in adminserver because the
/etc/adminserver/config/ is the location to store the config.json file.
And /etc/adminserver/key should be readonly.
The job logs directory's permission is not changed by prepare script
because the everything is moved from /data to /storage/data on VIC
appliance. This commit will make sure both cases the directory is
readable by user 10000:10000.
This PR also makes sure the config json of notary signer has 0644
permission.
Previously the log file was set to a hard coded file, but given this
redis should run in container, the update is made to have the process
output log messages to standard output, and redirect it to syslog in
docker-compose template.
This commit fixes a recently discovered issue on Kubernetes #4496
It make necessary to avoid calling `chown` to config files during the
bootstrap of the containers.
replace tcp host:port with
'redis://arbitrary_usrname:password@ipaddress:port/database_index'
update prepare to generate config yaml file of job service based on harbor.cfg
update harbor.cfg default values
Fix typo in Makefile under photon
Fix version tag issue of redis container
Assign container name for redis container
Update docker compose template to enable network for redis
Remove exposed ports of redis from compose yaml tpl
We have to add the uuid/id mapping as new job service will only store uuid.
Further work is in feature branch for now, commit this change to
accelerate migration work.
Default target version is 1.5.0
This is mainly for VIC-appliance upgrade, and should be considered
experimental for oss due to limited test.
Tested with 1.2 and 1.3 harbor.cfg from VIC appliance.
The following are done to avoid travis-ci failing due to too much log
size.
1) Update Makefile and scripts to make go build less verbose.
2) Make tdnf less verbose
As this is for tile deployment only, so add a shortcut for tile/bosh
script to add entry in /etc/hosts inside the container.
Due to effort consideration I don't think we want to render
docker-compose in `prepare` script.
Enable configuring the path of root cert of UAA in harbor.cfg. It only
takes effects if the verify_cert is set to "true" If the file does not
exist, the configuration is skipped.
The intention for this commit is to support integration with nested UAA
in PAS or PKS, we don't expect user to manually configure this value,
though he can do it if he wants.
