1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth. #10604 is opened
to track the follow up work.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit updates `prepare` and templates to populate the credential
for registry for basic authentication.
A temporary flag `registry_use_basic_auth` was added to avoid breakage.
It MUST be removed before the release.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
In order to replace travis.
Implement 5 CI jobs
- UTTEST
- APITEST_DB
- APITEST_LDAP
- OFFLINE
- UI_UT
Signed-off-by: Ziming Zhang <zziming@vmware.com>
As the quota sync is default called by harbor-core on every launch, and it will break the launch process if any failure throwed.
1, The commit is to provide an switcher for the system admin to bypass the quota sync.
2, In case Harbor goes into the restarting cycle.
Harbor already provides an internal API to sync quota data, in the failure case,
system admin can launch harbor and call the /api/internal/syncquota to sync quota.
Signed-off-by: wang yan <wangyan@vmware.com>
- create API folder
- move harbor API swagger file to API/harbor
- add scanner adapter open API swagger file to API/scanner
- update protal build Dockerfile
- update swagger explorer build command in Makefile
Signed-off-by: Steven Zou <szou@vmware.com>
1, replace the UIVERSION file with ldflags, which is generarted by make to inject into the UI core.
2, inject additional ldflags for harbor compiler
Signed-off-by: wang yan <wangyan@vmware.com>
1. Upgrade clair adapter to v1.0.0.
2. Make the clair adapter which installed by harbor immutable and using internal registry address.
3. Add support to build clair adapter image from binary.
4. Switch to ScannerPull action when make authorization for the scan request.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit modify the log message from upstream notary DB migrator, to
make sure the DB URL is not displayed.
Fixes#7510
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Bump up the golang for compiling the binaries to 1.12.12
This commit also includes some minor changes to Makefile to fix issue in
building the binary files.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Introduce NPM_REGISTRY in Makefile to support npm registry
configuration when build portal image.
2. Install npm pkgs before copy portal src so that build cache works for
npm install in portal image.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit is target to fix harbor issue #9186, which root cause is mentioned by
https://github.com/docker/distribution/issues/2553, and fixed by https://github.com/docker/distribution/pull/2879.
As the latest distribution release(v2.7.1) does not contain this fix, but it will break the quota migraion process on S3 storage, we have to path this fix into Harbor regsitry binary.
[Tag Version]
It uses the issue number(2553) as the tag naming convention, like v2.7.1-patch-2553, means that we patch the fix of issue 2553 into v2.7.1.
[Note]
So far, this fix is only targets on docker regsitry v2.7.1. If the registry has this fix in new release, we'll move on.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit modifies nginx configuration file to make sure the secure
flag is added to "Set-Cookie" header when Harbor is serving https
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
The way Harbor handles notification is problematic.
It currently triggers rescan, which will cause problem when there are
lot of images in the registry.
Such as #7316
This commit removes the notifier and we need to revisit the notification
to figure out how to map the notification to a particular image if need
the notification mechanism in future.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
this is for internal registry api call, the request should be intercpeted by quota middlerwares, like retag and delete.
Note: The api developer has to know that if the internal registry call in your api, please consider to use
NewRepositoryClientForLocal() to init the repository client, which can handle quota change.
Signed-off-by: wang yan <wangyan@vmware.com>
Add "MaxMessageSize" to the config of rsyslogd to avoid the mess of log file when the size of one log line > 8k
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* Fix migration script
1. port is string when parsed from configparser
2. remove index and db_user in if condition
Signed-off-by: Qian Deng <dengq@vmware.com>
* Add port to public_url
Add port to public_url
Signed-off-by: Qian Deng <dengq@vmware.com>
* Customized value for notary and clair
db config in notary and clair is hardcoded
Signed-off-by: Qian Deng <dengq@vmware.com>
* Add notary and clair db config in harbor.yml
Add notary clair config to harbor.yml and fix related regression
Signed-off-by: Qian Deng <dengq@vmware.com>
Mount the ca bunlder into registry controller, and add them into os
trust store that resolves the problem of garabe collection on ca
enabled registry.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit is to build a regsitry bases on v2.7.1 code and introduces
an fix on issue #2819, this is a P0 bug on v2.7.1 which causes GCS doesn't
work well on v2.7.1
For more details, refer to https://github.com/docker/distribution/pull/2821
Signed-off-by: wang yan <wangyan@vmware.com>
mount a temp dir input for all input files and configs
generated secrets file stored in data volumns keys dir
certs file stored in data volumns nginx dir
Signed-off-by: Qian Deng <dengq@vmware.com>
fixes#5863
The migrate binary that we include in notary is quite out dated.
Additionally it introduced a breaking change, more details see #5863
In this commit a go program was added to workaround this issue to ensure the
migration process works, and refined bootstrap scripts and make process accordingly.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
I guess that the purpose of this check is to verify that the container is listening on port 10514. Healthcheck default timeout is 30 sec. In places where the DNS resolver is not working properly, this check could take more than 30 sec, which leads to decide that the container health is unhealthy. I advise you to add to your check the option n, which prevents netstat trying to determine the symbolic host.
Signed-off-by: overdogwatch <overdogwatch@gmail.com>
resolve build error 'invalid from flag value 0: repository sha256 not found: does not exist or no pull access'
Signed-off-by: Brett Johnson <brett@sdbrett.com>
Update the entrypoint to allow the image accept other parms,
to help debug in the future.
If replace "$*" with "$@" only one parm will be passed to dumbinit
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit is to add the clair_version into the harbor images, then clair
will use it in the user-agent, and helpful for the debugging.
Signed-off-by: wang yan <wangyan@vmware.com>
Make necessary change to make things work with photon 2.0 docker image.
Remove distro-sync to mitigate the build issue and add `--pull` to docker build
command to make sure the latest photon:2.0 will be pulled during build process.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit is to upgrade the golang version to 1.9.4, it because a
bug of golang 17.3 could introduce one dns resolver issue for harbor
mentioned by #6031.
The bug of golang is https://github.com/golang/go/issues/15419, it makes
harbor containers to lookup 'endpoint.' firstly which may cause network
issue.
Signed-off-by: wang yan <wangyan@vmware.com>
The migrate tool will try to create table schema_migration upon opening
the connection to DB. This will cause error when there are multiple
instance of adminserver trying to access the migrator upon start.
This commit move the creation of the table during the initialization of
the DB container.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit revoke part of the change introduced in commit #1fc4142, by
calling chown to job log directory within the container when the job
service bootstraps. The reason is we are seeing permission issue in
helm-chart deployment, and we want to reduce effort to handle the
permission on different deployment approaches.
There are some code in `prepare` script to change the ownership of the
JOB_LOG directory, it will be left for now to avoid regression in VIC
integration.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Update the nginx.conf
2. Update Makefile
3. Update docker-compose
4. Update image name
5. Rename folder ui to core
6. Change the harbor-ui's package name to core
7. Remove unused static file on harbor-core
8. Remove unused code for harbor-portal
Signed-off-by: Qian Deng <dengq@vmware.com>
The PR to fix the Alpine issue has been merged to Clair's release-2.0
branch, and released v2.0.5.
This commit updates Harbor to include that change and re-enable
Clair's updaters by default.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Move the notary-server and notary signer into ./notary/release-${notaryversion} as this will not impact the
release branches, the binaries in ./notary are v0.5.1.
Signed-off-by: wang yan <wangyan@vmware.com>
The VOLUME definition in Dockerfile of chart museum will mount a volume automatically by docker if no specific volume is provided.
Signed-off-by: Wenkai Yin <yinw@vmware.com>
In some user's environment, there's local object storage hosted with
self-signed certificate.
Because registry process runs in a photon container, it has to trust
the certificate in the photon level such that the registry can access
the storage service.
This commit updates the registry image to append custom cert to the root
bundle when the container is started. And make the customer cert
configurable in `harbor.cfg`
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
append chart server related config options to the supporting list of adminserver
provide chart server related config access method in the API layer
update prepare script and ui env template file to enable cache driver config for chart server API
append flag info in the systeminfo API to indicate if chart server is deployed with Harbor
refactor the response rewriting logic to return structual error object
add api init method to initilizing objects required in API handlers
chage owner of the storage folder
update offline/online package scripts in Harbor-Util.robot
add env file template for chart repo server in make/common/config/chartserver
update the Makefiles to support build chart repo server
add docker file and related build scripts for upstream chart server - chartmuseum
update prepare to support generating chart server related configs
add docker compose file for the chart server
add build/install command options to install with/without chart repo server
update install.sh to support chart repo server installation
docker regsitry. This version has the API to call regsitry GC with jobservice
secret. Seprates it into a standalone container as do not want to invoke two
processes in one container.
It needs to mount the registry storage into this container in order to do GC,
and needs to copy the registry binary into it.
This commit make update to remove the code from ui container to init the
DB schema. As UI has dependency on admin server, so it's safe to assume
adminserver has to be ready first. Regardless the setting of the config
store of admin server, it will try to access and intialize the schema of
database.
This commit bump up clair to v2.0.4. The current build process is
download the binary from google storage, the update of the binary in
google storage is not reflected in this commit.
The init sql script name nad path was changed by PR #5197, this
commit is to update these and log the package command to console,
make it more easy to debug in future. Also remove the action to
pull migrator as it will built each time locally.
This commit is a temp fix to workaround coreos/clair#562
Recompiled the code at the tip of release-2.0 branch of clair and
updated Makefile.
Once clair provides a new release, we'll need to make update in
Makefiles and Dockerfiles again to consume it.
This commit fixes#5040, the harbor-db image will only contain empty
databases, and harbor ui container will use migrate tool to run initial
SQL scripts to do initialization. This is helpful for the case to
configure Harbor against external DB or DBaaS like RDS for HA deployment
However, this change will results some confusion as there are two tables
to track schema versions have been using alembic for migration, for this
release we'll try to use alembic to mock a `migration` table during
upgrade so the migrator will be bypassed, in future we'll consider to
consolidate to the golang based migrator.
Another issue is that the UI and adminserver containers will access DB
after start up in different congurations, can't ensure the sequence, so
both of them will try to update the schema when started up.
This commit is to enable data migrator to support migrates data
from mysql to pgsql, this is a specific step for user to upgrade
harbor across v1.5.0, as we have move harbor DB to pgsql from
1.5.0. It supports both harbor and notary db data migration,
and be split into two steps with dependency.
It also fix issue #4847, add build DB migrator in make process.
Narrow down the scope of `chown` in adminserver because the
/etc/adminserver/config/ is the location to store the config.json file.
And /etc/adminserver/key should be readonly.
Previously the log file was set to a hard coded file, but given this
redis should run in container, the update is made to have the process
output log messages to standard output, and redirect it to syslog in
docker-compose template.
This commit fixes a recently discovered issue on Kubernetes #4496
It make necessary to avoid calling `chown` to config files during the
bootstrap of the containers.
Fix typo in Makefile under photon
Fix version tag issue of redis container
Assign container name for redis container
Update docker compose template to enable network for redis
Remove exposed ports of redis from compose yaml tpl
We have to add the uuid/id mapping as new job service will only store uuid.
Further work is in feature branch for now, commit this change to
accelerate migration work.
The following are done to avoid travis-ci failing due to too much log
size.
1) Update Makefile and scripts to make go build less verbose.
2) Make tdnf less verbose
This change involves using non-root user to run the process of the
docker images. Also made update in Dockerfile to make the containers
support "read-only" and introduce "HEALTHCHECK". Note the "read-only"
options are not enabled in docker-compose, to cover the very corner
case when user wants to update the container filesystem manually.
Remove read only option from docker-compose template by default
This change reworked the vmware/harbor-db image to build it on top of
vmware/mariadb-photon.
Also made minor change in the entrypoint script of mariadb image to
execute upgrade script during bootstrap, and fix a file permission
issue in the bootstrap scripts.
This change mitigate problems with container restarts (stop, start) or automatic restart after host machine restart. Rsyslogd strictly checks existence of its pid file and won't start if such one exists.
