fixes#12889
Before the migration script to fix the nativate repo_id issue, is has to remove the duplicate tags
from the tag table, which may caused by user in v2.0.2 to retag & repush the missing image.
Signed-off-by: wang yan <wangyan@vmware.com>
fixes#12827
After user migrates Harbor from v2.0.2, user got 404 when to pull specific images, and no work after push the same images again.
Fix:
1, If the issue is caused by missing repository data, this fix can revert the missing repository data and all things should be fine.
2, If the issue is caused by missing blob data, this fix can revert the missing repository data and still left the media type of artifact
as 'UNKNOWN', which leads the meta data and build history of the image cannot be shown in UI. User can delete and push the image again to
resolve it.
Signed-off-by: wang yan <wangyan@vmware.com>
fixes#12849
1, gives a default value to blob status in the migration script, and use none to replace the empty string as
the StatusNone, that will more readable on debugging failure.
2, GC jobs marks all of blobs as StatusDelete in the mark phase, but if encounter any failure in the sweep phase,
GC job will quite and all of blobs are in StatusDelete. If user wants to execute the GC again, it will fail as the
StatusDelete cannot be marked as StatusDelete. So, add StatusDelete in the status map to make StatusDelete can be
marked as StatusDelete.
Signed-off-by: wang yan <wangyan@vmware.com>
We should use a golang that isn't having security issues.
This includes:
* go1.14.6 (released 2020/07/16) includes fixes to the go command, the
compiler, the linker, vet, and the database/sql, encoding/json,
net/http, reflect, and testing packages. See the Go 1.14.6 milestone on
our issue tracker for details.
* go1.14.7 (released 2020/08/06) includes security fixes to the
encoding/binary package. See the Go 1.14.7 milestone on our issue
tracker for details (CVE-2020-16845)
Signed-off-by: Dirk Mueller <dirk@dmllr.de>
Signed-off-by: Dirk Mueller <dmueller@suse.com>
Refresh the status of execution for every status changing of task to support filtering executions by status directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
The latest `photon:2.0` does not include `groupadd` and `useradd`
we need to install `shadow` package which includes these commands.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Fixes#11885
This part will not by default be packaged into release.
A README.md will be added in another commit.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Accept vendorType and vendorID when creating the schedule
2. Provide more methods in the scheduler interface to reduce the duplicated works of callers
3. Use a new ormer and transaction when creating the schedule
Signed-off-by: Wenkai Yin <yinw@vmware.com>
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit rename the var name, text appearance, and swagger of "master" role
to "maintainer" role.
It only covers backend code.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Change down version to list to accept multi verstion value
2. Update search function use BFS to find migration path
2. Add test case
Signed-off-by: DQ <dengq@vmware.com>
- define instance's api
- define extension models for api
- implement preheat controller
- implement preheat manager
- most code are picked up from the original P2P feat branch
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
* update blob controller & manager
1, add two more attributes, version, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.
Signed-off-by: wang yan <wangyan@vmware.com>
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.
Signed-off-by: wang yan <wangyan@vmware.com>
* Add api to delete blob and manifest
Enable the capability of registry controller to delete blob and manifest
Signed-off-by: wang yan <wangyan@vmware.com>
It's a workaround for issue https://github.com/goharbor/harbor/issues/11754
The phenomenon is the repository data is gone, but artifacts belong to the repository are still there.
To resolve it, just set the repository_id to a negative, and cannot duplicate.
Signed-off-by: wang yan <wangyan@vmware.com>
Since `prepare` generates the certs as needed during installation, these
certs should not exist in the repo.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Fixes#11624
All of the existing policies created v1.10 has no name, it fails the upgrade process.
When to set the unique constraint for policy name, the empty can be seen as duplicated key.
ERROR: could not create unique index "notification_policy_name_key"
DETAIL: Key (name)=() is duplicated.
Signed-off-by: wang yan <wangyan@vmware.com>
- Vendor the latest Trivy release 0.6.0
- Configure TLS 1.2 as min version when TLS is enabled
- Add more tracing to adapter config to facilitate troubleshooting
Resolves: #11544
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
Fixes#11606
As we DO NOT want to user to execute GC in the container, rename it and append the warning message.
Signed-off-by: wang yan <wangyan@vmware.com>
1. Partial helm api version number clear
2. Separate swagger to get v2.0 swagger and chart swagger
3. router add chart swagger
Signed-off-by: Yogi_Wang <yawang@vmware.com>
Fixes#11241
1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.
Signed-off-by: wang yan <wangyan@vmware.com>
Item in yaml without value will be as None in python, which will make
the password of redis as `None` in `get_redis_configs`. This fix will
not accept items of `false value` in `external_redis` configurations.
Closes#11367
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Provide more info in help message
Add requried opition and they will show missing option if you are not provide them instead of Exception
Signed-off-by: DQ <dengq@vmware.com>
User can put their ca key pair on internal cert dir and name them to `harbor_internal_ca.key` and `harbor_internal_ca.crt` we wil use them to generate other certs
Signed-off-by: DQ <dengq@vmware.com>
Allows configuring SCANNER_TRIVY_GITHUB_TOKEN environment variable,
which is passed to trivy executable binary when it starts scanning
a given artifact.
This is to increase GitHub requests rate limit from 60 per hours
(for anonymous requests) to 5000 when Trivy download its
vulnerabilities database.
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
This commit replaces beego's CSRF mechanism with gorilla's csrf library.
The criteria for requests to skip the csrf check remain the same.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1, loop each access log, change to resource/resource_type, and insert into audit log
2, loop each first push operation, change it to create repository and insert into audit log.
Signed-off-by: wang yan <wangyan@vmware.com>
1, enable dao test for artifact trash
2, set default flush trash table to false
3, hanlder empty parameter in API call
4, add registry auth info into jobservice container
Signed-off-by: wang yan <wangyan@vmware.com>
1, add API entry for get audit logs
2. add audit log manager to hanlder CRUD
Use the new format of audit log to cover differernt resource, artifact/tag/repostory/project
Signed-off-by: wang yan <wangyan@vmware.com>
As we store the repository name in the artifact table, we can use it direclty in the code to reduce the database query
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit add supporting for adding/removing label to/from artifacts and populates labels when listing artifacts
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth. #10604 is opened
to track the follow up work.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
This commit updates `prepare` and templates to populate the credential
for registry for basic authentication.
A temporary flag `registry_use_basic_auth` was added to avoid breakage.
It MUST be removed before the release.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Define the controller/manager interface for artifact and tag
2. Provide a null implementation for artifact manager
Signed-off-by: Wenkai Yin <yinw@vmware.com>
In order to replace travis.
Implement 5 CI jobs
- UTTEST
- APITEST_DB
- APITEST_LDAP
- OFFLINE
- UI_UT
Signed-off-by: Ziming Zhang <zziming@vmware.com>
This commits does some basic init work for supporting OCI:
1. Create the artifact and tag model
2. Create database tables
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit add an attribute to configurations, whose value is the
certificate of authproxy server. When this attribute is set Harbor will
pin to this cert when connecting authproxy.
This value will also be part of the response of systemInfo API.
This commit will be cherrypicked to 1.10 and 1.9 branch.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Fix#9614, all communication between internal components should bypass the proxy
Add chartmuseum, notary-server,clair-adapter to the no_proxy list in harbor.yml
Signed-off-by: stonezdj <stonezdj@gmail.com>
This commit removes scripts have been deprecated for a while and users have to use
helm chart to deploy Harbor on top of k8s cluster.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job
fix#9705fix#9722fix#9670
Signed-off-by: Steven Zou <szou@vmware.com>
As the quota sync is default called by harbor-core on every launch, and it will break the launch process if any failure throwed.
1, The commit is to provide an switcher for the system admin to bypass the quota sync.
2, In case Harbor goes into the restarting cycle.
Harbor already provides an internal API to sync quota data, in the failure case,
system admin can launch harbor and call the /api/internal/syncquota to sync quota.
Signed-off-by: wang yan <wangyan@vmware.com>
- create API folder
- move harbor API swagger file to API/harbor
- add scanner adapter open API swagger file to API/scanner
- update protal build Dockerfile
- update swagger explorer build command in Makefile
Signed-off-by: Steven Zou <szou@vmware.com>
1, replace the UIVERSION file with ldflags, which is generarted by make to inject into the UI core.
2, inject additional ldflags for harbor compiler
Signed-off-by: wang yan <wangyan@vmware.com>
1. Upgrade clair adapter to v1.0.0.
2. Make the clair adapter which installed by harbor immutable and using internal registry address.
3. Add support to build clair adapter image from binary.
4. Switch to ScannerPull action when make authorization for the scan request.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
- do changes to the sql schema
- add `UseInternalAddr` and `Immutable` properties to scanner registration
- support multiple authentication type
- basic
- bearer token
Signed-off-by: Steven Zou <szou@vmware.com>
This commit modify the log message from upstream notary DB migrator, to
make sure the DB URL is not displayed.
Fixes#7510
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Bump up the golang for compiling the binaries to 1.12.12
This commit also includes some minor changes to Makefile to fix issue in
building the binary files.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema
Signed-off-by: Steven Zou <szou@vmware.com>
- refactor the db schema \
- refactor permission checking in API handlers \
to follow the latest code/interface changes
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases
- update robot account making content
- hidden credential in the job log
Commnet scan related API test cases which will be re-activate later
fix#8985
fix the issues found by codacy
Signed-off-by: Steven Zou <szou@vmware.com>
1. Introduce NPM_REGISTRY in Makefile to support npm registry
configuration when build portal image.
2. Install npm pkgs before copy portal src so that build cache works for
npm install in portal image.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
This commit is target to fix harbor issue #9186, which root cause is mentioned by
https://github.com/docker/distribution/issues/2553, and fixed by https://github.com/docker/distribution/pull/2879.
As the latest distribution release(v2.7.1) does not contain this fix, but it will break the quota migraion process on S3 storage, we have to path this fix into Harbor regsitry binary.
[Tag Version]
It uses the issue number(2553) as the tag naming convention, like v2.7.1-patch-2553, means that we patch the fix of issue 2553 into v2.7.1.
[Note]
So far, this fix is only targets on docker regsitry v2.7.1. If the registry has this fix in new release, we'll move on.
Signed-off-by: wang yan <wangyan@vmware.com>
- change error collection in scan job
- add dead client checking in client pool
- change key word type to interface{} for q.Query
- update bearer authorizer
- add required UT cases
Signed-off-by: Steven Zou <szou@vmware.com>
- add scanner rest API v1 spec
- implement v1 client which is used to talk to scanner adapter
- adjust data/orm models
- adjust code package structure
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan client which is used to talk to scanner adapter
- implement scan job which take the work of communicating with scanner
- update scanner mgmt API routes
- add corresponding UT cases
This commit modifies nginx configuration file to make sure the secure
flag is added to "Set-Cookie" header when Harbor is serving https
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- add DAO layer for scanner registration
- add CURD manager for scanner registration
- add API controller for plug scanner
- add REST APIs for CURD of plug scanner
- add migration sql:0011_1.10.0
- add scan interface definition (no implementations)
- add related UT cases with testify
fix#8979#8990
Signed-off-by: Steven Zou <szou@vmware.com>
