It's a workaround for issue https://github.com/goharbor/harbor/issues/11754
The phenomenon is the repository data is gone, but artifacts belong to the repository are still there.
To resolve it, just set the repository_id to a negative, and cannot duplicate.
Signed-off-by: wang yan <wangyan@vmware.com>
Fixes#11624
All of the existing policies created v1.10 has no name, it fails the upgrade process.
When to set the unique constraint for policy name, the empty can be seen as duplicated key.
ERROR: could not create unique index "notification_policy_name_key"
DETAIL: Key (name)=() is duplicated.
Signed-off-by: wang yan <wangyan@vmware.com>
Fixes#11241
1, remove count quota from quota manager
2, remove count in DB scheme
3, remove UI relates on quota
4, update UT, API test and UI UT.
Signed-off-by: wang yan <wangyan@vmware.com>
1, loop each access log, change to resource/resource_type, and insert into audit log
2, loop each first push operation, change it to create repository and insert into audit log.
Signed-off-by: wang yan <wangyan@vmware.com>
1, add API entry for get audit logs
2. add audit log manager to hanlder CRUD
Use the new format of audit log to cover differernt resource, artifact/tag/repostory/project
Signed-off-by: wang yan <wangyan@vmware.com>
As we store the repository name in the artifact table, we can use it direclty in the code to reduce the database query
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.
Signed-off-by: wang yan <wangyan@vmware.com>
This commit add supporting for adding/removing label to/from artifacts and populates labels when listing artifacts
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1. Define the controller/manager interface for artifact and tag
2. Provide a null implementation for artifact manager
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commits does some basic init work for supporting OCI:
1. Create the artifact and tag model
2. Create database tables
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit add an attribute to configurations, whose value is the
certificate of authproxy server. When this attribute is set Harbor will
pin to this cert when connecting authproxy.
This value will also be part of the response of systemInfo API.
This commit will be cherrypicked to 1.10 and 1.9 branch.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
- update scan all job to avoid sending too many HTTP requets
- update scan controller to support scan options
- update the db schema of the scan report to introduce requester
- introduce scan all metrics to report the overall progress of scan all job
- fix the status updating bug in scan report
- enhance the admin job status updats
- add duplicate checking before triggering generic admin job
- update the db scheme of admin job
fix#9705fix#9722fix#9670
Signed-off-by: Steven Zou <szou@vmware.com>
- do changes to the sql schema
- add `UseInternalAddr` and `Immutable` properties to scanner registration
- support multiple authentication type
- basic
- bearer token
Signed-off-by: Steven Zou <szou@vmware.com>
- update the scan/scanner controller
- enhance the report summary generation
- do changes to the vulnerable handler
- remove the unused clair related code
- add more UT cases
- update the scan web hook event
- drop the unsed tables/index/triggers in sql schema
Signed-off-by: Steven Zou <szou@vmware.com>
- refactor the db schema \
- refactor permission checking in API handlers \
to follow the latest code/interface changes
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan controller
- add scan resource and update role bindings
- update registration model and related interfaces
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan API to do scan/get report/get log
- update repository rest API to produce scan report summary
- update scan job hook handler
- update some UT cases
- update robot account making content
- hidden credential in the job log
Commnet scan related API test cases which will be re-activate later
fix#8985
fix the issues found by codacy
Signed-off-by: Steven Zou <szou@vmware.com>
- change error collection in scan job
- add dead client checking in client pool
- change key word type to interface{} for q.Query
- update bearer authorizer
- add required UT cases
Signed-off-by: Steven Zou <szou@vmware.com>
- add scanner rest API v1 spec
- implement v1 client which is used to talk to scanner adapter
- adjust data/orm models
- adjust code package structure
Signed-off-by: Steven Zou <szou@vmware.com>
- implement scan client which is used to talk to scanner adapter
- implement scan job which take the work of communicating with scanner
- update scanner mgmt API routes
- add corresponding UT cases
- add DAO layer for scanner registration
- add CURD manager for scanner registration
- add API controller for plug scanner
- add REST APIs for CURD of plug scanner
- add migration sql:0011_1.10.0
- add scan interface definition (no implementations)
- add related UT cases with testify
fix#8979#8990
Signed-off-by: Steven Zou <szou@vmware.com>
Signed-off-by: wang yan <wangyan@vmware.com>
Add dao for quota
Signed-off-by: He Weiwei <hweiwei@vmware.com>
fix govet
Signed-off-by: wang yan <wangyan@vmware.com>
In v1.7.0, it creates an record for scan all but without cron string, just update the record with the cron in properties.
Signed-off-by: wang yan <wangyan@vmware.com>
* add scan all and gc schedule migration
Signed-off-by: wang yan <wangyan@vmware.com>
* Fix gofmt errors
Signed-off-by: wang yan <wangyan@vmware.com>
* Update code according to review comments
Signed-off-by: wang yan <wangyan@vmware.com>
* remove convertschedule return name just return value
Signed-off-by: wang yan <wangyan@vmware.com>
1. handle the public/private property when creating the projects
2. extend the length of access_secret
3. update the task status by using orm functions
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit introduces a solution to workaround the restriction of project creation API: only normal users can create projects
Signed-off-by: Wenkai Yin <yinw@vmware.com>
As CLI does not support oauth flow, we'll use secret for help OIDC user
to authenticate via CLI.
Add column to store secret and token, and add code to support
verify/refresh token associates with secret. Such that when the user is
removed from OIDC provider the secret will no longer work.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Add operation property for tasks
2. Add trigger property for executions
3. Update the getting registry info API to allow passing 0 as ID to get the info of local Harbor registry
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit updates the upgrade sql script to fix#6698: cannot recreate the same name replication policy after it is deleted
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit is to make the expiration of robot account configurable
1, The expiration could be set by system admin in the configuation page or
by /api/config with robot_token_expiration=60, the default value is 30 days.
2, The expiration could be shown in the robot account infor both on UI and API.
Signed-off-by: wang yan <wangyan@vmware.com>
* Support master role for project member create and update apis
Signed-off-by: He Weiwei <hweiwei@vmware.com>
* Fix description for role_id in swagger.yaml
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Rename the "name" colume in table "replication_policy" and "replication_target" before adding the "UNIQUE" constraint to avoid the upgrade failure
Signed-off-by: Wenkai Yin <yinw@vmware.com>
1. Fix#5102 by checking the existence of name when creating/editing replication rule
2. Add unique constraint to the name of replication policy and target
3. Fix bugs of testing library
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Fixes#6115
As for the change in migration sql file, in 1.7 we'll switch to
jobservice for scheduling "scan all" job. To avoid inconsistency,
this item will be reset and user will need to configure the policy again.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Root cause: Use default 'now'::timestamp will not generate timestamp for each transaction,
PG will convert now to a timestamp as soon as the constant is parsed. To fix it, update it
to defult CURRENT_TIMESTAMP, thie setting is the same as default now(), which returns the
start time of current transaction because ther are fuction calls, hey will give the desired
behavior of defaulting to the time of row insertion.
Reference: https://www.postgresql.org/docs/9.6/static/functions-datetime.html#FUNCTIONS-DATETIME-CURRENT
PG version: 9.6.9
It supports Harbor admin to trigger job either manual or
schedule. The job will be populated to job service to execute.
The api includes:
1. POST /api/system/gc
2, GET /api/system/gc/:id
3, GET /api/system/gc/:id/log
4, PUT/GET/POST /api/system/gc/schedule
This commit fixes#5040, the harbor-db image will only contain empty
databases, and harbor ui container will use migrate tool to run initial
SQL scripts to do initialization. This is helpful for the case to
configure Harbor against external DB or DBaaS like RDS for HA deployment
However, this change will results some confusion as there are two tables
to track schema versions have been using alembic for migration, for this
release we'll try to use alembic to mock a `migration` table during
upgrade so the migrator will be bypassed, in future we'll consider to
consolidate to the golang based migrator.
Another issue is that the UI and adminserver containers will access DB
after start up in different congurations, can't ensure the sequence, so
both of them will try to update the schema when started up.
