fixes#15332, for the dry run mode, gc job should not remove the untagged candidates.
To fix it, use the simulate untagged artifact deletion for dry-run.
Signed-off-by: Wang Yan <wangyan@vmware.com>
It takes about 1 hour to perform data for 40000 repositories per performance testing.
Roll back the runner count to 100 could speed the data preparation time.
It's safe since it only takes 100 DB connection counts at most per execution per core.
Signed-off-by: Wang Yan <wangyan@vmware.com>
fixes#15736
For the current imple, the GetWorker() may hang when there is no worker available, and will not release the DB connection.
In this case, the DB connection could reach the up limit that leads to harbor core for service unavailable.
1, move GetWorker() in the goroutine, release the DB connection for API.
2, reduce the worker count per harbor-core from 1024 to 10.
3, reduce the runner count per worker to 30.
After above, the max connection per harbor-core should be 300.
Worker: To control how many replicaiton exectuions can have at most at the same time.
Runner: To control the speed to generate an jobservice replicaiton job.
Signed-off-by: Wang Yan <wangyan@vmware.com>
Define user.Email as sql.NullString to avoid unique constraint when email is empty in LDAP/OIDC
Separate the common/models/User with the pkg/user/dao/User
Fixes#10400
Signed-off-by: stonezdj <stonezdj@gmail.com>
* Move request id to requestid middleware
* fix span pass to child ctx on orm
* fix typos
* remove unused code
* add operation name to Transaction
Signed-off-by: Qian Deng <dengq@vmware.com>
This commit adjust the priority when consolidating data from userinfo
endpoint and id token, making sure the auto-onboard username claim from
ID token has highest priority.
fixes#15504
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* Refactor common http GetTransport function signature
* Remove redendent GetHTTPTransport and similar functions
* Update Authorized function signature to meet new HTTPTransport
* Add trace for default Transport
Signed-off-by: Qian Deng <dengq@vmware.com>
- add a sync worker to sync db schedules when js starting. add missing ones and clear dirty ones.
- update task model to contain status revision info
- update job lifecycle tracker save() method
- update job ACK model
- add UT cases
- update malformat comments
fix#15323
Signed-off-by: Steven Zou <szou@vmware.com>
1. Use ctx from http request for the readonly middleware.
2. Refactor the AuthenticateHelper to let it get orm from ctx of the http request.
3. Change to use ctx from http request for oidc and authproxy http handlers.
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Fixes#15450
Add paging function to usergroup list/search API
Fix some 500 error when adding LDAP user/group to project member
Signed-off-by: stonezdj <stonezdj@gmail.com>
1, for admin only, the system level robot should contains the project creation access.
2, for not admin only, the system level robot can create project.
3, for the project that created by system level robot, use the admin ID as the ownerID.
No path for project level robot to create project.
Signed-off-by: wang yan <wangyan@vmware.com>
1, add permission check for API of List Projects
2, add permission check for API of List Repositories
3, use the self defined query to handle both names and public query
Signed-off-by: wang yan <wangyan@vmware.com>
update the jwt model to github.com/golang-jwt/jwt
Starting from v3.2.1, the import path has changed from github.com/dgrijalva/jwt-go to github.com/golang-jwt/jwt.
Signed-off-by: Wang Yan <wangyan@vmware.com>
This commit include 2 changes to mitigate and remediate the problem
described in #15241
1. When the token is to be updated in the "oidc_user" table, make sure
only the column "token" will be udpated.
2. Restore the subiss column for the record that has this column cleared
by mistake, by decoding the persisted token.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
The following information should cleanup before delete user:
Delete project member of this user.
Delete oidc_user when auth_mode is oidc_auth.
Fixes#8424
It also removes the deleted user from project member and the deleted condition in the project member query for consistency
Signed-off-by: stonezdj <stonezdj@gmail.com>
1. Limit API qps for the adapter.
2. Allow set qps via env.
3. Fix Tencnet SDK pagenation.
4. Fix resource filter.
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
QuerySeter.Delete will call two queies in the orm of beego. First it
will query all primary keys of the model by the QuerySeter, and then the
orm will delete the models using the IN statement of these primary keys.
This will be failed in the postgres when the records more than 65535.
This commit changes to use the raw query to delete the robots of the
project.
Closes#14678
Signed-off-by: He Weiwei <hweiwei@vmware.com>
Improve the performance of artifact related APIs by adding indexes and refactoring sql logic
Closes#13890#14813#14814
Signed-off-by: Wenkai Yin <yinw@vmware.com>
When the core service cannot response the checkin request in time, duplicated execution records may be created, this commit introduces the revision column to make sure there is only one record for one schedule trigger
Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit moves more user related funcs, such as ChangePassword,
Login, ChangeUserProfile from common/dao to rely on /pkg/user and
pkg/oidc.
It also removes the code for resetting user's password as it's disabled.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Add env CONFIG_OVERWRITE_JSON for declarative config
Init config with the json in CONFIG_OVERWRITE_JSON in main.go
Signed-off-by: stonezdj <stonezdj@gmail.com>
When ldap_group_search_filter is set, check other required fields
When ldap_group_search_filter is empty, skip to attach group
Signed-off-by: stonezdj <stonezdj@gmail.com>
Remove deps to common/dao
Move Manager interface to config.go
Remove duplicate code and change format of dao.go
Signed-off-by: stonezdj <stonezdj@gmail.com>
* Refactor labl api
move to the new program model
Signed-off-by: wang yan <wangyan@vmware.com>
* continue resolve review comments
Signed-off-by: Wang Yan <wangyan@vmware.com>
Add pkg/member/dao
Add pkg/member/models
Add pkg/member/manager
Add controller/member
Remove the old project member API
Signed-off-by: stonezdj <stonezdj@gmail.com>
This commit moves the legacy apis related to users to new model.
Some funcs under common/dao are left b/c they are used by other module,
which should also be shifted to leverage managers.
We'll handle them separately.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Improve the performance of replication by introducing a new API to check whether the blob can be mounted directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context
Signed-off-by: stonezdj <stonezdj@gmail.com>
This commit adds the attribute "http_authproxy_admin_usernames", which
is string that contains usernames separated by comma, when a user logs
in and the username in the tokenreview status matches the setting of
this attribute, the user will have administrator permission.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* Refactor registry API
Refactor registry API
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* Fix bugs of replications
1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy
Signed-off-by: Wenkai Yin <yinw@vmware.com>
