Commit Graph

111 Commits

Author SHA1 Message Date
Chlins Zhang f21b1481bb
refactor: refactor the old goroutine execution sweep (#18361)
refactor: refactor the old goroutine execution sweep to global execution sweep job

1. Delete the old goroutine execution sweeper when create execution.(in the case of high concurrency can cause goroutine backlogs, affect the performance of core)
2. Introduce the new way to sweep executions, a global scheduled job will take the work.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-21 10:10:55 +08:00
Wang Yan bb291aaa16
add middleware for artifact with subject (#18369)
As for the distribution spec 1.1, it supports client to push an manifest with subject field. By leverging this fidle, harbor could build up the linkage between the subject artifact and it's accessories.

Signed-off-by: wang yan <wangyan@vmware.com>
2023-03-19 20:09:41 +08:00
Wang Yan 277c78e212
update system artifact schedult timeout (#17902)
fixes #17887
enhance the timeout to 5m to avoid system artifact failed to schedule in the helm base installation.

Signed-off-by: Wang Yan <wangyan@vmware.com>

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-12-02 14:30:05 +08:00
Wang Yan 18a3373725
bump beego (#17801)
* bump beego

upgrade beego version from v1.10.12 to v2.0.5

1, beego v2 vserver/web refactor
2, beego v2 context refactor
3, beego v2 session refactor
4, beego v2 cache refactor
5, beego v2 orm refactor

Signed-off-by: MinerYang <yminer@vmware.com>
2022-11-24 18:07:42 +08:00
Chlins Zhang 1a1a6d2fd2
feat: log completion message when database migrated (#17684)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-10-31 10:45:05 +08:00
MinerYang f7ebf20d2e
Disable Nydus middleware for v2.6 (#17233)
disable nydus middleware for v2.6

Signed-off-by: yminer <yminer@vmware.com>

remove nyduus module import in main.go

Signed-off-by: yminer <yminer@vmware.com>
2022-07-27 08:41:33 +08:00
Loong Dai 09371b48e8
lint: sort imports (#17131)
* lint: add goimports

Signed-off-by: Loong Dai <loong.dai@intel.com>
2022-07-20 11:33:08 +08:00
MinerYang efd9632e96
add nydus middleware (#17126)
Signed-off-by: yminer <yminer@vmware.com>

remove comments

Signed-off-by: yminer <yminer@vmware.com>

update ut manifest

Signed-off-by: yminer <yminer@vmware.com>

modify comment manifest

Signed-off-by: yminer <yminer@vmware.com>

updtae ut testcase

Signed-off-by: yminer <yminer@vmware.com>

fixwhitespace lint

Signed-off-by: yminer <yminer@vmware.com>

update isNydus judgement && define annotation var

Signed-off-by: yminer <yminer@vmware.com>

whitespace lint

Signed-off-by: yminer <yminer@vmware.com>
2022-07-08 09:56:10 +08:00
Shengwen YU 315d8a8993
fix: update code in compliance with golangci-lint revive (#17087)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-07-01 14:52:47 +08:00
Wang Yan 94bf1c0ebd
Resovle the system artifact schedule issue (#17098)
Scheduling of system artifact depends on the jobservice, where gorountine is used to avoid the circular dependencies between core and jobservice.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-07-01 14:25:45 +08:00
stonezdj(Daojun Zhang) e778ec2edf
Add audit_log forward endpoint (#16914)
add config item to set log forward endpoint
  fallback the audit log to default log when endpoint in error

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-06-10 10:59:40 +08:00
prahaladdarkin 4d062c33d1
Introduce system artifact manager cleanup job (#16879)
Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-06-09 14:16:50 +08:00
Shengwen YU b43ba15f40
fix: golangci-lint errcheck (#16920)
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-06-07 17:00:36 +08:00
MinerYang 083b44e719
bump up astaxie/beego@v1.12.1 to beego/beego/@v1.12.7 (#16770)
Signed-off-by: yminer <yminer@vmmware.com>

goo mod tidy

Signed-off-by: yminer <yminer@vmmware.com>

Co-authored-by: yminer <yminer@vmmware.com>
2022-04-28 11:58:44 +08:00
chlins dbb8859ad8 feat: implement beego session provider
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-03-24 10:33:21 +08:00
stonezdj 478d21687b Improve the migrate process
In order to separate the migrate process from core process, add parameter -mode for main function, it can be normal, migrate, skip_migrate.
Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-02-14 13:15:18 +08:00
stonezdj 21601f5e3c Enable db config cache explicitly
previous config is not cached because it is initialized when cache is not ready

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-11-04 16:01:25 +08:00
stonezdj 06715af303 Allow empty email attribute for ldap/oidc user
Define user.Email as sql.NullString to avoid unique constraint when email is empty in LDAP/OIDC
  Separate the common/models/User with the pkg/user/dao/User
  Fixes #10400
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-09-30 14:35:18 +08:00
Qian Deng 31707dbf25 Using harbor lib config
using config manager to get env configs

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Qian Deng bad913cf6d Refactor trace code
* use lib trace helper function
* add gracefull shutdown
* Add commens for new added exposed function
* Add licence on top of new created files
* Update trace library
* Update configs
* Add attribute and namespance in config

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Qian Deng a15983432c Add trace for core
* Add trace related lib
* Add trace middleware for core
* add rid for middleware

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Daniel Jiang 422894a0f5 Remediate the empty subiss problem
This commit include 2 changes to mitigate and remediate the problem
described in #15241

1. When the token is to be updated in the "oidc_user" table, make sure
   only the column "token" will be udpated.
2. Restore the subiss column for the record that has this column cleared
   by mistake, by decoding the persisted token.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-07-16 00:07:47 +08:00
Daniel Jiang 952644e23f Remove GetUser and Onboard from common/dao
Replaced by funcs in src/pkg/user and src/controller/user

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-05-18 15:28:41 +08:00
stonezdj(Daojun Zhang) 0dfc801a50
Error string should not be capitalized (#14840)
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-05-12 16:53:37 +08:00
Daniel Jiang bd9a1c6722
Merge pull request #14829 from reasonerjt/user-common-dao-cleanup
Move user related funcs from common/dao
2021-05-12 10:56:18 +08:00
Daniel Jiang 6d0e391740 Move user related funcs from common/dao
This commit moves more user related funcs, such as ChangePassword,
Login, ChangeUserProfile from common/dao to rely on /pkg/user and
pkg/oidc.
It also removes the code for resetting user's password as it's disabled.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-05-12 01:12:57 +08:00
stonezdj 3825220ca6 Implement declarative configure feature
Add env CONFIG_OVERWRITE_JSON for declarative config
Init config with the json in CONFIG_OVERWRITE_JSON in main.go

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-05-11 20:58:19 +08:00
Wenkai Yin 654f4d9202 Refactor health check API
Refactor the health check API

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-04-15 14:50:27 +08:00
stonezdj 60478f4990 Move common config api to lib/config
Register all config managers, and get it by getConfigManger()

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-13 19:43:33 +08:00
stonezdj 107e468b60 Refactor configure api to new programming model
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-09 08:10:11 +08:00
Wenkai Yin(尹文开) 28596c3ffb
Refactor registry API (#14528)
* Refactor registry API

Refactor registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Fix bugs of replications

1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-31 15:49:23 +08:00
He Weiwei 9161a3fbdf
refactor: use ormer from the ctx for scanner ctl mgr and dao (#14313)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-03-01 12:02:40 +08:00
He Weiwei e92674a42a
feat: add cache library and enable it in config manager (#13525)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-08 17:40:03 +08:00
DQ 590212b485 Remove clair related code
- clair code in harbor core
- clair code in frontend
- clair code in robotcase

Signed-off-by: DQ <dengq@vmware.com>
2020-11-27 14:01:04 +08:00
DQ eb470501be Add metrics to Harbor Core
1. Add configs in prepare
 2. Add models and config items in Core
 3. Encapdulate getting metric in commom package
 4. Add a middleware for global request to collect 3 metrics

Signed-off-by: DQ <dengq@vmware.com>
2020-11-03 14:33:10 +08:00
He Weiwei f4938ca966
fix(scanner): move RemoveImmutableScanners before EnsureScanners (#12957)
RemoveImmutableScanners may delete the default scanner, so move it
before the EnsureScanners.

Closes #12938

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-09-03 11:11:16 +08:00
Steven Zou 46d7434d0b
Merge pull request #12473 from ywk253100/200706_scheduler
Refactor the scheduler with the task manager mechanism
2020-07-20 15:53:14 +08:00
Wenkai Yin 4dc4b6728c Refactor the scheduler with the task manager mechanism
Refactor the scheduler with the task manager mechanism, this will reduce the duplicate code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-20 14:03:15 +08:00
Ziming Zhang 8857e89e40 feature(redis) support redis sentinel
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-07-19 21:19:03 +08:00
He Weiwei 039aef5356 refactor: remove initialization of clair db
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-09 15:26:14 +00:00
Wenkai Yin ea20690264 Implement task and execution manager
Implement task and execution manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-23 17:10:58 +08:00
He Weiwei 385aaac00d
Merge pull request #11620 from heww/fix-issue-11524
feat(scanner): make Clair and Trivy as reserved name for scanners
2020-04-15 15:21:35 +08:00
He Weiwei f5487479dd feat(scanner): make Clair and Trivy as reserved name for scanners
Closes #11524

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-04-15 02:26:02 +00:00
DQ b3db293091 TLS update min version and cipher suits
min version set to tls 1.2
suit only use ecdhe and strenth above 256

Signed-off-by: DQ <dengq@vmware.com>
2020-04-13 18:13:30 +08:00
wang yan a11a70d941 move logger from common to lib
The logger is the fundamental library, so move it into lib folder
Signed-off-by: wang yan <wangyan@vmware.com>
2020-04-02 14:09:03 +08:00
Wenkai Yin(尹文开) 8984979bd2
Relocate/rename some packages (#11183)
Fixes #11016
1. src/pkg/q->src/internal/q
2. src/internal->src/lib (internal is a reserved package name of golang)
3. src/api->src/controller

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-24 20:45:45 +08:00
Wenkai Yin 0453709b74 Rewrite the filters with middleware mechinism
Fixes 10532,rewrite the filters with middleware mechinism

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-23 09:26:20 +08:00
DQ b93092e012 Add tls for trivy
Add trivy tls cert files
Add tivey tls env and config
enhance gencert

Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:10 +08:00
DQ 115185894f Merge internal Transport and Secure Transport
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00
DQ dcc6950af7 Feat: auto install ca in registry
Signed-off-by: DQ <dengq@vmware.com>
2020-03-18 19:22:09 +08:00