Commit Graph

81 Commits

Author SHA1 Message Date
Qian Deng
354a2bd80d Enhance the trace related code
* Move request id to requestid middleware
* fix span pass to child ctx on orm
* fix typos
* remove unused code
* add operation name to Transaction

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-22 04:14:43 +00:00
Qian Deng
31707dbf25 Using harbor lib config
using config manager to get env configs

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Qian Deng
bad913cf6d Refactor trace code
* use lib trace helper function
* add gracefull shutdown
* Add commens for new added exposed function
* Add licence on top of new created files
* Update trace library
* Update configs
* Add attribute and namespance in config

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
Qian Deng
a15983432c Add trace for core
* Add trace related lib
* Add trace middleware for core
* add rid for middleware

Signed-off-by: Qian Deng <dengq@vmware.com>
2021-09-18 10:58:52 +00:00
He Weiwei
ecc1a04c92
refactor: implement a lock free quota (#15399)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-09-17 11:26:37 +08:00
Junyoung, Sung
251feea1fd
Fix typo maxDeepth -> maxDepth (#15574)
Signed-off-by: Junyoung, Sung <junyoung.sung@naverlabs.com>
2021-09-13 21:23:31 +08:00
He Weiwei
d482a0c323
fix: avoid panic in the RetryUntil (#15501)
1. Use jpillora/backoff to get the backoff to avoid the panic in RetryUntil.
2. Return with last err when retry timeout.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-08-27 17:28:33 +08:00
stonezdj(Daojun Zhang)
7b84f4e137
Merge pull request #15270 from stonezdj/21jun05_fix_rest_int_parse_fail
Fix invalid syntax errors when int value is convert to scientific notation
2021-07-11 21:00:06 +08:00
Wang Yan
b158086642
fix conformance failure (#15261)
fixes #15252

Give 404 for invalid digest when to get/head manifest

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-07-06 15:11:13 +08:00
stonezdj
41968091a2 Fix invalid syntax errors when int value is convert to scientific notation
Add parseInt function to parse the scientific notation

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-07-05 22:20:00 +08:00
stonezdj(Daojun Zhang)
08d47f471a
Move genyaml to cmd/swagger (#15062)
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-06-29 14:29:03 +08:00
Wenkai Yin
9ebd517c7e Add validation for destination namespace when creating/updating replication policy
Add validation for destination namespace when creating/updating replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-06-10 22:16:05 +08:00
He Weiwei
f6a1c31f9f refactor: convert scan report in scan job
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-05-29 12:19:36 +00:00
Daniel Jiang
12420ae31d
Merge pull request #14957 from stonezdj/21may24_update_configure
Change configurations in swagger.yaml make it more readable
2021-05-25 23:58:40 +08:00
stonezdj
fe830ac4d2 Change configurations in swagger.yaml make it more readable
Update the genyaml to generate the configurations type
Add toCfgMap in config handler

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-05-24 19:13:55 +08:00
Wenkai Yin(尹文开)
544fa806fa
Set null list when or list is nil in query (#14941)
Set null list when or list is nil in query

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-24 16:22:39 +08:00
Wenkai Yin
5ebc8b4fdd Truncate the string when deletinng the resources when the length exceeds the limit
Truncate the string when deletinng the resources when the length exceeds the limit

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-18 17:17:41 +08:00
He Weiwei
6f3607cebd
perf: cache the metadata of the scanner (#14879)
1. Cache the metadata of scanner 30s.
2. Change the scanner client request timeout to 5s.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-05-14 14:27:37 +08:00
Moshe Immerman
d1b553fd3a
feat: add ORM_DEBUG env option (#14709)
Signed-off-by: Moshe Immerman <moshe@flanksource.com>
2021-05-12 10:56:30 +08:00
stonezdj(Daojun Zhang)
a6d92ca807
Merge pull request #14777 from stonezdj/21apr15_declarative_config
Implement declarative configure feature
2021-05-12 09:21:57 +08:00
stonezdj
3825220ca6 Implement declarative configure feature
Add env CONFIG_OVERWRITE_JSON for declarative config
Init config with the json in CONFIG_OVERWRITE_JSON in main.go

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-05-11 20:58:19 +08:00
Wenkai Yin(尹文开)
e006f4bab5
Support specifying multiple fields for default sorting (#14788)
Support specifying multiple fields for default sorting
Fixes #14433

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-10 17:04:10 +08:00
stonezdj
9010eeb832 Add the ldap group config validation
When ldap_group_search_filter is set, check other required fields
When ldap_group_search_filter is empty, skip to attach group

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-29 22:11:38 +08:00
stonezdj(Daojun Zhang)
0b9cad33b1
Refine the code of config (#14675)
Remove deps to common/dao
 Move Manager interface to config.go
 Remove duplicate code and change format of dao.go

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-19 21:08:25 +08:00
stonezdj
ef73c45681 review comments
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-14 12:26:27 +08:00
stonezdj
9807a5a9ff Merge branch 'master' into 21apr13_move_config_exp
# Conflicts:
#	src/common/dao/project/projectmember_test.go
#	src/core/api/projectmember.go
#	src/core/auth/ldap/ldap_test.go
#	src/server/v2.0/handler/project.go
2021-04-14 09:45:48 +08:00
stonezdj
751d404519 Refactor project member api to new programming model
Add pkg/member/dao
    Add pkg/member/models
    Add pkg/member/manager
    Add controller/member
    Remove the old project member API

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-13 21:28:54 +08:00
stonezdj
60478f4990 Move common config api to lib/config
Register all config managers, and get it by getConfigManger()

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-13 19:43:33 +08:00
Daniel Jiang
1d01db3d3c
Merge pull request #14604 from reasonerjt/users-api-refact-2
API for users to new model
2021-04-13 16:21:51 +08:00
Daniel Jiang
d4cd2b87bd API for users to new model
This commit moves the legacy apis related to users to new model.
Some funcs under common/dao are left b/c they are used by other module,
which should also be shifted to leverage managers.
We'll handle them separately.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-04-13 11:11:50 +08:00
Wenkai Yin
09c3d042ea Improve the performance of replication
Improve the performance of replication by introducing a new API to check whether the blob can be mounted directly

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-04-12 09:54:25 +08:00
stonezdj
107e468b60 Refactor configure api to new programming model
Changes include:
1. Move core/config to controller/config
2. Change the job_service and gcreadonly to depends on lib/config instead of core/config
3. Move the config related dao, manager and driver to pkg/config
4. Adjust the invocation of the config API, most of then should provide a context parameter, when accessing system config, you can call it with background context, when accessing user config, the context should provide orm.Context

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-04-09 08:10:11 +08:00
Qian Deng
c5d12ce8ee
Merge pull request #14542 from ninjadq/add_task_info_in_exporter
Add task info in exporter
2021-04-07 18:17:26 +08:00
DQ
fd62932a1f Enhance: Refactor metrics exporter
* refactor register logic
* remove unused code
* edit some wording

Signed-off-by: DQ <dengq@vmware.com>
2021-04-07 14:57:30 +08:00
Wenkai Yin(尹文开)
28596c3ffb
Refactor registry API (#14528)
* Refactor registry API

Refactor registry API

Signed-off-by: Wenkai Yin <yinw@vmware.com>

* Fix bugs of replications

1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-31 15:49:23 +08:00
DQ
fbe9cd88f8 Enabled Prometheus for Jobservice
* Add prom server on jobservice
* Enabeld configs in templates
* Enabeld jobservice metrics in nginx

Signed-off-by: DQ <dengq@vmware.com>
2021-03-30 08:52:59 +00:00
Wenkai Yin
506d1ad465 Introduce "sort" in query to provide a general solution for sorting
Introduce "sort" in query to provide a general solution for sorting

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-11 08:25:49 +08:00
He Weiwei
9161a3fbdf
refactor: use ormer from the ctx for scanner ctl mgr and dao (#14313)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-03-01 12:02:40 +08:00
Wenkai Yin
fe0216ce94 Fix sql issue for querying tasks
Fix sql issue for querying tasks

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-02-09 22:15:06 +08:00
He Weiwei
a0561da5ce
test: fix the TestRetryUntil (#14210)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-02-09 13:25:22 +08:00
He Weiwei
44ba7de738
fix: wrap report vuls record creating in transaction (#14176)
Make the creating of the ReportVulnerabilityRecord in transaction to
avoid parallel problem

Closes #14171

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-02-05 12:15:52 +08:00
DQ
28ae77e5c6 Aggregate metrics
1. Add operation id in ctx in baseapi before prepare
2. add operation id for registry proxy request
3. use url for other request

Signed-off-by: DQ <dengq@vmware.com>
2021-01-25 09:59:10 +08:00
Daniel Jiang
efa63d905a
Update the reg-exp to match v2/catalog api (#13941)
A more strict check is applied such that all requests to
/v2/_catalog/...  will be verified.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-09 13:17:51 +08:00
Daniel Jiang
3b04d2f8f5
Escape the values to contains operator in dao packages (#13774)
fixes #13018

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-16 14:19:20 +08:00
He Weiwei
10e4350360
test(cache): increase sleep time to fix ut (#13761)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 19:22:42 +08:00
He Weiwei
08580f9fec
refactor(scan): refactor scan/scan all job to task manager (#13684)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 13:34:35 +08:00
He Weiwei
e92674a42a
feat: add cache library and enable it in config manager (#13525)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-08 17:40:03 +08:00
Daniel Jiang
73c3f7c7ea Fix the pattern to match v2 catalog URI
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-03 12:53:04 +08:00
Wenkai Yin
294385c34d Refactor the replication execution
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-11-23 14:24:10 +08:00
DQ
eb470501be Add metrics to Harbor Core
1. Add configs in prepare
 2. Add models and config items in Core
 3. Encapdulate getting metric in commom package
 4. Add a middleware for global request to collect 3 metrics

Signed-off-by: DQ <dengq@vmware.com>
2020-11-03 14:33:10 +08:00