Commit Graph

1019 Commits

Author SHA1 Message Date
Wenkai Yin(尹文开)
4ef93565f3
Merge pull request #14369 from ywk253100/210303_sort
Introduce "sort" in query to provide a general solution for sorting
2021-03-11 09:28:34 +08:00
Wenkai Yin
506d1ad465 Introduce "sort" in query to provide a general solution for sorting
Introduce "sort" in query to provide a general solution for sorting

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-11 08:25:49 +08:00
stonezdj
5a35b7a9c4 Move ldap API to new program model
Fix some issue with the LDAP connection test

Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-03-10 16:26:45 +08:00
Daniel Jiang
e96c1cbced Switch API to ping OIDC endpoint to new model
This commit updates the API POST /api/v2.0/system/oidc/ping to new
programming model, in which the code will be generated by go-swagger.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-03-04 15:44:11 +08:00
He Weiwei
4b033c266a refactor: generate quota APIs by go-swagger
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-02-25 08:19:55 +00:00
piotrekfilip
db47cf7f46
Add support for http proxy in oidc insecureTransport
Signed-off-by: piotrekfilip <43957913+piotrekfilip@users.noreply.github.com>
2021-02-06 12:42:38 +01:00
Daniel Jiang
ea76594469 Improve the way config store transforms a value to string
This commit provide a better way to transform the value to string when
they are loaded from the driver.
Fixes #14074
However the way the config driver loaded config values and configstore
stores it back and forth seems repetitive and should be optimized.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-27 08:41:49 +00:00
Daniel Jiang
5ea43abc67 Fix a potential nil pointer issue
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-22 19:02:45 +08:00
Sven Haardiek
b2fe254974
Username from /userinfo (#14038)
This patch enabled Harbor to receive the username from the /userinfo endpoint
instead of only from the ID Token.

Closes #14037

Signed-off-by: Sven Haardiek <sven@haardiek.de>
2021-01-22 18:48:53 +08:00
Wang Yan
2d4456c630
refractor project rbac (#13924)
As the system rbac introduced, move the code of project rbac into project directory

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-11 11:27:26 +08:00
Ziming Zhang
39fb500318 feat(retention) refactor to use go swagger api
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-01-08 07:09:28 +00:00
He Weiwei
755c6490f9
feat: remove duplicate CVE in scan report and summary (#13918)
1. Remove the duplicate CVE records in the report/summary for the image
index.
2. Add scanner field in the scan overview for the API.

Closes #13913

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-01-08 11:00:43 +08:00
Wang Yan
0cf43d766c
enable system resource access (#13826)
1, introduce & define the system resources.
2, replace the IsSysAdmin judge method.
3, give the robot the system access capability.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-07 15:45:04 +08:00
Daniel Jiang
06e993ff76
Remove scanner-pull from system admin's permission (#13901)
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-06 12:35:58 +08:00
Daniel Jiang
d0152cb446
Merge pull request #13872 from reasonerjt/token-scope
Refine the token scope generation
2021-01-04 11:16:59 +08:00
Wang Yan
0271efd3f7
enable visible when to list/create robot (#13840)
1, enable the visible attribute when to create/list robots
2, rename package name from robot2 to robot

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-01-04 10:24:31 +08:00
Daniel Jiang
eb75123638 Refine the token scope generation
This commit directly maps the actoin permission in security context to
the scope generated by the token service in harbor-core.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2021-01-03 23:12:04 +08:00
Daniel Jiang
c660727877
Merge pull request #13800 from reasonerjt/authproxy-redirect
Add handler to handle redirect via authproxy
2020-12-23 03:00:18 +08:00
stonezdj(Daojun Zhang)
bc0b6b43ed
Merge pull request #13791 from reasonerjt/oidc-redirect-extra-parm
Add extra parms when forming redirect URI for OIDC
2020-12-22 21:45:53 +08:00
Daniel Jiang
7321e3547d Add handler to handle redirect via authproxy
This commit add a handler to handle the request to
"/c/authproxy/redirect".  Harbor is configured to authenticate against
an authproxy, if a request with query string `?token=xxxx`
is sent to this URI, the handler will do tokenreview according to the
setting of authproxy and simulate a `login` workflow based on the result
of token review.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-22 18:59:17 +08:00
He Weiwei
3831e82b20
refactor: remove code of admin job (#13819)
Remove code of admin job as it's not needed by scan all/gc now.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-22 11:48:16 +08:00
Wang Yan
9bc6f3cee4
fix robot account update issue (#13741)
* fix robot account update issue

enable the update method to support both v1 & v2 robot update

Signed-off-by: Wang Yan <wangyan@vmware.com>

* resolve review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 20:01:26 +08:00
He Weiwei
18b850782e fix: fix errors detected by codeql
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-17 06:15:34 +00:00
Daniel Jiang
c1c55d0cee Add extra parms when forming redirect URI for OIDC
Fixes #13092

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-16 19:41:13 +08:00
Daniel Jiang
3b04d2f8f5
Escape the values to contains operator in dao packages (#13774)
fixes #13018

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-16 14:19:20 +08:00
Wenkai Yin(尹文开)
43104ab0b9
Merge pull request #13724 from reasonerjt/http-auth-admin-grp
Support admin group in http authproxy
2020-12-11 13:06:26 +08:00
Daniel Jiang
60e3668d43 Support admin group in http authproxy
This commit adds admin_groups into the configuration of http_auth
settings, it's a string in the form of "group1, group2".  If the token
review result shows the user is in one of the groups in the setting he
will have the administrator role in Harbor.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-10 15:57:15 +08:00
DQ
ade69e20ef Fix typo
Signed-off-by: DQ <dengq@vmware.com>
2020-12-09 20:42:21 +08:00
He Weiwei
e92674a42a
feat: add cache library and enable it in config manager (#13525)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-08 17:40:03 +08:00
Daniel Jiang
fef5317aef
Merge pull request #13382 from flaviodsr/fix_core_init
Fix deadlock on harbor-core initialization
2020-12-04 19:51:56 +08:00
Wenkai Yin(尹文开)
ddb29f2243
Set timezone as UTC for database connection (#13661)
Set timezone as UTC for database connection

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-12-03 08:55:48 +08:00
Daniel Jiang
34d776b062 Bump up go-ldap to v3.2.4
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-02 15:28:54 +08:00
Qian Deng
b80b1a7abf
Merge pull request #13617 from ninjadq/remove_clair_in_harbor_code
Remove clair code in harbor
2020-11-30 15:28:02 +08:00
Daniel Jiang
db8ce49133
Rework systeminfo API. (#13606)
This commit rework the systeminfo API under new programming model.
Also fixes #9149

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-11-30 14:15:18 +08:00
DQ
590212b485 Remove clair related code
- clair code in harbor core
- clair code in frontend
- clair code in robotcase

Signed-off-by: DQ <dengq@vmware.com>
2020-11-27 14:01:04 +08:00
stonezdj(Daojun Zhang)
dec12308a1
Merge pull request #13621 from stonezdj/201127_fail_to_add_ldap_group
Lowercase the LDAP DN in UnderBaseDN
2020-11-27 11:45:07 +08:00
stonezdj(Daojun Zhang)
5a34f4e8fa
Merge pull request #13548 from wy65701436/robot2-swagger-api-dev
add robot account 2 api handler
2020-11-27 11:32:24 +08:00
stonezdj
ca245d3545 Lowercase the LDAP DN in UnderBaseDN
Fixes #13362: Unable to add LDAP group with different letter case in DN

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-11-27 10:30:19 +08:00
Daniel Jiang
fd54a568d0
Merge pull request #13500 from thechristschn/fix-searchgroupbyname
Fix api ldap group search by name
2020-11-26 16:38:13 +08:00
Wang Yan
02846194e0 parent 8e61a3ea31
author Wang Yan <wangyan@vmware.com> 1605849192 +0800
committer Wang Yan <wangyan@vmware.com> 1606361046 +0800

update code per review comments

Signed-off-by: wang yan <wangyan@vmware.com>
2020-11-26 14:10:12 +08:00
Christian Baumann
4530e9feee Fix api ldap group search by name
Signed-off-by: Christian Baumann <thechristschn@gmail.com>
2020-11-12 23:01:02 +00:00
He Weiwei
76f1afbe0d
refactor: remove core/promgr pkg (#13408)
* refactor: remove core/promgr pkg

Remove `core/promgr` package and use `controller/project` instead of it.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-11-12 15:33:13 +08:00
Wang Yan
62208dc3e4 fix code conflict
Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-11-03 09:42:37 -08:00
Wang Yan
5a22019e3d add robot name prefix
The system admin can set the prefix in configuration UI.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-11-02 23:12:57 -08:00
DQ
eb470501be Add metrics to Harbor Core
1. Add configs in prepare
 2. Add models and config items in Core
 3. Encapdulate getting metric in commom package
 4. Add a middleware for global request to collect 3 metrics

Signed-off-by: DQ <dengq@vmware.com>
2020-11-03 14:33:10 +08:00
Daniel Jiang
535728d11f
Merge pull request #13306 from heww/refactor-security-context
refactor(security): use controller instead of promgr in security
2020-10-29 02:39:59 +08:00
Flávio Ramalho
ef6414be3e
Fix deadlock on harbor-core initialization
During the harbor core initialization if the database takes longer to
be ready there is a risk of deadlock when checking for the TCP connection
with the database.

The `TestTCPConn` function uses unbuffered channels to check when the
connection succeeds/timeouts. The timeout check is executed in parallel
with the connection check (this runs in a gorountine). The deadlock happens
when the goroutine execution takes longer than the function timeout
(hence setting `cancel <- 1`) and the DialTimeout call succeeds (hence
setting `success <- 1`). At this point both threads are waiting for the
channels values to be read.

This is reproducible mostly on slow systems where initializing the
database takes longer and finishes during the 5th time of the
`DialTimeout` call where it eventually exceeds the TestTCPConn timeout.

This fix sets the `success` and `cancel` channels as buffered
(non-blocking).

Signed-off-by: Flávio Ramalho <framalho@suse.com>
2020-10-28 17:09:34 +01:00
Daniel Jiang
9c1da3a405 Add more info in log message to help debug
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-10-22 18:55:01 +08:00
Daniel Jiang
eadb65f988
Merge pull request #13312 from reasonerjt/oidc-admin-group
Add admin group support to OIDC auth mode
2020-10-22 18:30:10 +08:00
Daniel Jiang
649c9814e4 Address review comment by Yan
Resolve review comment in PR #13312

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-10-22 16:53:37 +08:00
stonezdj
ca7258617b Remove dup call to createGroupSearchFilter in searchGroup
Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-10-20 19:07:33 +08:00
Daniel Jiang
f4ff369ed0 Add admin group support to OIDC auth mode
Add oidc_admin_group to configuration, and make sure a token with the
group name in group claim has the admin authority.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-10-20 15:38:58 +08:00
He Weiwei
ea0fbbeace refactor(security): use controller instead of promgr in security
Use `project.Controller` instead of `promgr.ProjectManager` in security
implementations because we will remove `promgr` package later.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-10-19 10:33:51 +00:00
stonezdj
20ef7d3219 Refine LDAP searchGroup function
Search LDAP group with groupDN+filter, then match baseDN
Create a default filter when ldap group filter is empty
Fixes #13156

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-10-16 16:20:41 +08:00
Wenkai Yin(尹文开)
4d78fd4e4e
Merge pull request #13126 from Thoro/fix-10913-oidc-error-after-restart
fix #10913: initialize oidc provider before calling Load
2020-10-14 10:13:52 +08:00
Thomas Rosenstein
874b0b1c0c fix #10913: initialize provider before calling Load
Signed-off-by: Thomas Rosenstein <thomas.rosenstein@creamfinance.com>
2020-09-30 17:25:29 +02:00
Thomas Rosenstein
452a0c9c45 Add error log in case encryption on config save fails
Signed-off-by: Thomas Rosenstein <thomas.rosenstein@creamfinance.com>
2020-09-22 10:09:47 +02:00
Daniel Jiang
354eaac195
Escape the query string in list user (#13013)
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-09-10 20:17:53 +08:00
stonezdj(Daojun Zhang)
97b9cc2d5e
Merge pull request #12997 from stonezdj/200902_ldap_filter_fail
Add ldap filter syntax validation when create search filter
2020-09-09 16:21:01 +08:00
stonezdj
b9752f3112 Add ldap filter syntax validation when create search filter
Correct ldap search filter is enclosed with '(' and ')'
Search ldap group with the ldap group base DN instead of group DN
Fixes #12613 LDAP Group Filter and Group Base DN have no affect

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-09-09 10:07:07 +08:00
Daniel Jiang
513c48d47c
Merge pull request #12936 from wy65701436/fix-swagger-dep
remove the dependency on swagger models
2020-09-08 18:14:42 +08:00
He Weiwei
6d50988c8b fix(project): change to use user id to query projects of member
We know the user id when query projects by member, so use the user id
as entity_id directly in project_member, no need to join harbor_user
table.

Closes #12968

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-09-07 06:35:26 +00:00
He Weiwei
41c839af88 feat(project): ignore enable_content_trust for proxy project
Ignore enable_content_trust metadata for proxy cache project, see
https://github.com/goharbor/harbor/issues/12940 to get more info

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-09-02 15:24:45 +00:00
wang yan
116d295462 remove the dependency on swagger models
1, remove the pkg dependency on v2.0/server/models
2, remove the controller dependency on v2.0/server/models

Signed-off-by: wang yan <wangyan@vmware.com>
2020-09-01 17:43:44 +08:00
Wenkai Yin(尹文开)
3abe8b8fab
Merge pull request #12456 from julienvey/fix-sql-typo
Fix typo in sql log
2020-08-28 10:04:27 +08:00
Ted Guan
645dea36a6
Fix for duplicate webhook policy name (#12729)
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-08-20 18:02:13 +08:00
wang yan
1cc73bd92a Merge branch 'master' of https://github.com/goharbor/harbor into fix-resource-order 2020-08-19 12:21:45 +08:00
wang yan
648b80bc34 udpate resource list order
1, order label by creation time.
2, order webhook policy by creation time.
3, order replication policy by creation time.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-08-19 11:20:31 +08:00
He Weiwei
ef37bd1afb refactor(scan): remove duplicate CVESet types
Closes #9471

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-18 06:33:17 +00:00
He Weiwei
f309896f2f refactor(api): generate project apis by go-swagger
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-08-15 16:10:57 +00:00
Wenkai Yin(尹文开)
e8f9fb63c0
Merge pull request #12674 from reasonerjt/standalone-db-migrator
Provide a standalone migrator to migrate DB schema.
2020-08-10 15:11:52 +08:00
Wenkai Yin
219b9910eb Show the detail error message when failed to fetch the artifacts during replication
Show the detail error message when failed to fetch the artifacts during replication

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-08-07 18:31:02 +08:00
Daniel Jiang
4f94f59d2a Provide a standalone migrator to migrate DB schema.
Fixes #11885
This part will not by default be packaged into release.
A README.md will be added in another commit.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-08-06 18:57:55 +08:00
He Weiwei
df1bdc1020 refactor(project): add more methods to project controller and manager
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-31 17:55:35 +00:00
Wang Yan
e14e6938da
add gc read only job (#12591)
Add the read only job as a back up plan, user still can use it but just with API, and specify the parameter read_only:true

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-30 15:30:52 +08:00
stonezdj(Daojun Zhang)
518a1721a7
Merge pull request #12571 from ywk253100/200723_proxy_cache_secret
Limit the permission of secret used by proxy cache service
2020-07-30 14:04:54 +08:00
stonezdj(Daojun Zhang)
5b1b94f25c
Merge pull request #12512 from ywk253100/200717_summary
Include the registry info in the project summary API for proxy cache project
2020-07-29 14:39:30 +08:00
stonezdj
7d97ae6ea2 Set LDAP groupname when PopulateGroup
Search ldap group name with default ldap group attribute name
fixes #10940

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-07-28 10:27:15 +08:00
Wenkai Yin
ced7b73322 Limit the permission of secret used by proxy cache service
Limit the permission of secret used by proxy cache service, fixes #12257

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-27 10:15:00 +08:00
stonezdj(Daojun Zhang)
8bbfb811a4
Merge pull request #12543 from stonezdj/200721_release_conn_ldap
Release connection after search ldap user
2020-07-23 11:06:14 +08:00
Wang Yan
eeb8fca255
add debugging env for GC time window (#12528)
* add debugging env for GC time window

For debugging, the tester/users wants to run GC to delete the removed artifact immediately instead of waitting for two hours, add the env(GC_BLOB_TIME_WINDOW) to meet this.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-07-22 11:09:01 +08:00
stonezdj
07694db90d Release connection after search ldap user
Fixes: 12162

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-07-21 18:00:39 +08:00
Ziming Zhang
8857e89e40 feature(redis) support redis sentinel
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-07-19 21:19:03 +08:00
Wenkai Yin
d7327d8d8e Include the registry info in the project summary API for proxy cache project
Include the registry info in the project summary API for proxy cache project

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-07-17 16:14:45 +08:00
Alvaro Iradier
81a7239c66 Better error handling
* Raise an internal error if username claim is not found, instead of just logging a warning
* Don't remove userInfoKey for session on error when it is not required
* Rename "OIDC Username Claim" to just "Username claim"

Signed-off-by: Alvaro Iradier <airadier@gmail.com>
2020-07-16 12:12:08 +02:00
Alvaro Iradier
6f88ff7429 Fix test suite and add test for userClaim
Signed-off-by: Alvaro Iradier <airadier@gmail.com>
2020-07-16 12:12:08 +02:00
Alvaro Iradier
714f989759 Add options for automatic onboarding and username claim
- Add an option in the UI to enable or disable the automatic user onboarding
- Add an option to specify the claim name where the username is retrieved from.

Signed-off-by: Alvaro Iradier <airadier@gmail.com>
2020-07-16 12:12:08 +02:00
He Weiwei
cadcd4b877
Merge pull request #12480 from heww/move-pkg-types
refactor(quota): move pkg/types to pkg/quota/types
2020-07-15 11:32:27 +08:00
Daniel Jiang
947eadaa72
Merge pull request #12440 from heww/remove-init-clair-db
refactor: remove initialization of clair db
2020-07-15 00:38:12 +08:00
He Weiwei
a22d803a95 refactor(quota): move pkg/types to pkg/quota/types
Closes #9664

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-14 14:28:53 +00:00
fanjiankong
a99aa21c8a Enable RBAC control in the preheat API
Signed-off-by: fanjiankong <fanjiankong@tencent.com>
2020-07-13 11:06:25 +08:00
Julien Vey
84c34afa22 Fix typo in sql log
Signed-off-by: Julien Vey <vey.julien@gmail.com>
2020-07-10 14:29:04 +02:00
He Weiwei
039aef5356 refactor: remove initialization of clair db
To fetch vulnerability database updated time of the Clair had moved to
the Clair adapter so removes the initialization of clair db in the core.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-09 15:26:14 +00:00
He Weiwei
e095958a27 fix(db): set max open conns of sql.DB manually
Due to the issues of beego v1.12.1 and v1.12.2, we set the max open conns
ourselves.

Closes #12403

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-07-09 03:35:41 +00:00
Daniel Jiang
1637e6a588 Rename master role to maintainer
This commit rename the var name, text appearance, and swagger of "master" role
to "maintainer" role.
It only covers backend code.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-07-08 09:20:07 +08:00
AllForNothing
fff6f7529a Replace all whitelist with allowlist
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-06-24 16:17:17 +08:00
Wenkai Yin(尹文开)
202916e396
Merge pull request #12280 from ywk253100/200616_task_manager
Implement task and execution manager
2020-06-23 18:44:44 +08:00
Wenkai Yin
ea20690264 Implement task and execution manager
Implement task and execution manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-06-23 17:10:58 +08:00
Wang Yan
de504993ad update blob controller & manager
1, add two more attributes, update_time and status
2, add delete and fresh update time method in blob mgr & ctr.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-23 13:10:57 +08:00
Wang Yan
58b7242a25
move send error to source lib (#12175)
* move send error to source lib

Move the sendError into library in case the cycle dependency as regsitry and core are now the consumers.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-06-19 01:04:50 +08:00