Upstream
/
harbor
mirror of https://github.com/goharbor/harbor.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

add get scheduled and periodic executions APIs 

Signed-off-by: Steven Zou <szou@vmware.com>
This commit is contained in:
Steven Zou 2019-04-19 13:54:23 +08:00
parent 608354dd04 15846ac524
commit f8feaa192e
483 changed files with 32982 additions and 13708 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -1,4 +1,3 @@
harbor
!/contrib/helm/harbor
make/docker-compose.yml

26
CHANGELOG.md
View File

@ -1,5 +1,31 @@
# Changelog
## v1.7.5 (2019-04-02)
* Bumped up Clair to v2.0.8
* Fixed issues in supporting windows images. #6992 #6369
* Removed user-agent check-in notification handler. #5729
* Fixed the issue global search not working if chartmusuem is not installed #6753
## v1.7.4 (2019-03-04)
[Full list of issues fixed in v1.7.4](https://github.com/goharbor/harbor/issues?q=is%3Aissue+is%3Aclosed+label%3Atarget%2F1.7.4)
## v1.7.1 (2019-01-07)
[Full list of issues fixed in v1.7.1](https://github.com/goharbor/harbor/issues?q=is%3Aissue+is%3Aclosed+label%3Atarget%2F1.7.1)
## v1.7.0 (2018-12-19)
* Support deploy Harbor with Helm Chart, enables the user to have high availability of Harbor services, refer to the [Installation and Configuration Guide](https://github.com/goharbor/harbor-helm/tree/1.0.0).
* Support on-demand Garbage Collection, enables the admin to configure run docker registry garbage collection manually or automatically with a cron schedule.
* Support Image Retag, enables the user to tag image to different repositories and projects, this is particularly useful in cases when images need to be retagged programmatically in a CI pipeline.
* Support Image Build History, makes it easy to see the contents of a container image, refer to the [User Guide](https://github.com/goharbor/harbor/blob/release-1.7.0/docs/user_guide.md#build-history).
* Support Logger customization, enables the user to customize STDOUT / STDERR / FILE / DB logger of running jobs.
* Improve user experience of Helm Chart Repository:
- Chart searching included in the global search results
- Show chart versions total number in the chart list
- Mark labels to helm charts
- The latest version can be downloaded as default one on the chart list view
- The chart can be deleted by deleting all the versions under it
## v1.6.0 (2018-09-11)
- Support manages Helm Charts: From version 1.6.0, Harbor is upgraded to be a composite cloud-native registry, which supports both image management and helm charts management.

6
Makefile
View File

@ -98,7 +98,7 @@ VERSIONFILENAME=UIVERSION
PREPARE_VERSION_NAME=versions
#versions
REGISTRYVERSION=v2.7.1
REGISTRYVERSION=v2.7.1-patch-2819
NGINXVERSION=$(VERSIONTAG)
NOTARYVERSION=v0.6.1
CLAIRVERSION=v2.0.7
@ -295,7 +295,7 @@ compile: check_environment versions_prepare compile_core compile_jobservice comp
update_prepare_version:
@echo "substitude the prepare version tag in prepare file..."
$(SEDCMD) -i -e 's/goharbor\/prepare:.*[[:space:]]\+/goharbor\/prepare:$(VERSIONTAG) /' $(MAKEPATH)/prepare ;
@$(SEDCMD) -i -e 's/goharbor\/prepare:.*[[:space:]]\+/goharbor\/prepare:$(VERSIONTAG) /' $(MAKEPATH)/prepare ;
prepare: update_prepare_version
@echo "preparing..."
@ -310,7 +310,7 @@ build:
install: compile ui_version build prepare start
package_online: prepare
package_online: update_prepare_version
@echo "packing online package ..."
@cp -r make $(HARBORPKG)
@if [ -n "$(REGISTRYSERVER)" ] ; then \

BIN
docs/img/robotaccount/add_robot_account.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 114 KiB

BIN
docs/img/robotaccount/add_robot_account_2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
docs/img/robotaccount/copy_robot_account_token.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 67 KiB

BIN
docs/img/robotaccount/disable_delete_robot_account.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 118 KiB

BIN
docs/img/robotaccount/set_robot_account_token_duration.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 78 KiB

908
docs/swagger.yaml
View File

File diff suppressed because it is too large Load Diff

40
docs/user_guide.md
View File

@ -29,6 +29,7 @@ This guide walks you through the fundamentals of using Harbor. You'll learn how
* [Working with Helm CLI](#working-with-helm-cli)
* [Online Garbage Collection.](#online-garbage-collection)
* [View build history.](#build-history)
* [Manage robot account of a project.](#robot-account)
## Role Based Access Control(RBAC)
@ -597,3 +598,42 @@ Build history make it easy to see the contents of a container image, find the co
In Harbor portal, enter your project, select the repository, click on the link of tag name you'd like to see its build history, the detail page will be opened. Then switch to `Build History` tab, you can see the build history information.
![build_ history](img/build_history.png)
## Robot Account
Robot Accounts are accounts created by project admins that are intended for automated operations. They have the following limitations:
1, Robot Accounts cannot login Harbor portal
2, Robot Accounts can only perform `docker push`/`docker pull` operations with a token.
### Add a Robot Account
If you are a project admin, you can create a Robot Account by clicking "New Robot Account" in the `Robot Accounts` tab of a project, and enter a name, a description and permission.
![add_robot_account](img/robotaccount/add_robot_account.png)
![add_robot_account](img/robotaccount/add_robot_account_2.png)
> **NOTE:** The name will become `robot$<accountname>` and will be used to distinguish a robot account from a normal harbor user.
![copy_robot_account_token](img/robotaccount/copy_robot_account_token.png)
As Harbor doesn't store your account token, please make sure to copy it in the pop up dialog after creating, otherwise, there is no way to get it from Harbor.
### Configure duration of robot account
If you are a system admin, you can configure the robot account token duration in days.
![set_robot_account_token_duration](img/robotaccount/set_robot_account_token_duration.png)
### Authenticate with a robot account
To authenticate with a Robot Account, use `docker login` as below,
```
docker login harbor.io
Username: robot$accountname
Password: Thepasswordgeneratedbyprojectadmin
```
### Disable a robot account
If you are a project admin, you can disable a Robot Account by clicking "Disable Account" in the `Robot Accounts` tab of a project.
![disable_robot_account](img/robotaccount/disable_delete_robot_account.png)
### Delete a robot account
If you are a project admin, you can delete a Robot Account by clicking "Delete" in the `Robot Accounts` tab of a project.
![delete_robot_account](img/robotaccount/disable_delete_robot_account.png)

155
make/harbor.yml
View File

@ -1,114 +1,93 @@
## Configuration file of Harbor
#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 1.7.0
#The IP address or hostname to access admin UI and registry service.
#DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
#DO NOT comment out this line, modify the value of "hostname" directly, or the installation will fail.
hostname: reg.mydomain.com
# core, harbor
http:
port: 80
#The protocol for accessing the UI and token/notification service, by default it is http.
#It can be set to https if ssl is enabled on nginx.
ui_url_protocol: https
# https:
# port: 443
# #The path of cert and key files for nginx
# certificate: /your/certificate/path
# private_key: /your/private/key/path
#Maximum number of job workers in job service
max_job_workers: 10
# Uncomment extearnal_url if you want to enable external proxy
# And when it enabled the hostname will no longger used
# external_url: https://reg.mydomain.com:8433
# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345
## Harbor DB configuration
database:
#The password for the root user of Harbor DB. Change this before any production use.
password: root123
# The default data volume
data_volume: /data
#The path of cert and key files for nginx, they are applied only the protocol is set to https
ssl_cert: /data/cert/server.crt
ssl_cert_key: /data/cert/server.key
# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
# # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
# # of registry's and chart repository's containers. This is usually needed when the user hosts a internal storage with self signed certificate.
# ca_bundle:
#The path of secretkey storage
secretkey_path: /data
# # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
# # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
# filesystem:
# maxthreads: 100
#Admiral's url, comment this attribute, or set its value to NA when Harbor is standalone
admiral_url: NA
# Clair configuration
clair:
# The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
updaters_interval: 12
# Config http proxy for Clair, e.g. http://my.proxy.com:3128
# Clair doesn't need to connect to harbor internal components via http proxy.
http_proxy:
https_proxy:
no_proxy: 127.0.0.1,localhost,core,registry
jobservice:
# Maximum number of job workers in job service
max_job_workers: 10
# Log configurations
log:
# options are debug, info, warn, error
level: info
# Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
rotate_count: 50
# Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
# If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
# are all valid.
rotate_size: 200M
# The directory that store log files
# The directory on your host that store log
location: /var/log/harbor
#NOTES: The properties between BEGIN INITIAL PROPERTIES and END INITIAL PROPERTIES
#only take effect in the first boot, the subsequent changes of these properties
#should be performed on web ui
#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 1.7.0
##The initial password of Harbor admin, only works for the first time when Harbor starts.
#It has no effect after the first launch of Harbor.
#Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345
# Uncomment external_database if using external database. And the password will replace the the password setting in database.
# And currently ontly support postgres.
# external_database:
# host: postgresql
# port: 5432
# username: postgres
# password: root123
# ssl_mode: disable
## Harbor DB configuration
database:
#The address of the Harbor database. Only need to change when using external db.
host: postgresql
#The port of Harbor database host
port: 5432
#The user name of Harbor database
username: postgres
#The password for the root user of Harbor DB. Change this before any production use.
password: root123
# Redis server configuration
redis:
# Redis connection address
host: redis
# Redis connection port
port: 6379
# Redis connection password
password:
# Redis connection db index
# db_index 1,2,3 is for registry, jobservice and chartmuseum.
# db_index 0 is for UI, it's unchangeable
db_index: 1,2,3
# Clair DB configuration
clair:
# Clair DB host address. Only change it when using an exteral DB.
db_host: postgresql
# The password of the Clair's postgres database. Only effective when Harbor is deployed with Clair.
# Please update it before deployment. Subsequent update will cause Clair's API server and Harbor unable to access Clair's database.
db_password: root123
# Clair DB connect port
db_port: 5432
# Clair DB username
db_username: postgres
# Clair default database
db: postgres
# The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
updaters_interval: 12
#Config http proxy for Clair, e.g. http://my.proxy.com:3128
#Clair doesn't need to connect to harbor internal components via http proxy.
http_proxy:
https_proxy:
no_proxy: 127.0.0.1,localhost,core,registry
# Harbor Storage settings
storage:
#Please be aware that the following storage settings will be applied to both docker registry and helm chart repository.
#registry_storage_provider can be: filesystem, s3, gcs, azure, etc.
registry_storage_provider_name: filesystem
#registry_storage_provider_config is a comma separated "key: value" pairs, e.g. "key1: value, key2: value2".
#To avoid duplicated configurations, both docker registry and chart repository follow the same storage configuration specifications of docker registry.
#Refer to https://docs.docker.com/registry/configuration/#storage for all available configuration.
registry_storage_provider_config:
#registry_custom_ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#of registry's and chart repository's containers. This is usually needed when the user hosts a internal storage with self signed certificate.
registry_custom_ca_bundle:
#If reload_config=true, all settings which present in harbor.yml take effect after prepare and restart harbor, it overwrites exsiting settings.
#reload_config=true
#Regular expression to match skipped environment variables
#skip_reload_env_pattern: (^EMAIL.*)|(^LDAP.*)
# Umcomments external_redis if using external Redis server
# external_redis:
# host: redis
# port: 6379
# password:
# # db_index 0 is for core, it's unchangeable
# registry_db_index: 1
# jobservice_db_index: 2
# chartmuseum_db_index: 3

4
make/install.sh
View File

@ -192,9 +192,9 @@ docker-compose up -d
protocol=http
hostname=reg.mydomain.com
if [[ $(cat ./harbor.yml) =~ ui_url_protocol:[[:blank:]]*(https?) ]]
if [ -n "$(grep '^[^#]*https:' ./harbor.yml)" ]
then
protocol=${BASH_REMATCH[1]}
protocol=https
fi
if [[ $(grep '^[[:blank:]]*hostname:' ./harbor.yml) =~ hostname:[[:blank:]]*(.*) ]]

114
make/migrations/postgresql/0004_1.8.0_schema.up.sql
View File

@ -16,6 +16,9 @@ CREATE TRIGGER robot_update_time_at_modtime BEFORE UPDATE ON robot FOR EACH ROW
CREATE TABLE oidc_user (
id SERIAL NOT NULL,
user_id int NOT NULL,
/*
Encoded secret
*/
secret varchar(255) NOT NULL,
/*
Subject and Issuer
@ -24,9 +27,14 @@ CREATE TABLE oidc_user (
The sub (subject) and iss (issuer) Claims, used together, are the only Claims that an RP can rely upon as a stable identifier for the End-User
*/
subiss varchar(255) NOT NULL,
/*
Encoded token
*/
token text,
creation_time timestamp default CURRENT_TIMESTAMP,
update_time timestamp default CURRENT_TIMESTAMP,
PRIMARY KEY (id),
FOREIGN KEY (user_id) REFERENCES harbor_user(user_id),
UNIQUE (subiss)
);
@ -42,4 +50,108 @@ WHERE j.policy_id = p.id AND p.deleted = TRUE;
/*delete replication policy which has been marked as "deleted"*/
DELETE FROM replication_policy AS p
WHERE p.deleted = TRUE;
WHERE p.deleted = TRUE;
/*upgrade the replication_target to registry*/
DROP TRIGGER replication_target_update_time_at_modtime ON replication_target;
ALTER TABLE replication_target RENAME TO registry;
ALTER TABLE registry ALTER COLUMN url TYPE varchar(256);
ALTER TABLE registry ADD COLUMN credential_type varchar(16);
ALTER TABLE registry RENAME COLUMN username TO access_key;
ALTER TABLE registry RENAME COLUMN password TO access_secret;
ALTER TABLE registry ALTER COLUMN access_secret TYPE varchar(1024);
ALTER TABLE registry ADD COLUMN type varchar(32);
ALTER TABLE registry DROP COLUMN target_type;
ALTER TABLE registry ADD COLUMN description text;
ALTER TABLE registry ADD COLUMN health varchar(16);
UPDATE registry SET type='harbor';
UPDATE registry SET credential_type='basic';
/*upgrade the replication_policy*/
ALTER TABLE replication_policy ADD COLUMN creator varchar(256);
ALTER TABLE replication_policy ADD COLUMN src_registry_id int;
/*The predefined filters will be cleared and replaced by "project_name/"+double star.
if harbor is integrated with the external project service, we cannot get the project name by ID,
which means the repilcation policy will match all resources.*/
UPDATE replication_policy r SET filters=(SELECT CONCAT('[{"type":"name","value":"', p.name,'/**"}]') FROM project p WHERE p.project_id=r.project_id);
ALTER TABLE replication_policy RENAME COLUMN target_id TO dest_registry_id;
ALTER TABLE replication_policy ALTER COLUMN dest_registry_id DROP NOT NULL;
ALTER TABLE replication_policy ADD COLUMN dest_namespace varchar(256);
ALTER TABLE replication_policy ADD COLUMN override boolean;
ALTER TABLE replication_policy DROP COLUMN project_id;
ALTER TABLE replication_policy RENAME COLUMN cron_str TO trigger;
DROP TRIGGER replication_immediate_trigger_update_time_at_modtime ON replication_immediate_trigger;
DROP TABLE replication_immediate_trigger;
create table replication_execution (
id SERIAL NOT NULL,
policy_id int NOT NULL,
status varchar(32),
/*the status text may contain error message whose length is very long*/
status_text text,
total int NOT NULL DEFAULT 0,
failed int NOT NULL DEFAULT 0,
succeed int NOT NULL DEFAULT 0,
in_progress int NOT NULL DEFAULT 0,
stopped int NOT NULL DEFAULT 0,
trigger varchar(64),
start_time timestamp default CURRENT_TIMESTAMP,
end_time timestamp NULL,
PRIMARY KEY (id)
);
CREATE INDEX execution_policy ON replication_execution (policy_id);
create table replication_task (
id SERIAL NOT NULL,
execution_id int NOT NULL,
resource_type varchar(64),
src_resource varchar(256),
dst_resource varchar(256),
operation varchar(32),
job_id varchar(64),
status varchar(32),
start_time timestamp default CURRENT_TIMESTAMP,
end_time timestamp NULL,
PRIMARY KEY (id)
);
CREATE INDEX task_execution ON replication_task (execution_id);
/*migrate each replication_job record to one replication_execution and one replication_task record*/
DO $$
DECLARE
job RECORD;
execid integer;
BEGIN
FOR job IN SELECT * FROM replication_job WHERE operation != 'schedule'
LOOP
/*insert one execution record*/
INSERT INTO replication_execution (policy_id, start_time) VALUES (job.policy_id, job.creation_time) RETURNING id INTO execid;
/*insert one task record
doesn't record the tags info in "src_resource" and "dst_resource" as the length
of the tags may longer than the capability of the column*/
INSERT INTO replication_task (execution_id, resource_type, src_resource, dst_resource, operation, job_id, status, start_time, end_time)
VALUES (execid, 'image', job.repository, job.repository, job.operation, job.job_uuid, job.status, job.creation_time, job.update_time);
END LOOP;
END $$;
UPDATE replication_task SET status='Pending' WHERE status='pending';
UPDATE replication_task SET status='InProgress' WHERE status='scheduled';
UPDATE replication_task SET status='InProgress' WHERE status='running';
UPDATE replication_task SET status='Failed' WHERE status='error';
UPDATE replication_task SET status='Succeed' WHERE status='finished';
UPDATE replication_task SET operation='copy' WHERE operation='transfer';
UPDATE replication_task SET operation='deletion' WHERE operation='delete';
/*upgrade the replication_job to replication_schedule_job*/
DELETE FROM replication_job WHERE operation != 'schedule';
ALTER TABLE replication_job RENAME COLUMN job_uuid TO job_id;
ALTER TABLE replication_job DROP COLUMN repository;
ALTER TABLE replication_job DROP COLUMN operation;
ALTER TABLE replication_job DROP COLUMN tags;
ALTER TABLE replication_job DROP COLUMN op_uuid;
DROP INDEX policy;
DROP INDEX poid_uptime;
DROP INDEX poid_status;
DROP TRIGGER replication_job_update_time_at_modtime ON replication_job;
ALTER TABLE replication_job RENAME TO replication_schedule_job;

6
make/photon/prepare/g.py
View File

@ -20,10 +20,8 @@ private_key_pem_path = Path('/secret/core/private_key.pem')
root_crt_path = Path('/secret/registry/root.crt')
config_file_path = '/compose_location/harbor.yml'
input_config_path = '/input/harbor.yml'
versions_file_path = Path('/usr/src/app/versions')
cert_dir = os.path.join(config_dir, "nginx", "cert")
core_cert_dir = os.path.join(config_dir, "core", "certificates")
registry_custom_ca_bundle_storage_path = Path('/secret/common/custom-ca-bundle.crt')
registry_custom_ca_bundle_storage_input_path = Path('/input/common/custom-ca-bundle.crt')
core_cert_dir = os.path.join(config_dir, "core", "certificates")

16
make/photon/prepare/main.py
View File

@ -4,7 +4,7 @@ import click
from utils.misc import delfile
from utils.configs import validate, parse_yaml_config
from utils.cert import prepare_ca, SSL_CERT_KEY_PATH, SSL_CERT_PATH, get_secret_key, copy_ssl_cert, copy_secret_keys
from utils.cert import prepare_ca, SSL_CERT_KEY_PATH, SSL_CERT_PATH, get_secret_key
from utils.db import prepare_db
from utils.jobservice import prepare_job_service
from utils.registry import prepare_registry
@ -16,13 +16,12 @@ from utils.clair import prepare_clair
from utils.chart import prepare_chartmuseum
from utils.docker_compose import prepare_docker_compose
from utils.nginx import prepare_nginx, nginx_confd_dir
from g import (config_dir, config_file_path, private_key_pem_path, root_crt_path,
registry_custom_ca_bundle_storage_path, registry_custom_ca_bundle_storage_input_path, secret_key_dir,
from g import (config_dir, input_config_path, private_key_pem_path, root_crt_path, secret_key_dir,
old_private_key_pem_path, old_crt_path)
# Main function
@click.command()
@click.option('--conf', default=config_file_path, help="the path of Harbor configuration file")
@click.option('--conf', default=input_config_path, help="the path of Harbor configuration file")
@click.option('--with-notary', is_flag=True, help="the Harbor instance is to be deployed with notary")
@click.option('--with-clair', is_flag=True, help="the Harbor instance is to be deployed with clair")
@click.option('--with-chartmuseum', is_flag=True, help="the Harbor instance is to be deployed with chart repository supporting")
@ -40,21 +39,14 @@ def main(conf, with_notary, with_clair, with_chartmuseum):
prepare_db(config_dict)
prepare_job_service(config_dict)
copy_secret_keys()
get_secret_key(secret_key_dir)
if config_dict['protocol'] == 'https':
copy_ssl_cert()
# If Customized cert enabled
prepare_ca(
private_key_pem_path=private_key_pem_path,
root_crt_path=root_crt_path,
old_private_key_pem_path=old_private_key_pem_path,
old_crt_path=old_crt_path,
registry_custom_ca_bundle_config=registry_custom_ca_bundle_storage_input_path,
registry_custom_ca_bundle_storage_path=registry_custom_ca_bundle_storage_path)
old_crt_path=old_crt_path)
if with_notary:
prepare_notary(config_dict, nginx_confd_dir, SSL_CERT_PATH, SSL_CERT_KEY_PATH)

41
make/photon/prepare/templates/core/config_env.jinja
View File

@ -1,16 +1,6 @@
PORT=8080
LOG_LEVEL=info
LOG_LEVEL={{log_level}}
EXT_ENDPOINT={{public_url}}
SELF_REGISTRATION={{self_registration}}
LDAP_URL={{ldap_url}}
LDAP_SEARCH_DN={{ldap_searchdn}}
LDAP_SEARCH_PWD={{ldap_search_pwd}}
LDAP_BASE_DN={{ldap_basedn}}
LDAP_FILTER={{ldap_filter}}
LDAP_UID={{ldap_uid}}
LDAP_SCOPE={{ldap_scope}}
LDAP_TIMEOUT={{ldap_timeout}}
LDAP_VERIFY_CERT={{ldap_verify_cert}}
DATABASE_TYPE=postgresql
POSTGRESQL_HOST={{db_host}}
POSTGRESQL_PORT={{db_port}}
@ -18,50 +8,29 @@ POSTGRESQL_USERNAME={{db_user}}
POSTGRESQL_PASSWORD={{db_password}}
POSTGRESQL_DATABASE=registry
POSTGRESQL_SSLMODE=disable
LDAP_GROUP_BASEDN={{ldap_group_basedn}}
LDAP_GROUP_FILTER={{ldap_group_filter}}
LDAP_GROUP_GID={{ldap_group_gid}}
LDAP_GROUP_SCOPE={{ldap_group_scope}}
REGISTRY_URL={{registry_url}}
TOKEN_SERVICE_URL={{token_service_url}}
EMAIL_HOST={{email_host}}
EMAIL_PORT={{email_port}}
EMAIL_USR={{email_usr}}
EMAIL_PWD={{email_pwd}}
EMAIL_SSL={{email_ssl}}
EMAIL_FROM={{email_from}}
EMAIL_IDENTITY={{email_identity}}
EMAIL_INSECURE={{email_insecure}}
HARBOR_ADMIN_PASSWORD={{harbor_admin_password}}
PROJECT_CREATION_RESTRICTION={{project_creation_restriction}}
MAX_JOB_WORKERS={{max_job_workers}}
CORE_SECRET={{core_secret}}
JOBSERVICE_SECRET={{jobservice_secret}}
TOKEN_EXPIRATION={{token_expiration}}
CFG_EXPIRATION=5
ADMIRAL_URL={{admiral_url}}
WITH_NOTARY={{with_notary}}
WITH_CLAIR={{with_clair}}
CLAIR_DB_PASSWORD={{clair_db_password}}
CLAIR_DB_HOST={{clair_db_host}}
CLAIR_DB_PORT={{clair_db_port}}
CLAIR_DB_USERNAME={{clair_db_username}}
CLAIR_DB_PASSWORD={{db_password}}
CLAIR_DB_HOST={{db_host}}
CLAIR_DB_PORT={{db_port}}
CLAIR_DB_USERNAME={{db_user}}
CLAIR_DB={{clair_db}}
CLAIR_DB_SSLMODE=disable
RESET={{reload_config}}
UAA_ENDPOINT={{uaa_endpoint}}
UAA_CLIENTID={{uaa_clientid}}
UAA_CLIENTSECRET={{uaa_clientsecret}}
UAA_VERIFY_CERT={{uaa_verify_cert}}
CORE_URL={{core_url}}
JOBSERVICE_URL={{jobservice_url}}
CLAIR_URL={{clair_url}}
NOTARY_URL={{notary_url}}
REGISTRY_STORAGE_PROVIDER_NAME={{storage_provider_name}}
READ_ONLY=false
SKIP_RELOAD_ENV_PATTERN={{skip_reload_env_pattern}}
RELOAD_KEY={{reload_key}}
CHART_REPOSITORY_URL={{chart_repository_url}}
LDAP_GROUP_ADMIN_DN={{ldap_group_admin_dn}}
REGISTRY_CONTROLLER_URL={{registry_controller_url}}
WITH_CHARTMUSEUM={{with_chartmuseum}}

27
make/photon/prepare/templates/docker_compose/docker-compose.yml.jinja
View File

@ -33,8 +33,11 @@ services:
- {{data_volume}}/registry:/storage:z
- ./common/config/registry/:/etc/registry/:z
- {{data_volume}}/secret/registry/root.crt:/etc/registry/root.crt:z
{%if registry_custom_ca_bundle_storage_path %}
- {{data_volume}}/secret/common/custom-ca-bundle.crt:/harbor_cust_cert/custom-ca-bundle.crt:z
{% if gcs_keyfile %}
- {{gcs_keyfile}}:/etc/registry/gcs.key
{% endif %}
{%if registry_custom_ca_bundle_path %}
- {{registry_custom_ca_bundle_path}}:/harbor_cust_cert/custom-ca-bundle.crt:z
{% endif %}
networks:
- harbor
@ -247,8 +250,8 @@ services:
volumes:
- ./common/config/nginx:/etc/nginx:z
{% if protocol == 'https' %}
- {{data_volume}}/secret/nginx/server.key:/etc/nginx/cert/server.key
- {{data_volume}}/secret/nginx/server.crt:/etc/nginx/cert/server.crt
- {{cert_key_path}}:/etc/nginx/cert/server.key:z
- {{cert_path}}:/etc/nginx/cert/server.crt:z
{% endif %}
networks:
- harbor
@ -257,9 +260,13 @@ services:
{% endif %}
dns_search: .
ports:
- 80:80
- 443:443
- {{http_port}}:80
{% if protocol == 'https' %}
- {{https_port}}:443
{% endif %}
{% if with_notary %}
- 4443:4443
{% endif %}
depends_on:
- postgresql
- registry
@ -337,8 +344,8 @@ services:
- postgresql
volumes:
- ./common/config/clair/config.yaml:/etc/clair/config.yaml:z
{%if registry_custom_ca_bundle_storage_path %}
- {{data_volume}}/secret/common/custom-ca-bundle.crt:/harbor_cust_cert/custom-ca-bundle.crt:z
{%if registry_custom_ca_bundle_path %}
- {{registry_custom_ca_bundle_path}}:/harbor_cust_cert/custom-ca-bundle.crt:z
{% endif %}
logging:
driver: "syslog"
@ -368,8 +375,8 @@ services:
volumes:
- {{data_volume}}/chart_storage:/chart_storage:z
- ./common/config/chartserver:/etc/chartserver:z
{%if registry_custom_ca_bundle_storage_path %}
- {{data_volume}}/secret/common/custom-ca-bundle.crt:/harbor_cust_cert/custom-ca-bundle.crt:z
{%if registry_custom_ca_bundle_path %}
- {{registry_custom_ca_bundle_path}}:/harbor_cust_cert/custom-ca-bundle.crt:z
{% endif %}
logging:
driver: "syslog"

5
make/photon/prepare/templates/registry/config.yml.jinja
View File

@ -1,6 +1,6 @@
version: 0.1
log:
level: info
level: {{log_level}}
fields:
service: registry
storage:
@ -37,3 +37,6 @@ notifications:
timeout: 3000ms
threshold: 5
backoff: 1s
compatibility:
schema1:
enabled: true

38
make/photon/prepare/utils/cert.py
View File

@ -10,16 +10,11 @@ from .misc import generate_random_string
SSL_CERT_PATH = os.path.join("/etc/nginx/cert", "server.crt")
SSL_CERT_KEY_PATH = os.path.join("/etc/nginx/cert", "server.key")
input_cert = '/input/nginx/server.crt'
input_cert_key = '/input/nginx/server.key'
secret_cert_dir = '/secret/nginx'
secret_cert = '/secret/nginx/server.crt'
secret_cert_key = '/secret/nginx/server.key'
input_secret_keys_dir = '/input/keys'
secret_keys_dir = '/secret/keys'
allowed_secret_key_names = ['defaultalias', 'secretkey']
def _get_secret(folder, filename, length=16):
key_file = os.path.join(folder, filename)
@ -50,26 +45,6 @@ def get_alias(path):
alias = _get_secret(path, "defaultalias", length=8)
return alias
def copy_secret_keys():
"""
Copy the secret keys, which used for encrypt user password, from input keys dir to secret keys dir
"""
if os.path.isdir(input_secret_keys_dir) and os.path.isdir(secret_keys_dir):
input_files = os.listdir(input_secret_keys_dir)
secret_files = os.listdir(secret_keys_dir)
files_need_copy = [x for x in input_files if (x in allowed_secret_key_names) and (x not in secret_files) ]
for f in files_need_copy:
shutil.copy(f, secret_keys_dir)
def copy_ssl_cert():
"""
Copy the ssl certs key paris, which used in nginx ssl certificate, from input dir to secret cert dir
"""
if os.path.isfile(input_cert_key) and os.path.isfile(input_cert):
os.makedirs(secret_cert_dir, exist_ok=True)
shutil.copy(input_cert, secret_cert)
shutil.copy(input_cert_key, secret_cert_key)
## decorator actions
def stat_decorator(func):
@wraps(func)
@ -115,9 +90,7 @@ def prepare_ca(
private_key_pem_path: Path,
root_crt_path: Path,
old_private_key_pem_path: Path,
old_crt_path: Path,
registry_custom_ca_bundle_config: Path,
registry_custom_ca_bundle_storage_path: Path):
old_crt_path: Path):
if not ( private_key_pem_path.exists() and root_crt_path.exists() ):
# From version 1.8 the cert storage path is changed
# if old key paris not exist create new ones
@ -132,11 +105,4 @@ def prepare_ca(
mark_file(root_crt_path)
else:
shutil.move(old_crt_path, root_crt_path)
shutil.move(old_private_key_pem_path, private_key_pem_path)
if not registry_custom_ca_bundle_storage_path.exists() and registry_custom_ca_bundle_config.exists():
registry_custom_ca_bundle_storage_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(registry_custom_ca_bundle_config, registry_custom_ca_bundle_storage_path)
mark_file(registry_custom_ca_bundle_storage_path)
print("Copied custom ca bundle: %s" % registry_custom_ca_bundle_config)
shutil.move(old_private_key_pem_path, private_key_pem_path)

16
make/photon/prepare/utils/chart.py
View File

@ -24,11 +24,6 @@ def prepare_chartmuseum(config_dict):
print ("Create config folder: %s" % chartm_config_dir)
os.makedirs(chartm_config_dir)
# handle custom ca bundle
if len(registry_custom_ca_bundle_path) > 0 and os.path.isfile(registry_custom_ca_bundle_path):
shutil.copyfile(registry_custom_ca_bundle_path, os.path.join(chartm_config_dir, "custom-ca-bundle.crt"))
print("Copied custom ca bundle: %s" % os.path.join(chartm_config_dir, "custom-ca-bundle.crt"))
# process redis info
cache_store = "redis"
cache_redis_password = redis_password
@ -42,18 +37,9 @@ def prepare_chartmuseum(config_dict):
# storage provider configurations
# please be aware that, we do not check the validations of the values for the specified keys
# convert the configs to config map
storage_provider_configs = storage_provider_config.split(",")
storgae_provider_confg_map = {}
storgae_provider_confg_map = storage_provider_config
storage_provider_config_options = []
for k_v in storage_provider_configs:
if len(k_v) > 0:
kvs = k_v.split(": ") # add space suffix to avoid existing ":" in the value
if len(kvs) == 2:
#key must not be empty
if kvs[0].strip() != "":
storgae_provider_confg_map[kvs[0].strip()] = kvs[1].strip()
if storage_provider_name == "s3":
# aws s3 storage
storage_driver = "amazon"

10
make/photon/prepare/utils/clair.py
View File

@ -27,17 +27,17 @@ def prepare_clair(config_dict):
render_jinja(
postgres_env_template,
postgres_env_path,
password=config_dict['clair_db_password'])
password=config_dict['db_password'])
render_jinja(
clair_config_template,
clair_config_path,
uid=DEFAULT_UID,
gid=DEFAULT_GID,
password= config_dict['clair_db_password'],
username= config_dict['clair_db_username'],
host= config_dict['clair_db_host'],
port= config_dict['clair_db_port'],
password= config_dict['db_password'],
username= config_dict['db_user'],
host= config_dict['db_host'],
port= config_dict['db_port'],
dbname= config_dict['clair_db'],
interval= config_dict['clair_updaters_interval'])

230
make/photon/prepare/utils/configs.py
View File

@ -37,10 +37,6 @@ def validate(conf, **kwargs):
raise Exception(
"Error: redis_port in harbor.cfg needs to point to the port of Redis server or cluster.")
redis_db_index = conf.get("redis_db_index")
if len(redis_db_index.split(",")) != 3:
raise Exception(
"Error invalid value for redis_db_index: %s. please set it as 1,2,3" % redis_db_index)
def parse_versions():
if not versions_file_path.is_file():
@ -58,137 +54,127 @@ def parse_yaml_config(config_file_path):
with open(config_file_path) as f:
configs = yaml.load(f)
config_dict = {}
config_dict['adminserver_url'] = "http://adminserver:8080"
config_dict['registry_url'] = "http://registry:5000"
config_dict['registry_controller_url'] = "http://registryctl:8080"
config_dict['core_url'] = "http://core:8080"
config_dict['token_service_url'] = "http://core:8080/service/token"
config_dict = {
'adminserver_url': "http://adminserver:8080",
'registry_url': "http://registry:5000",
'registry_controller_url': "http://registryctl:8080",
'core_url': "http://core:8080",
'token_service_url': "http://core:8080/service/token",
'jobservice_url': 'http://jobservice:8080',
'clair_url': 'http://clair:6060',
'notary_url': 'http://notary-server:4443',
'chart_repository_url': 'http://chartmuseum:9999'
}
config_dict['jobservice_url'] = "http://jobservice:8080"
config_dict['clair_url'] = "http://clair:6060"
config_dict['notary_url'] = "http://notary-server:4443"
config_dict['chart_repository_url'] = "http://chartmuseum:9999"
config_dict['hostname'] = configs["hostname"]
if configs.get("reload_config"):
config_dict['reload_config'] = configs.get("reload_config")
else:
config_dict['reload_config'] = "false"
config_dict['protocol'] = 'http'
http_config = configs.get('http') or {}
config_dict['http_port'] = http_config.get('port', 80)
config_dict['hostname'] = configs.get("hostname")
config_dict['protocol'] = configs.get("ui_url_protocol")
config_dict['public_url'] = config_dict['protocol'] + "://" + config_dict['hostname']
https_config = configs.get('https')
if https_config:
config_dict['protocol'] = 'https'
config_dict['https_port'] = https_config.get('port', 443)
config_dict['cert_path'] = https_config["certificate"]
config_dict['cert_key_path'] = https_config["private_key"]
# Data path volume
config_dict['data_volume'] = configs.get("data_volume")
# Email related configs
config_dict['email_identity'] = configs.get("email_identity")
config_dict['email_host'] = configs.get("email_server")
config_dict['email_port'] = configs.get("email_server_port")
config_dict['email_usr'] = configs.get("email_username")
config_dict['email_pwd'] = configs.get("email_password")
config_dict['email_from'] = configs.get("email_from")
config_dict['email_ssl'] = configs.get("email_ssl")
config_dict['email_insecure'] = configs.get("email_insecure")
config_dict['harbor_admin_password'] = configs.get("harbor_admin_password")
config_dict['auth_mode'] = configs.get("auth_mode")
config_dict['ldap_url'] = configs.get("ldap_url")
# LDAP related configs
# this two options are either both set or unset
if configs.get("ldap_searchdn"):
config_dict['ldap_searchdn'] = configs["ldap_searchdn"]
config_dict['ldap_search_pwd'] = configs["ldap_search_pwd"]
else:
config_dict['ldap_searchdn'] = ""
config_dict['ldap_search_pwd'] = ""
config_dict['ldap_basedn'] = configs.get("ldap_basedn")
# ldap_filter is null by default
if configs.get("ldap_filter"):
config_dict['ldap_filter'] = configs["ldap_filter"]
else:
config_dict['ldap_filter'] = ""
config_dict['ldap_uid'] = configs.get("ldap_uid")
config_dict['ldap_scope'] = configs.get("ldap_scope")
config_dict['ldap_timeout'] = configs.get("ldap_timeout")
config_dict['ldap_verify_cert'] = configs.get("ldap_verify_cert")
config_dict['ldap_group_basedn'] = configs.get("ldap_group_basedn")
config_dict['ldap_group_filter'] = configs.get("ldap_group_filter")
config_dict['ldap_group_gid'] = configs.get("ldap_group_gid")
config_dict['ldap_group_scope'] = configs.get("ldap_group_scope")
# Admin dn
config_dict['ldap_group_admin_dn'] = configs.get("ldap_group_admin_dn") or ''
config_dict['public_url'] = configs.get('external_url') or '{protocol}://{hostname}'.format(**config_dict)
# DB configs
db_configs = configs.get('database')
config_dict['db_host'] = db_configs.get("host")
config_dict['db_port'] = db_configs.get("port")
config_dict['db_user'] = db_configs.get("username")
config_dict['db_password'] = db_configs.get("password")
if db_configs:
config_dict['db_host'] = 'postgresql'
config_dict['db_port'] = 5432
config_dict['db_user'] = 'postgres'
config_dict['db_password'] = db_configs.get("password") or ''
config_dict['ssl_mode'] = 'disable'
config_dict['self_registration'] = configs.get("self_registration")
config_dict['project_creation_restriction'] = configs.get("project_creation_restriction")
# secure configs
if config_dict['protocol'] == "https":
config_dict['cert_path'] = configs.get("ssl_cert")
config_dict['cert_key_path'] = configs.get("ssl_cert_key")
config_dict['customize_crt'] = configs.get("customize_crt")
config_dict['max_job_workers'] = configs.get("max_job_workers")
config_dict['token_expiration'] = configs.get("token_expiration")
# Data path volume
config_dict['data_volume'] = configs['data_volume']
config_dict['secretkey_path'] = configs["secretkey_path"]
# Admiral configs
if configs.get("admiral_url"):
config_dict['admiral_url'] = configs["admiral_url"]
# Initial Admin Password
config_dict['harbor_admin_password'] = configs["harbor_admin_password"]
# Registry storage configs
storage_config = configs.get('storage_service') or {}
config_dict['registry_custom_ca_bundle_path'] = storage_config.get('ca_bundle') or ''
if storage_config.get('filesystem'):
config_dict['storage_provider_name'] = 'filesystem'
config_dict['storage_provider_config'] = storage_config['filesystem']
elif storage_config.get('azure'):
config_dict['storage_provider_name'] = 'azure'
config_dict['storage_provider_config'] = storage_config['azure']
elif storage_config.get('gcs'):
config_dict['storage_provider_name'] = 'gcs'
config_dict['storage_provider_config'] = storage_config['gcs']
elif storage_config.get('s3'):
config_dict['storage_provider_name'] = 's3'
config_dict['storage_provider_config'] = storage_config['s3']
elif storage_config.get('swift'):
config_dict['storage_provider_name'] = 'swift'
config_dict['storage_provider_config'] = storage_config['swift']
elif storage_config.get('oss'):
config_dict['storage_provider_name'] = 'oss'
config_dict['storage_provider_config'] = storage_config['oss']
else:
config_dict['admiral_url'] = ""
config_dict['storage_provider_name'] = 'filesystem'
config_dict['storage_provider_config'] = {}
# Clair configs
clair_configs = configs.get("clair") or {}
config_dict['clair_db_password'] = clair_configs.get("db_password") or ''
config_dict['clair_db_host'] = clair_configs.get("db_host") or ''
config_dict['clair_db_port'] = clair_configs.get("db_port") or ''
config_dict['clair_db_username'] = clair_configs.get("db_username") or ''
config_dict['clair_db'] = clair_configs.get("db") or ''
config_dict['clair_updaters_interval'] = clair_configs.get("updaters_interval") or ''
config_dict['clair_db'] = 'postgres'
config_dict['clair_updaters_interval'] = clair_configs.get("updaters_interval") or 12
config_dict['clair_http_proxy'] = clair_configs.get('http_proxy') or ''
config_dict['clair_https_proxy'] = clair_configs.get('https_proxy') or ''
config_dict['clair_no_proxy'] = clair_configs.get('no_proxy') or ''
config_dict['clair_no_proxy'] = clair_configs.get('no_proxy') or '127.0.0.1,localhost,core,registry'
# jobservice config
js_config = configs.get('jobservice') or {}
config_dict['max_job_workers'] = js_config["max_job_workers"]
config_dict['jobservice_secret'] = generate_random_string(16)
# UAA configs
config_dict['uaa_endpoint'] = configs.get("uaa_endpoint")
config_dict['uaa_clientid'] = configs.get("uaa_clientid")
config_dict['uaa_clientsecret'] = configs.get("uaa_clientsecret")
config_dict['uaa_verify_cert'] = configs.get("uaa_verify_cert")
config_dict['uaa_ca_cert'] = configs.get("uaa_ca_cert")
# Log configs
log_configs = configs.get('log') or {}
config_dict['log_location'] = log_configs.get("location")
config_dict['log_rotate_count'] = log_configs.get("rotate_count")
config_dict['log_rotate_size'] = log_configs.get("rotate_size")
config_dict['log_location'] = log_configs["location"]
config_dict['log_rotate_count'] = log_configs["rotate_count"]
config_dict['log_rotate_size'] = log_configs["rotate_size"]
config_dict['log_level'] = log_configs['level']
# Redis configs
redis_configs = configs.get("redis")
# external DB, if external_db enabled, it will cover the database config
external_db_configs = configs.get('external_database') or {}
if external_db_configs:
config_dict['db_password'] = external_db_configs.get('password') or ''
config_dict['db_host'] = external_db_configs['host']
config_dict['db_port'] = external_db_configs['port']
config_dict['db_user'] = db_configs['username']
if external_db_configs.get('ssl_mode'):
config_dict['db_ssl_mode'] = external_db_configs['ssl_mode']
# redis config
redis_configs = configs.get("external_redis")
if redis_configs:
config_dict['redis_host'] = redis_configs.get("host") or ''
config_dict['redis_port'] = redis_configs.get("port") or ''
# using external_redis
config_dict['redis_host'] = redis_configs['host']
config_dict['redis_port'] = redis_configs['port']
config_dict['redis_password'] = redis_configs.get("password") or ''
config_dict['redis_db_index'] = redis_configs.get("db_index") or ''
db_indexs = config_dict['redis_db_index'].split(',')
config_dict['redis_db_index_reg'] = db_indexs[0]
config_dict['redis_db_index_js'] = db_indexs[1]
config_dict['redis_db_index_chart'] = db_indexs[2]
config_dict['redis_db_index_reg'] = redis_configs.get('registry_db_index') or 1
config_dict['redis_db_index_js'] = redis_configs.get('jobservice_db_index') or 2
config_dict['redis_db_index_chart'] = redis_configs.get('chartmuseum_db_index') or 3
else:
config_dict['redis_host'] = ''
config_dict['redis_port'] = ''
## Using local redis
config_dict['redis_host'] = 'redis'
config_dict['redis_port'] = 6379
config_dict['redis_password'] = ''
config_dict['redis_db_index'] = ''
config_dict['redis_db_index_reg'] = ''
config_dict['redis_db_index_js'] = ''
config_dict['redis_db_index_chart'] = ''
config_dict['redis_db_index_reg'] = 1
config_dict['redis_db_index_js'] = 2
config_dict['redis_db_index_chart'] = 3
# redis://[arbitrary_username:password@]ipaddress:port/database_index
if config_dict.get('redis_password'):
@ -198,26 +184,10 @@ def parse_yaml_config(config_file_path):
config_dict['redis_url_js'] = "redis://%s:%s/%s" % (config_dict['redis_host'], config_dict['redis_port'], config_dict['redis_db_index_js'])
config_dict['redis_url_reg'] = "redis://%s:%s/%s" % (config_dict['redis_host'], config_dict['redis_port'], config_dict['redis_db_index_reg'])
if configs.get("skip_reload_env_pattern"):
config_dict['skip_reload_env_pattern'] = configs["skip_reload_env_pattern"]
else:
config_dict['skip_reload_env_pattern'] = "$^"
# Registry storage configs
storage_config = configs.get('storage')
if storage_config:
config_dict['storage_provider_name'] = storage_config.get("registry_storage_provider_name") or ''
config_dict['storage_provider_config'] = storage_config.get("registry_storage_provider_config") or ''
# yaml requires 1 or more spaces between the key and value
config_dict['storage_provider_config'] = config_dict['storage_provider_config'].replace(":", ": ", 1)
config_dict['registry_custom_ca_bundle_path'] = storage_config.get("registry_custom_ca_bundle") or ''
else:
config_dict['storage_provider_name'] = ''
config_dict['storage_provider_config'] = ''
config_dict['registry_custom_ca_bundle_path'] = ''
# auto generate secret string
# auto generated secret string for core
config_dict['core_secret'] = generate_random_string(16)
config_dict['jobservice_secret'] = generate_random_string(16)
# Admiral configs
config_dict['admiral_url'] = configs.get("admiral_url") or ""
return config_dict

2
make/photon/prepare/utils/core.py
View File

@ -43,7 +43,6 @@ def copy_core_config(core_templates_path, core_config_path):
def render_config_env(config_dict, with_notary, with_clair, with_chartmuseum):
# Use reload_key to avoid reload config after restart harbor
reload_key = generate_random_string(6) if config_dict['reload_config'] == "true" else ""
render_jinja(
core_config_env_template,
@ -51,6 +50,5 @@ def render_config_env(config_dict, with_notary, with_clair, with_chartmuseum):
with_notary=with_notary,
with_clair=with_clair,
with_chartmuseum=with_chartmuseum,
reload_key=reload_key,
**config_dict
)

15
make/photon/prepare/utils/docker_compose.py
View File

@ -25,13 +25,22 @@ def prepare_docker_compose(configs, with_clair, with_notary, with_chartmuseum):
'chartmuseum_version': '{}-{}'.format(CHARTMUSEUM_VERSION, VERSION_TAG),
'data_volume': configs['data_volume'],
'log_location': configs['log_location'],
'cert_key_path': configs['cert_key_path'],
'cert_path': configs['cert_path'],
'protocol': configs['protocol'],
'registry_custom_ca_bundle_storage_path': configs['registry_custom_ca_bundle_path'],
'http_port': configs['http_port'],
'registry_custom_ca_bundle_path': configs['registry_custom_ca_bundle_path'],
'with_notary': with_notary,
'with_clair': with_clair,
'with_chartmuseum': with_chartmuseum
}
storage_config = configs.get('storage_provider_config') or {}
if storage_config.get('keyfile'):
rendering_variables['gcs_keyfile'] = storage_config['keyfile']
if configs.get('https_port'):
rendering_variables['https_port'] = configs['https_port']
if configs['protocol'] == 'https':
rendering_variables['cert_key_path'] = configs['cert_key_path']
rendering_variables['cert_path'] = configs['cert_path']
render_jinja(docker_compose_template_path, docker_compose_yml_path, **rendering_variables)

7
make/photon/prepare/utils/misc.py
View File

@ -36,13 +36,6 @@ def validate(conf, **kwargs):
raise Exception(
"Error: The path for certificate key: %s is invalid" % cert_key_path)
# Project validate
project_creation = conf.get(
"configuration", "project_creation_restriction")
if project_creation != "everyone" and project_creation != "adminonly":
raise Exception(
"Error invalid value for project_creation_restriction: %s" % project_creation)
# Storage validate
valid_storage_drivers = ["filesystem",
"azure", "gcs", "s3", "swift", "oss"]

8
make/photon/prepare/utils/nginx.py
View File

@ -22,11 +22,13 @@ def prepare_nginx(config_dict):
def render_nginx_template(config_dict):
if config_dict['protocol'] == "https":
render_jinja(nginx_https_conf_template, nginx_conf,
ssl_cert = SSL_CERT_PATH,
ssl_cert_key = SSL_CERT_KEY_PATH)
ssl_cert=SSL_CERT_PATH,
ssl_cert_key=SSL_CERT_KEY_PATH)
location_file_pattern = CUSTOM_NGINX_LOCATION_FILE_PATTERN_HTTPS
else:
render_jinja(nginx_http_conf_template, nginx_conf)
render_jinja(
nginx_http_conf_template,
nginx_conf)
location_file_pattern = CUSTOM_NGINX_LOCATION_FILE_PATTERN_HTTP
copy_nginx_location_configs_if_exist(nginx_template_ext_dir, nginx_confd_dir, location_file_pattern)

4
make/photon/prepare/utils/notary.py
View File

@ -19,7 +19,7 @@ notary_signer_env_path = os.path.join(notary_config_dir, "signer_env")
notary_server_env_path = os.path.join(notary_config_dir, "server_env")
def prepare_env_notary(customize_crt, nginx_config_dir):
def prepare_env_notary(nginx_config_dir):
notary_config_dir = prepare_config_dir(config_dir, "notary")
old_signer_cert_secret_path = pathlib.Path(os.path.join(config_dir, 'notary-signer.crt'))
old_signer_key_secret_path = pathlib.Path(os.path.join(config_dir, 'notary-signer.key'))
@ -87,7 +87,7 @@ def prepare_env_notary(customize_crt, nginx_config_dir):
def prepare_notary(config_dict, nginx_config_dir, ssl_cert_path, ssl_cert_key_path):
prepare_env_notary(config_dict['customize_crt'], nginx_config_dir)
prepare_env_notary(nginx_config_dir)
render_jinja(
notary_signer_pg_template,

32
make/photon/prepare/utils/registry.py
View File

@ -1,4 +1,4 @@
import os, shutil
import os, copy
from g import config_dir, templates_dir, DEFAULT_GID, DEFAULT_UID
from utils.misc import prepare_config_dir
@ -11,12 +11,11 @@ registry_conf = os.path.join(config_dir, "registry", "config.yml")
def prepare_registry(config_dict):
prepare_registry_config_dir()
prepare_config_dir(registry_config_dir)
storage_provider_info = get_storage_provider_info(
config_dict['storage_provider_name'],
config_dict['storage_provider_config'],
registry_config_dir)
config_dict['storage_provider_config'])
render_jinja(
registry_config_template_path,
@ -26,26 +25,17 @@ def prepare_registry(config_dict):
storage_provider_info=storage_provider_info,
**config_dict)
def prepare_registry_config_dir():
prepare_config_dir(registry_config_dir)
def get_storage_provider_info(provider_name, provider_config, registry_config_dir_path):
def get_storage_provider_info(provider_name, provider_config):
provider_config_copy = copy.deepcopy(provider_config)
if provider_name == "filesystem":
if not provider_config:
storage_provider_config = "rootdirectory: /storage"
elif "rootdirectory:" not in storage_provider_config:
storage_provider_config = "rootdirectory: /storage" + "," + storage_provider_config
if not (provider_config_copy and provider_config_copy.has_key('rootdirectory')):
provider_config_copy['rootdirectory'] = '/storage'
if provider_name == 'gcs' and provider_config_copy.get('keyfile'):
provider_config_copy['keyfile'] = '/etc/registry/gcs.key'
# generate storage configuration section in yaml format
storage_provider_conf_list = [provider_name + ':']
for c in storage_provider_config.split(","):
kvs = c.split(": ")
if len(kvs) == 2:
if kvs[0].strip() == "keyfile":
srcKeyFile = kvs[1].strip()
if os.path.isfile(srcKeyFile):
shutil.copyfile(srcKeyFile, os.path.join(registry_config_dir_path, "gcs.key"))
storage_provider_conf_list.append("keyfile: %s" % "/etc/registry/gcs.key")
continue
storage_provider_conf_list.append(c.strip())
for config in provider_config_copy.items():
storage_provider_conf_list.append('{}: {}'.format(*config))
storage_provider_info = ('\n' + ' ' * 4).join(storage_provider_conf_list)
return storage_provider_info

11
make/photon/prepare/utils/uaa.py
View File

@ -1,11 +0,0 @@
import os, shutil
def prepare_uaa_cert_file(uaa_ca_cert, core_cert_dir):
if os.path.isfile(uaa_ca_cert):
if not os.path.isdir(core_cert_dir):
os.makedirs(core_cert_dir)
core_uaa_ca = os.path.join(core_cert_dir, "uaa_ca.pem")
print("Copying UAA CA cert to %s" % core_uaa_ca)
shutil.copyfile(uaa_ca_cert, core_uaa_ca)
else:
print("Can not find UAA CA cert: %s, skip" % uaa_ca_cert)

2
make/photon/registry/Dockerfile.binary
View File

@ -1,7 +1,7 @@
FROM golang:1.11
ENV DISTRIBUTION_DIR /go/src/github.com/docker/distribution
ENV DOCKER_BUILDTAGS include_oss include_gcs
ENV BUILDTAGS include_oss include_gcs
WORKDIR $DISTRIBUTION_DIR
COPY . $DISTRIBUTION_DIR

51
make/prepare
View File

@ -1,39 +1,40 @@
#!/bin/bash
set +e
# If compling source code this dir is harbor's make dir
# If install harbor via pacakge, this dir is harbor's root dir
harbor_prepare_path="$( cd "$(dirname "$0")" ; pwd -P )"
echo host make path is set to ${harbor_prepare_path}
echo "prepare base dir is set to ${harbor_prepare_path}"
data_path=$(grep '^[^#]*data_volume:' ${harbor_prepare_path}/harbor.yml | awk '{print $NF}')
log_path=$(grep '^[^#]*location:' ${harbor_prepare_path}/harbor.yml | awk '{print $NF}')
secretkey_path=$(grep '^[^#]*secretkey_path:' ${harbor_prepare_path}/harbor.yml | awk '{print $NF}')
ssl_cert_path=$(grep '^[^#]*ssl_cert:' ${harbor_prepare_path}/harbor.yml | awk '{print $NF}')
ssl_cert_key_path=$(grep '^[^#]*ssl_cert_key:' ${harbor_prepare_path}/harbor.yml | awk '{print $NF}')
registry_custom_ca_bundle=$(grep '^[^#]*registry_custom_ca_bundle:' ${harbor_prepare_path}/harbor.yml | awk '{print $NF}')
# If previous secretkeys exist, move it to new location
previous_secretkey_path=/data/secretkey
previous_defaultalias_path=/data/defaultalias
if [ -f $previous_secretkey_path ]; then
mkdir -p $data_path/secret/keys
mv $previous_secretkey_path $data_path/secret/keys
fi
if [ -f $previous_defaultalias_path ]; then
mkdir -p $data_path/secret/keys
mv $previous_defaultalias_path $data_path/secret/keys
fi
# Clean up input dir
rm -rf ${harbor_prepare_path}/input
# Create a input dirs
mkdir -p ${harbor_prepare_path}/input
input_dir=${harbor_prepare_path}/input
mkdir -p $input_dir/nginx
mkdir -p $input_dir/keys
mkdir -p $input_dir/common
# Copy nginx config file to input dir
cp $ssl_cert_path $input_dir/nginx/server.crt
cp $ssl_cert_key_path $input_dir/nginx/server.key
# Copy secretkey to input dir
cp -r $secretkey_path $input_dir/keys
# Copy ca bundle to input dir
if [ -f $registry_custom_ca_bundle ]
then
cp -r $registry_custom_ca_bundle $input_dir/common/custom-ca-bundle.crt
fi
set -e
# Copy harbor.yml to input dir
cp ${harbor_prepare_path}/harbor.yml $input_dir/harbor.yml
if [[ ! "$1" =~ ^\-\- ]] && [ -f "$1" ]
then
cp $1 $input_dir/harbor.yml
else
cp ${harbor_prepare_path}/harbor.yml $input_dir/harbor.yml
fi
# Create secret dir
secret_dir=${data_path}/secret
@ -44,8 +45,8 @@ docker run -it --rm -v $input_dir:/input \
-v $harbor_prepare_path:/compose_location \
-v $config_dir:/config \
-v $secret_dir:/secret \
-v $log_path:/var/log/harbor \
goharbor/prepare:dev $@
echo "Clean up the input dir"
# Clean up input dir
rm -rf ${harbor_prepare_path}/input
rm -rf ${harbor_prepare_path}/input

17
src/Gopkg.lock generated
View File

@ -81,6 +81,14 @@
pruneopts = "UT"
revision = "e87155e8f0c05bf323d0b13470e1b97af0cb5652"
[[projects]]
digest = "1:2aaf2cc045d0219bba79655e4df795b973168c310574669cb75786684f7287d3"
name = "github.com/bmatcuk/doublestar"
packages = ["."]
pruneopts = "UT"
revision = "85a78806aa1b4707d1dbace9be592cf1ece91ab3"
version = "v1.1.1"
[[projects]]
digest = "1:76ca0dfcbf951d1868c7449453981dba9e1f79034706d1500a5a785000f5f222"
name = "github.com/casbin/casbin"
@ -651,11 +659,10 @@
revision = "f534d624797b270e5e46104dc7e2c2d61edbb85d"
[[projects]]
digest = "1:b2a0bdcfc59bed6a64d3ade946f9bf807f8fcd105892d940a008b0b2816babe5"
digest = "1:131682c26796b64f0abb77ac3d85525712706fde0b085aaa7b6d10b4398167cc"
name = "k8s.io/client-go"
packages = [
"kubernetes/scheme",
"kubernetes/typed/authentication/v1beta1",
"pkg/apis/clientauthentication",
"pkg/apis/clientauthentication/v1alpha1",
"pkg/apis/clientauthentication/v1beta1",
@ -714,6 +721,7 @@
"github.com/astaxie/beego/session/redis",
"github.com/astaxie/beego/validation",
"github.com/beego/i18n",
"github.com/bmatcuk/doublestar",
"github.com/casbin/casbin",
"github.com/casbin/casbin/model",
"github.com/casbin/casbin/persist",
@ -729,7 +737,6 @@
"github.com/docker/distribution/reference",
"github.com/docker/distribution/registry/auth/token",
"github.com/docker/distribution/registry/client/auth/challenge",
"github.com/docker/distribution/uuid",
"github.com/docker/libtrust",
"github.com/docker/notary",
"github.com/docker/notary/client",
@ -760,7 +767,9 @@
"gopkg.in/yaml.v2",
"k8s.io/api/authentication/v1beta1",
"k8s.io/apimachinery/pkg/apis/meta/v1",
"k8s.io/client-go/kubernetes/typed/authentication/v1beta1",
"k8s.io/apimachinery/pkg/runtime/schema",
"k8s.io/apimachinery/pkg/runtime/serializer",
"k8s.io/client-go/kubernetes/scheme",
"k8s.io/client-go/rest",
"k8s.io/helm/cmd/helm/search",
"k8s.io/helm/pkg/chartutil",

4
src/Gopkg.toml
View File

@ -123,3 +123,7 @@ ignored = ["github.com/goharbor/harbor/tests*"]
[[constraint]]
name = "k8s.io/api"
version = "kubernetes-1.13.4"
[[constraint]]
name = "github.com/bmatcuk/doublestar"
version = "1.1.1"

37
src/chartserver/handler_manipulation.go
View File

@ -7,7 +7,14 @@ import (
"strings"
"github.com/ghodss/yaml"
"github.com/goharbor/harbor/src/replication"
rep_event "github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/replication/model"
helm_repo "k8s.io/helm/pkg/repo"
"os"
"github.com/goharbor/harbor/src/common/utils/log"
)
// ListCharts gets the chart list under the namespace
@ -63,7 +70,35 @@ func (c *Controller) DeleteChartVersion(namespace, chartName, version string) er
url := fmt.Sprintf("%s/%s/%s", c.APIPrefix(namespace), chartName, version)
return c.apiClient.DeleteContent(url)
err := c.apiClient.DeleteContent(url)
if err != nil {
return err
}
// send notification to replication handler
// Todo: it used as the replacement of webhook, will be removed when webhook to be introduced.
if os.Getenv("UTTEST") != "true" {
go func() {
e := &rep_event.Event{
Type: rep_event.EventTypeChartDelete,
Resource: &model.Resource{
Type: model.ResourceTypeChart,
Deleted: true,
Metadata: &model.ResourceMetadata{
Repository: &model.Repository{
Name: fmt.Sprintf("%s/%s", namespace, chartName),
},
Vtags: []string{version},
},
},
}
if err := replication.EventHandler.Handle(e); err != nil {
log.Errorf("failed to handle event: %v", err)
}
}()
}
return nil
}
// GetChartVersion returns the summary of the specified chart version.

36
src/chartserver/reverse_proxy.go
View File

@ -12,6 +12,12 @@ import (
"os"
"strconv"
"strings"
"github.com/goharbor/harbor/src/common"
hlog "github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/replication"
rep_event "github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/replication/model"
)
const (
@ -80,6 +86,36 @@ func director(target *url.URL, cred *Credential, req *http.Request) {
// Modify the http response
func modifyResponse(res *http.Response) error {
// Upload chart success, then to the notification to replication handler
if res.StatusCode == http.StatusCreated {
// 201 and has chart_upload(namespace-repository-version) context
// means this response is for uploading chart success.
chartUpload := res.Request.Context().Value(common.ChartUploadCtxKey).(string)
if chartUpload != "" {
chartUploadSplitted := strings.Split(chartUpload, ":")
if len(chartUploadSplitted) == 3 {
// Todo: it used as the replacement of webhook, will be removed when webhook to be introduced.
go func() {
e := &rep_event.Event{
Type: rep_event.EventTypeChartUpload,
Resource: &model.Resource{
Type: model.ResourceTypeChart,
Metadata: &model.ResourceMetadata{
Repository: &model.Repository{
Name: fmt.Sprintf("%s/%s", chartUploadSplitted[0], chartUploadSplitted[1]),
},
Vtags: []string{chartUploadSplitted[2]},
},
},
}
if err := replication.EventHandler.Handle(e); err != nil {
hlog.Errorf("failed to handle event: %v", err)
}
}()
}
}
}
// Accept cases
// Success or redirect
if res.StatusCode >= http.StatusOK && res.StatusCode <= http.StatusTemporaryRedirect {

168
src/common/api/base.go
View File

@ -24,6 +24,7 @@ import (
commonhttp "github.com/goharbor/harbor/src/common/http"
"github.com/goharbor/harbor/src/common/utils/log"
"errors"
"github.com/astaxie/beego"
)
@ -48,68 +49,6 @@ func (b *BaseAPI) GetInt64FromPath(key string) (int64, error) {
return strconv.ParseInt(value, 10, 64)
}
// HandleNotFound ...
func (b *BaseAPI) HandleNotFound(text string) {
log.Info(text)
b.RenderError(http.StatusNotFound, text)
}
// HandleUnauthorized ...
func (b *BaseAPI) HandleUnauthorized() {
log.Info("unauthorized")
b.RenderError(http.StatusUnauthorized, "")
}
// HandleForbidden ...
func (b *BaseAPI) HandleForbidden(text string) {
log.Infof("forbidden: %s", text)
b.RenderError(http.StatusForbidden, text)
}
// HandleBadRequest ...
func (b *BaseAPI) HandleBadRequest(text string) {
log.Info(text)
b.RenderError(http.StatusBadRequest, text)
}
// HandleStatusPreconditionFailed ...
func (b *BaseAPI) HandleStatusPreconditionFailed(text string) {
log.Info(text)
b.RenderError(http.StatusPreconditionFailed, text)
}
// HandleConflict ...
func (b *BaseAPI) HandleConflict(text ...string) {
msg := ""
if len(text) > 0 {
msg = text[0]
}
log.Infof("conflict: %s", msg)
b.RenderError(http.StatusConflict, msg)
}
// HandleInternalServerError ...
func (b *BaseAPI) HandleInternalServerError(text string) {
log.Error(text)
b.RenderError(http.StatusInternalServerError, "")
}
// ParseAndHandleError : if the err is an instance of utils/error.Error,
// return the status code and the detail message contained in err, otherwise
// return 500
func (b *BaseAPI) ParseAndHandleError(text string, err error) {
if err == nil {
return
}
log.Errorf("%s: %v", text, err)
if e, ok := err.(*commonhttp.Error); ok {
b.RenderError(e.Code, e.Message)
return
}
b.RenderError(http.StatusInternalServerError, "")
}
// Render returns nil as it won't render template
func (b *BaseAPI) Render() error {
return nil
@ -120,23 +59,35 @@ func (b *BaseAPI) RenderError(code int, text string) {
http.Error(b.Ctx.ResponseWriter, text, code)
}
// RenderFormattedError renders errors with well formatted style
func (b *BaseAPI) RenderFormattedError(errorCode int, errorMsg string) {
error := commonhttp.Error{
Code: errorCode,
Message: errorMsg,
}
formattedErrMsg := error.String()
log.Errorf("%s %s failed with error: %s", b.Ctx.Request.Method, b.Ctx.Request.URL.String(), formattedErrMsg)
b.RenderError(error.Code, formattedErrMsg)
}
// DecodeJSONReq decodes a json request
func (b *BaseAPI) DecodeJSONReq(v interface{}) {
func (b *BaseAPI) DecodeJSONReq(v interface{}) error {
err := json.Unmarshal(b.Ctx.Input.CopyBody(1<<32), v)
if err != nil {
log.Errorf("Error while decoding the json request, error: %v, %v",
err, string(b.Ctx.Input.CopyBody(1 << 32)[:]))
b.CustomAbort(http.StatusBadRequest, "Invalid json request")
return errors.New("Invalid json request")
}
return nil
}
// Validate validates v if it implements interface validation.ValidFormer
func (b *BaseAPI) Validate(v interface{}) {
func (b *BaseAPI) Validate(v interface{}) (bool, error) {
validator := validation.Validation{}
isValid, err := validator.Valid(v)
if err != nil {
log.Errorf("failed to validate: %v", err)
b.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
return false, err
}
if !isValid {
@ -144,14 +95,17 @@ func (b *BaseAPI) Validate(v interface{}) {
for _, e := range validator.Errors {
message += fmt.Sprintf("%s %s \n", e.Field, e.Message)
}
b.CustomAbort(http.StatusBadRequest, message)
return false, errors.New(message)
}
return true, nil
}
// DecodeJSONReqAndValidate does both decoding and validation
func (b *BaseAPI) DecodeJSONReqAndValidate(v interface{}) {
b.DecodeJSONReq(v)
b.Validate(v)
func (b *BaseAPI) DecodeJSONReqAndValidate(v interface{}) (bool, error) {
if err := b.DecodeJSONReq(v); err != nil {
return false, err
}
return b.Validate(v)
}
// Redirect does redirection to resource URI with http header status code.
@ -163,18 +117,18 @@ func (b *BaseAPI) Redirect(statusCode int, resouceID string) {
}
// GetIDFromURL checks the ID in request URL
func (b *BaseAPI) GetIDFromURL() int64 {
func (b *BaseAPI) GetIDFromURL() (int64, error) {
idStr := b.Ctx.Input.Param(":id")
if len(idStr) == 0 {
b.CustomAbort(http.StatusBadRequest, "invalid ID in URL")
return 0, errors.New("invalid ID in URL")
}
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil || id <= 0 {
b.CustomAbort(http.StatusBadRequest, "invalid ID in URL")
return 0, errors.New("invalid ID in URL")
}
return id
return id, nil
}
// SetPaginationHeader set"Link" and "X-Total-Count" header for pagination request
@ -213,15 +167,15 @@ func (b *BaseAPI) SetPaginationHeader(total, page, pageSize int64) {
}
// GetPaginationParams ...
func (b *BaseAPI) GetPaginationParams() (page, pageSize int64) {
page, err := b.GetInt64("page", 1)
func (b *BaseAPI) GetPaginationParams() (page, pageSize int64, err error) {
page, err = b.GetInt64("page", 1)
if err != nil || page <= 0 {
b.CustomAbort(http.StatusBadRequest, "invalid page")
return 0, 0, errors.New("invalid page")
}
pageSize, err = b.GetInt64("page_size", defaultPageSize)
if err != nil || pageSize <= 0 {
b.CustomAbort(http.StatusBadRequest, "invalid page_size")
return 0, 0, errors.New("invalid page_size")
}
if pageSize > maxPageSize {
@ -229,5 +183,61 @@ func (b *BaseAPI) GetPaginationParams() (page, pageSize int64) {
log.Debugf("the parameter page_size %d exceeds the max %d, set it to max", pageSize, maxPageSize)
}
return page, pageSize
return page, pageSize, nil
}
// ParseAndHandleError : if the err is an instance of utils/error.Error,
// return the status code and the detail message contained in err, otherwise
// return 500
func (b *BaseAPI) ParseAndHandleError(text string, err error) {
if err == nil {
return
}
log.Errorf("%s: %v", text, err)
if e, ok := err.(*commonhttp.Error); ok {
b.RenderFormattedError(e.Code, e.Message)
return
}
b.SendInternalServerError(errors.New(""))
}
// SendUnAuthorizedError sends unauthorized error to the client.
func (b *BaseAPI) SendUnAuthorizedError(err error) {
b.RenderFormattedError(http.StatusUnauthorized, err.Error())
}
// SendConflictError sends conflict error to the client.
func (b *BaseAPI) SendConflictError(err error) {
b.RenderFormattedError(http.StatusConflict, err.Error())
}
// SendNotFoundError sends not found error to the client.
func (b *BaseAPI) SendNotFoundError(err error) {
b.RenderFormattedError(http.StatusNotFound, err.Error())
}
// SendBadRequestError sends bad request error to the client.
func (b *BaseAPI) SendBadRequestError(err error) {
b.RenderFormattedError(http.StatusBadRequest, err.Error())
}
// SendInternalServerError sends internal server error to the client.
func (b *BaseAPI) SendInternalServerError(err error) {
log.Error(err.Error())
b.RenderFormattedError(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
}
// SendForbiddenError sends forbidden error to the client.
func (b *BaseAPI) SendForbiddenError(err error) {
b.RenderFormattedError(http.StatusForbidden, err.Error())
}
// SendPreconditionFailedError sends forbidden error to the client.
func (b *BaseAPI) SendPreconditionFailedError(err error) {
b.RenderFormattedError(http.StatusPreconditionFailed, err.Error())
}
// SendStatusServiceUnavailableError sends forbidden error to the client.
func (b *BaseAPI) SendStatusServiceUnavailableError(err error) {
b.RenderFormattedError(http.StatusServiceUnavailable, err.Error())
}

10
src/common/config/keyprovider.go
View File

@ -46,3 +46,13 @@ func (f *FileKeyProvider) Get(params map[string]interface{}) (string, error) {
}
return string(b), nil
}
// PresetKeyProvider returns the preset key disregarding the parm, this is for testing only
type PresetKeyProvider struct {
Key string
}
// Get ...
func (p *PresetKeyProvider) Get(params map[string]interface{}) (string, error) {
return p.Key, nil
}

10
src/common/config/keyprovider_test.go
View File

@ -15,6 +15,7 @@
package config
import (
"github.com/stretchr/testify/assert"
"io/ioutil"
"os"
"testing"
@ -42,3 +43,12 @@ func TestGetOfFileKeyProvider(t *testing.T) {
return
}
}
func TestPresetKeyProvider(t *testing.T) {
kp := &PresetKeyProvider{
Key: "mykey",
}
k, err := kp.Get(nil)
assert.Nil(t, err)
assert.Equal(t, "mykey", k)
}

4
src/common/config/metadata/metadatalist.go
View File

@ -133,7 +133,7 @@ var (
{Name: common.HTTPAuthProxyEndpoint, Scope: UserScope, Group: HTTPAuthGroup, ItemType: &StringType{}},
{Name: common.HTTPAuthProxyTokenReviewEndpoint, Scope: UserScope, Group: HTTPAuthGroup, ItemType: &StringType{}},
{Name: common.HTTPAuthProxySkipCertVerify, Scope: UserScope, Group: HTTPAuthGroup, DefaultValue: "false", ItemType: &BoolType{}},
{Name: common.HTTPAuthProxyVerifyCert, Scope: UserScope, Group: HTTPAuthGroup, DefaultValue: "true", ItemType: &BoolType{}},
{Name: common.HTTPAuthProxyAlwaysOnboard, Scope: UserScope, Group: HTTPAuthGroup, DefaultValue: "false", ItemType: &BoolType{}},
{Name: common.OIDCName, Scope: UserScope, Group: OIDCGroup, ItemType: &StringType{}},
@ -141,7 +141,7 @@ var (
{Name: common.OIDCCLientID, Scope: UserScope, Group: OIDCGroup, ItemType: &StringType{}},
{Name: common.OIDCClientSecret, Scope: UserScope, Group: OIDCGroup, ItemType: &PasswordType{}},
{Name: common.OIDCScope, Scope: UserScope, Group: OIDCGroup, ItemType: &StringType{}},
{Name: common.OIDCSkipCertVerify, Scope: UserScope, Group: OIDCGroup, DefaultValue: "false", ItemType: &BoolType{}},
{Name: common.OIDCVerifyCert, Scope: UserScope, Group: OIDCGroup, DefaultValue: "true", ItemType: &BoolType{}},
{Name: "with_chartmuseum", Scope: SystemScope, Group: BasicGroup, EnvKey: "WITH_CHARTMUSEUM", DefaultValue: "false", ItemType: &BoolType{}, Editable: true},
{Name: "with_clair", Scope: SystemScope, Group: BasicGroup, EnvKey: "WITH_CLAIR", DefaultValue: "false", ItemType: &BoolType{}, Editable: true},

5
src/common/config/store/driver/db.go
View File

@ -15,13 +15,14 @@
package driver
import (
"os"
"github.com/goharbor/harbor/src/common/config/encrypt"
"github.com/goharbor/harbor/src/common/config/metadata"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"os"
)
// Database - Used to load/save configuration in database
@ -63,7 +64,7 @@ func (d *Database) Save(cfgs map[string]interface{}) error {
for key, value := range cfgs {
if item, ok := metadata.Instance().GetByName(key); ok {
if os.Getenv("UTTEST") != "true" && item.Scope == metadata.SystemScope {
log.Errorf("system setting can not updated, key %v", key)
// skip to save system setting to db
continue
}
strValue := utils.GetStrValueOfAnyType(value)

8
src/common/const.go
View File

@ -14,6 +14,8 @@
package common
type contextKey string
// const variables
const (
DBAuth = "db_auth"
@ -98,13 +100,13 @@ const (
UAAVerifyCert = "uaa_verify_cert"
HTTPAuthProxyEndpoint = "http_authproxy_endpoint"
HTTPAuthProxyTokenReviewEndpoint = "http_authproxy_tokenreview_endpoint"
HTTPAuthProxySkipCertVerify = "http_authproxy_skip_cert_verify"
HTTPAuthProxyVerifyCert = "http_authproxy_verify_cert"
HTTPAuthProxyAlwaysOnboard = "http_authproxy_always_onboard"
OIDCName = "oidc_name"
OIDCEndpoint = "oidc_endpoint"
OIDCCLientID = "oidc_client_id"
OIDCClientSecret = "oidc_client_secret"
OIDCSkipCertVerify = "oidc_skip_cert_verify"
OIDCVerifyCert = "oidc_verify_cert"
OIDCScope = "oidc_scope"
DefaultClairEndpoint = "http://clair:6060"
@ -136,4 +138,6 @@ const (
RobotTokenDuration = "robot_token_duration"
OIDCCallbackPath = "/c/oidc/callback"
ChartUploadCtxKey = contextKey("chart_upload")
)

1476
src/common/dao/dao_test.go
View File

File diff suppressed because it is too large Load Diff

424
src/common/dao/replication_job.go
View File

@ -1,424 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package dao
import (
"time"
"strings"
"github.com/astaxie/beego/orm"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils/log"
)
// AddRepTarget ...
func AddRepTarget(target models.RepTarget) (int64, error) {
o := GetOrmer()
sql := "insert into replication_target (name, url, username, password, insecure, target_type) values (?, ?, ?, ?, ?, ?) RETURNING id"
var targetID int64
err := o.Raw(sql, target.Name, target.URL, target.Username, target.Password, target.Insecure, target.Type).QueryRow(&targetID)
if err != nil {
return 0, err
}
return targetID, nil
}
// GetRepTarget ...
func GetRepTarget(id int64) (*models.RepTarget, error) {
o := GetOrmer()
t := models.RepTarget{ID: id}
err := o.Read(&t)
if err == orm.ErrNoRows {
return nil, nil
}
return &t, err
}
// GetRepTargetByName ...
func GetRepTargetByName(name string) (*models.RepTarget, error) {
o := GetOrmer()
t := models.RepTarget{Name: name}
err := o.Read(&t, "Name")
if err == orm.ErrNoRows {
return nil, nil
}
return &t, err
}
// GetRepTargetByEndpoint ...
func GetRepTargetByEndpoint(endpoint string) (*models.RepTarget, error) {
o := GetOrmer()
t := models.RepTarget{
URL: endpoint,
}
err := o.Read(&t, "URL")
if err == orm.ErrNoRows {
return nil, nil
}
return &t, err
}
// DeleteRepTarget ...
func DeleteRepTarget(id int64) error {
o := GetOrmer()
_, err := o.Delete(&models.RepTarget{ID: id})
return err
}
// UpdateRepTarget ...
func UpdateRepTarget(target models.RepTarget) error {
o := GetOrmer()
sql := `update replication_target
set url = ?, name = ?, username = ?, password = ?, insecure = ?, update_time = ?
where id = ?`
_, err := o.Raw(sql, target.URL, target.Name, target.Username, target.Password, target.Insecure, time.Now(), target.ID).Exec()
return err
}
// FilterRepTargets filters targets by name
func FilterRepTargets(name string) ([]*models.RepTarget, error) {
o := GetOrmer()
var args []interface{}
sql := `select * from replication_target `
if len(name) != 0 {
sql += `where name like ? `
args = append(args, "%"+Escape(name)+"%")
}
sql += `order by creation_time`
var targets []*models.RepTarget
if _, err := o.Raw(sql, args).QueryRows(&targets); err != nil {
return nil, err
}
return targets, nil
}
// AddRepPolicy ...
func AddRepPolicy(policy models.RepPolicy) (int64, error) {
o := GetOrmer()
sql := `insert into replication_policy (name, project_id, target_id, enabled, description, cron_str, creation_time, update_time, filters, replicate_deletion)
values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`
params := []interface{}{}
now := time.Now()
params = append(params, policy.Name, policy.ProjectID, policy.TargetID, true,
policy.Description, policy.Trigger, now, now, policy.Filters,
policy.ReplicateDeletion)
var policyID int64
err := o.Raw(sql, params...).QueryRow(&policyID)
if err != nil {
return 0, err
}
return policyID, nil
}
// GetRepPolicy ...
func GetRepPolicy(id int64) (*models.RepPolicy, error) {
o := GetOrmer()
sql := `select * from replication_policy where id = ? and deleted = false`
var policy models.RepPolicy
if err := o.Raw(sql, id).QueryRow(&policy); err != nil {
if err == orm.ErrNoRows {
return nil, nil
}
return nil, err
}
return &policy, nil
}
// GetTotalOfRepPolicies returns the total count of replication policies
func GetTotalOfRepPolicies(name string, projectID int64) (int64, error) {
qs := GetOrmer().QueryTable(&models.RepPolicy{}).Filter("deleted", false)
if len(name) != 0 {
qs = qs.Filter("name__icontains", name)
}
if projectID != 0 {
qs = qs.Filter("project_id", projectID)
}
return qs.Count()
}
// FilterRepPolicies filters policies by name and project ID
func FilterRepPolicies(name string, projectID, page, pageSize int64) ([]*models.RepPolicy, error) {
o := GetOrmer()
var args []interface{}
sql := `select rp.id, rp.project_id, rp.target_id,
rt.name as target_name, rp.name, rp.description,
rp.cron_str, rp.filters, rp.replicate_deletion,
rp.creation_time, rp.update_time,
count(rj.status) as error_job_count
from replication_policy rp
left join replication_target rt on rp.target_id=rt.id
left join replication_job rj on rp.id=rj.policy_id and (rj.status='error'
or rj.status='retrying')
where rp.deleted = false `
if len(name) != 0 && projectID != 0 {
sql += `and rp.name like ? and rp.project_id = ? `
args = append(args, "%"+Escape(name)+"%")
args = append(args, projectID)
} else if len(name) != 0 {
sql += `and rp.name like ? `
args = append(args, "%"+Escape(name)+"%")
} else if projectID != 0 {
sql += `and rp.project_id = ? `
args = append(args, projectID)
}
sql += `group by rt.name, rp.id order by rp.creation_time`
if page > 0 && pageSize > 0 {
sql += ` limit ? offset ?`
args = append(args, pageSize, (page-1)*pageSize)
}
var policies []*models.RepPolicy
if _, err := o.Raw(sql, args).QueryRows(&policies); err != nil {
return nil, err
}
return policies, nil
}
// GetRepPolicyByName ...
func GetRepPolicyByName(name string) (*models.RepPolicy, error) {
o := GetOrmer()
sql := `select * from replication_policy where deleted = false and name = ?`
var policy models.RepPolicy
if err := o.Raw(sql, name).QueryRow(&policy); err != nil {
if err == orm.ErrNoRows {
return nil, nil
}
return nil, err
}
return &policy, nil
}
// GetRepPolicyByProject ...
func GetRepPolicyByProject(projectID int64) ([]*models.RepPolicy, error) {
o := GetOrmer()
sql := `select * from replication_policy where deleted = false and project_id = ?`
var policies []*models.RepPolicy
if _, err := o.Raw(sql, projectID).QueryRows(&policies); err != nil {
return nil, err
}
return policies, nil
}
// GetRepPolicyByTarget ...
func GetRepPolicyByTarget(targetID int64) ([]*models.RepPolicy, error) {
o := GetOrmer()
sql := `select * from replication_policy where deleted = false and target_id = ?`
var policies []*models.RepPolicy
if _, err := o.Raw(sql, targetID).QueryRows(&policies); err != nil {
return nil, err
}
return policies, nil
}
// GetRepPolicyByProjectAndTarget ...
func GetRepPolicyByProjectAndTarget(projectID, targetID int64) ([]*models.RepPolicy, error) {
o := GetOrmer()
sql := `select * from replication_policy where deleted = false and project_id = ? and target_id = ?`
var policies []*models.RepPolicy
if _, err := o.Raw(sql, projectID, targetID).QueryRows(&policies); err != nil {
return nil, err
}
return policies, nil
}
// UpdateRepPolicy ...
func UpdateRepPolicy(policy *models.RepPolicy) error {
o := GetOrmer()
sql := `update replication_policy
set project_id = ?, target_id = ?, name = ?, description = ?, cron_str = ?, filters = ?, replicate_deletion = ?, update_time = ?
where id = ?`
_, err := o.Raw(sql, policy.ProjectID, policy.TargetID, policy.Name, policy.Description, policy.Trigger, policy.Filters, policy.ReplicateDeletion, time.Now(), policy.ID).Exec()
return err
}
// DeleteRepPolicy ...
func DeleteRepPolicy(id int64) error {
_, err := GetOrmer().Delete(&models.RepPolicy{
ID: id,
})
return err
}
// AddRepJob ...
func AddRepJob(job models.RepJob) (int64, error) {
o := GetOrmer()
if len(job.Status) == 0 {
job.Status = models.JobPending
}
if len(job.TagList) > 0 {
job.Tags = strings.Join(job.TagList, ",")
}
return o.Insert(&job)
}
// GetRepJob ...
func GetRepJob(id int64) (*models.RepJob, error) {
o := GetOrmer()
j := models.RepJob{ID: id}
err := o.Read(&j)
if err == orm.ErrNoRows {
return nil, nil
}
genTagListForJob(&j)
return &j, nil
}
// GetTotalCountOfRepJobs ...
func GetTotalCountOfRepJobs(query ...*models.RepJobQuery) (int64, error) {
qs := repJobQueryConditions(query...)
return qs.Count()
}
// GetRepJobs ...
func GetRepJobs(query ...*models.RepJobQuery) ([]*models.RepJob, error) {
jobs := []*models.RepJob{}
qs := repJobQueryConditions(query...)
if len(query) > 0 && query[0] != nil {
qs = paginateForQuerySetter(qs, query[0].Page, query[0].Size)
}
qs = qs.OrderBy("-UpdateTime")
if _, err := qs.All(&jobs); err != nil {
return jobs, err
}
genTagListForJob(jobs...)
return jobs, nil
}
func repJobQueryConditions(query ...*models.RepJobQuery) orm.QuerySeter {
qs := GetOrmer().QueryTable(new(models.RepJob))
if len(query) == 0 || query[0] == nil {
return qs
}
q := query[0]
if q.PolicyID != 0 {
qs = qs.Filter("ID", q.PolicyID)
}
if len(q.OpUUID) > 0 {
qs = qs.Filter("OpUUID__exact", q.OpUUID)
}
if len(q.Repository) > 0 {
qs = qs.Filter("Repository__icontains", q.Repository)
}
if len(q.Statuses) > 0 {
qs = qs.Filter("Status__in", q.Statuses)
}
if len(q.Operations) > 0 {
qs = qs.Filter("Operation__in", q.Operations)
}
if q.StartTime != nil {
qs = qs.Filter("CreationTime__gte", q.StartTime)
}
if q.EndTime != nil {
qs = qs.Filter("CreationTime__lte", q.EndTime)
}
return qs
}
// DeleteRepJob ...
func DeleteRepJob(id int64) error {
o := GetOrmer()
_, err := o.Delete(&models.RepJob{ID: id})
return err
}
// DeleteRepJobs deletes replication jobs by policy ID
func DeleteRepJobs(policyID int64) error {
_, err := GetOrmer().QueryTable(&models.RepJob{}).Filter("ID", policyID).Delete()
return err
}
// UpdateRepJobStatus ...
func UpdateRepJobStatus(id int64, status string) error {
o := GetOrmer()
j := models.RepJob{
ID: id,
Status: status,
UpdateTime: time.Now(),
}
n, err := o.Update(&j, "Status", "UpdateTime")
if n == 0 {
log.Warningf("no records are updated when updating replication job %d", id)
}
return err
}
// SetRepJobUUID ...
func SetRepJobUUID(id int64, uuid string) error {
o := GetOrmer()
j := models.RepJob{
ID: id,
UUID: uuid,
}
n, err := o.Update(&j, "UUID")
if n == 0 {
log.Warningf("no records are updated when updating replication job %d", id)
}
return err
}
func genTagListForJob(jobs ...*models.RepJob) {
for _, j := range jobs {
if len(j.Tags) > 0 {
j.TagList = strings.Split(j.Tags, ",")
}
}
}

56
src/common/dao/replication_job_test.go
View File

@ -1,56 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package dao
import (
"testing"
"github.com/goharbor/harbor/src/common/models"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestDeleteRepJobs(t *testing.T) {
var policyID int64 = 999
_, err := AddRepJob(models.RepJob{
PolicyID: policyID,
Repository: "library/hello-world",
Operation: "delete",
Status: "success",
})
require.Nil(t, err)
_, err = AddRepJob(models.RepJob{
PolicyID: policyID,
Repository: "library/hello-world",
Operation: "delete",
Status: "success",
})
require.Nil(t, err)
jobs, err := GetRepJobs(&models.RepJobQuery{
PolicyID: policyID,
})
require.Nil(t, err)
require.Equal(t, 2, len(jobs))
err = DeleteRepJobs(policyID)
require.Nil(t, err)
jobs, err = GetRepJobs(&models.RepJobQuery{
PolicyID: policyID,
})
require.Nil(t, err)
assert.Equal(t, 0, len(jobs))
}

2
src/common/dao/sqlite.go
View File

@ -18,7 +18,7 @@ import (
"fmt"
"github.com/astaxie/beego/orm"
_ "github.com/mattn/go-sqlite3" // register sqlite driver
// _ "github.com/mattn/go-sqlite3" // register sqlite driver
)
type sqlite struct {

71
src/common/dao/watch_item.go
View File

@ -1,71 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package dao
import (
"time"
"github.com/goharbor/harbor/src/common/models"
)
// DefaultDatabaseWatchItemDAO is an instance of DatabaseWatchItemDAO
var DefaultDatabaseWatchItemDAO WatchItemDAO = &DatabaseWatchItemDAO{}
// WatchItemDAO defines operations about WatchItem
type WatchItemDAO interface {
Add(*models.WatchItem) (int64, error)
DeleteByPolicyID(int64) error
Get(namespace, operation string) ([]models.WatchItem, error)
}
// DatabaseWatchItemDAO implements interface WatchItemDAO for database
type DatabaseWatchItemDAO struct{}
// Add a WatchItem
func (d *DatabaseWatchItemDAO) Add(item *models.WatchItem) (int64, error) {
o := GetOrmer()
var triggerID int64
now := time.Now()
sql := "insert into replication_immediate_trigger (policy_id, namespace, on_deletion, on_push, creation_time, update_time) values (?, ?, ?, ?, ?, ?) RETURNING id"
err := o.Raw(sql, item.PolicyID, item.Namespace, item.OnDeletion, item.OnPush, now, now).QueryRow(&triggerID)
if err != nil {
return 0, err
}
return triggerID, nil
}
// DeleteByPolicyID deletes the WatchItem specified by policy ID
func (d *DatabaseWatchItemDAO) DeleteByPolicyID(policyID int64) error {
_, err := GetOrmer().QueryTable(&models.WatchItem{}).Filter("ID", policyID).Delete()
return err
}
// Get returns WatchItem list according to the namespace and operation
func (d *DatabaseWatchItemDAO) Get(namespace, operation string) ([]models.WatchItem, error) {
qs := GetOrmer().QueryTable(&models.WatchItem{}).Filter("Namespace", namespace)
if operation == "push" {
qs = qs.Filter("OnPush", true)
} else if operation == "delete" {
qs = qs.Filter("OnDeletion", true)
}
items := []models.WatchItem{}
_, err := qs.All(&items)
return items, err
}

71
src/common/dao/watch_item_test.go
View File

@ -1,71 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package dao
import (
"testing"
"github.com/goharbor/harbor/src/common/models"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMethodsOfWatchItem(t *testing.T) {
targetID, err := AddRepTarget(models.RepTarget{
Name: "test_target_for_watch_item",
URL: "http://127.0.0.1",
})
require.Nil(t, err)
defer DeleteRepTarget(targetID)
policyID, err := AddRepPolicy(models.RepPolicy{
Name: "test_policy_for_watch_item",
ProjectID: 1,
TargetID: targetID,
})
require.Nil(t, err)
defer DeleteRepPolicy(policyID)
item := &models.WatchItem{
PolicyID: policyID,
Namespace: "library",
OnPush: false,
OnDeletion: true,
}
// test Add
id, err := DefaultDatabaseWatchItemDAO.Add(item)
require.Nil(t, err)
// test Get: operation-push
items, err := DefaultDatabaseWatchItemDAO.Get("library", "push")
require.Nil(t, err)
assert.Equal(t, 0, len(items))
// test Get: operation-delete
items, err = DefaultDatabaseWatchItemDAO.Get("library", "delete")
require.Nil(t, err)
assert.Equal(t, 1, len(items))
assert.Equal(t, id, items[0].ID)
assert.Equal(t, "library", items[0].Namespace)
assert.True(t, items[0].OnDeletion)
// test DeleteByPolicyID
err = DefaultDatabaseWatchItemDAO.DeleteByPolicyID(policyID)
require.Nil(t, err)
items, err = DefaultDatabaseWatchItemDAO.Get("library", "delete")
require.Nil(t, err)
assert.Equal(t, 0, len(items))
}

14
src/common/http/client.go
View File

@ -95,12 +95,16 @@ func (c *Client) Head(url string) error {
func (c *Client) Post(url string, v ...interface{}) error {
var reader io.Reader
if len(v) > 0 {
data, err := json.Marshal(v[0])
if err != nil {
return err
}
if r, ok := v[0].(io.Reader); ok {
reader = r
} else {
data, err := json.Marshal(v[0])
if err != nil {
return err
}
reader = bytes.NewReader(data)
reader = bytes.NewReader(data)
}
}
req, err := http.NewRequest(http.MethodPost, url, reader)

14
src/common/http/error.go
View File

@ -15,16 +15,26 @@
package http
import (
"encoding/json"
"fmt"
)
// Error wrap HTTP status code and message as an error
type Error struct {
Code int
Message string
Code int `json:"code"`
Message string `json:"message"`
}
// Error ...
func (e *Error) Error() string {
return fmt.Sprintf("http error: code %d, message %s", e.Code, e.Message)
}
// String wraps the error msg to the well formatted error message
func (e *Error) String() string {
data, err := json.Marshal(&e)
if err != nil {
return e.Message
}
return string(data)
}

17
src/common/http/error_test.go Normal file
View File

@ -0,0 +1,17 @@
package http
import (
"github.com/stretchr/testify/assert"
"testing"
)
// Test case for error wrapping function.
func TestWrapError(t *testing.T) {
err := Error{
Code: 1,
Message: "test",
}
assert.Equal(t, err.String(), "{\"code\":1,\"message\":\"test\"}")
}

6
src/common/job/const.go
View File

@ -5,12 +5,6 @@ const (
ImageScanJob = "IMAGE_SCAN"
// ImageScanAllJob is the name of "scanall" job in job service
ImageScanAllJob = "IMAGE_SCAN_ALL"
// ImageTransfer : the name of image transfer job in job service
ImageTransfer = "IMAGE_TRANSFER"
// ImageDelete : the name of image delete job in job service
ImageDelete = "IMAGE_DELETE"
// ImageReplicate : the name of image replicate job in job service
ImageReplicate = "IMAGE_REPLICATE"
// ImageGC the name of image garbage collection job in job service
ImageGC = "IMAGE_GC"

2
src/common/job/test/server.go
View File

@ -27,7 +27,7 @@ func currPath() string {
return path.Dir(f)
}
// NewJobServiceServer
// NewJobServiceServer ...
func NewJobServiceServer() *httptest.Server {
mux := http.NewServeMux()
mux.HandleFunc(fmt.Sprintf("%s/%s/log", jobsPrefix, jobUUID),

5
src/common/models/base.go
View File

@ -19,9 +19,7 @@ import (
)
func init() {
orm.RegisterModel(new(RepTarget),
new(RepPolicy),
new(RepJob),
orm.RegisterModel(
new(User),
new(Project),
new(Role),
@ -30,7 +28,6 @@ func init() {
new(RepoRecord),
new(ImgScanOverview),
new(ClairVulnTimestamp),
new(WatchItem),
new(ProjectMetadata),
new(ConfigEntry),
new(Label),

16
src/common/models/config.go
View File

@ -69,19 +69,19 @@ type Email struct {
type HTTPAuthProxy struct {
Endpoint string `json:"endpoint"`
TokenReviewEndpoint string `json:"tokenreivew_endpoint"`
SkipCertVerify bool `json:"skip_cert_verify"`
VerifyCert bool `json:"verify_cert"`
AlwaysOnBoard bool `json:"always_onboard"`
}
// OIDCSetting wraps the settings for OIDC auth endpoint
type OIDCSetting struct {
Name string `json:"name"`
Endpoint string `json:"endpoint"`
SkipCertVerify bool `json:"skip_cert_verify"`
ClientID string `json:"client_id"`
ClientSecret string `json:"client_secret"`
RedirectURL string `json:"redirect_url"`
Scope []string `json:"scope"`
Name string `json:"name"`
Endpoint string `json:"endpoint"`
VerifyCert bool `json:"verify_cert"`
ClientID string `json:"client_id"`
ClientSecret string `json:"client_secret"`
RedirectURL string `json:"redirect_url"`
Scope []string `json:"scope"`
}
// ConfigEntry ...

10
src/common/models/oidc_user.go
View File

@ -6,10 +6,14 @@ import (
// OIDCUser ...
type OIDCUser struct {
ID int64 `orm:"pk;auto;column(id)" json:"id"`
UserID int `orm:"column(user_id)" json:"user_id"`
Secret string `orm:"column(secret)" json:"secret"`
ID int64 `orm:"pk;auto;column(id)" json:"id"`
UserID int `orm:"column(user_id)" json:"user_id"`
// encrypted secret
Secret string `orm:"column(secret)" json:"-"`
// secret in plain text
PlainSecret string `orm:"-" json:"secret"`
SubIss string `orm:"column(subiss)" json:"subiss"`
Token string `orm:"column(token)" json:"-"`
CreationTime time.Time `orm:"column(creation_time);auto_now_add" json:"creation_time"`
UpdateTime time.Time `orm:"column(update_time);auto_now" json:"update_time"`
}

6
src/common/rbac/const.go
View File

@ -42,8 +42,10 @@ const (
ResourceLog = Resource("log")
ResourceMember = Resource("member")
ResourceMetadata = Resource("metadata")
ResourceReplication = Resource("replication")
ResourceReplicationJob = Resource("replication-job")
ResourceReplication = Resource("replication") // TODO remove
ResourceReplicationJob = Resource("replication-job") // TODO remove
ResourceReplicationExecution = Resource("replication-execution")
ResourceReplicationTask = Resource("replication-task")
ResourceRepository = Resource("repository")
ResourceRepositoryLabel = Resource("repository-label")
ResourceRepositoryTag = Resource("repository-tag")

12
src/common/rbac/project/util.go
View File

@ -75,6 +75,18 @@ var (
{Resource: rbac.ResourceReplicationJob, Action: rbac.ActionRead},
{Resource: rbac.ResourceReplicationJob, Action: rbac.ActionList},
{Resource: rbac.ResourceReplicationExecution, Action: rbac.ActionRead},
{Resource: rbac.ResourceReplicationExecution, Action: rbac.ActionList},
{Resource: rbac.ResourceReplicationExecution, Action: rbac.ActionCreate},
{Resource: rbac.ResourceReplicationExecution, Action: rbac.ActionUpdate},
{Resource: rbac.ResourceReplicationExecution, Action: rbac.ActionDelete},
{Resource: rbac.ResourceReplicationTask, Action: rbac.ActionRead},
{Resource: rbac.ResourceReplicationTask, Action: rbac.ActionList},
{Resource: rbac.ResourceReplicationTask, Action: rbac.ActionCreate},
{Resource: rbac.ResourceReplicationTask, Action: rbac.ActionUpdate},
{Resource: rbac.ResourceReplicationTask, Action: rbac.ActionDelete},
{Resource: rbac.ResourceLabel, Action: rbac.ActionCreate},
{Resource: rbac.ResourceLabel, Action: rbac.ActionRead},
{Resource: rbac.ResourceLabel, Action: rbac.ActionUpdate},

2
src/common/token/claims.go
View File

@ -1,9 +1,9 @@
package token
import (
"errors"
"github.com/dgrijalva/jwt-go"
"github.com/goharbor/harbor/src/common/rbac"
"github.com/pkg/errors"
)
// RobotClaims implements the interface of jwt.Claims

54
src/common/utils/oidc/helper.go
View File

@ -41,14 +41,14 @@ type providerHelper struct {
}
type endpoint struct {
url string
skipCertVerify bool
url string
VerifyCert bool
}
func (p *providerHelper) get() (*gooidc.Provider, error) {
if p.instance.Load() != nil {
s := p.setting.Load().(models.OIDCSetting)
if s.Endpoint != p.ep.url || s.SkipCertVerify != p.ep.skipCertVerify { // relevant settings have changed, need to re-create provider.
if s.Endpoint != p.ep.url || s.VerifyCert != p.ep.VerifyCert { // relevant settings have changed, need to re-create provider.
if err := p.create(); err != nil {
return nil, err
}
@ -90,24 +90,15 @@ func (p *providerHelper) create() error {
return errors.New("the configuration is not loaded")
}
s := p.setting.Load().(models.OIDCSetting)
var client *http.Client
if s.SkipCertVerify {
client = &http.Client{
Transport: insecureTransport,
}
} else {
client = &http.Client{}
}
ctx := context.Background()
gooidc.ClientContext(ctx, client)
ctx := clientCtx(context.Background(), s.VerifyCert)
provider, err := gooidc.NewProvider(ctx, s.Endpoint)
if err != nil {
return fmt.Errorf("failed to create OIDC provider, error: %v", err)
}
p.instance.Store(provider)
p.ep = endpoint{
url: s.Endpoint,
skipCertVerify: s.SkipCertVerify,
url: s.Endpoint,
VerifyCert: s.VerifyCert,
}
return nil
}
@ -170,6 +161,8 @@ func ExchangeToken(ctx context.Context, code string) (*Token, error) {
log.Errorf("Failed to get OAuth configuration, error: %v", err)
return nil, err
}
setting := provider.setting.Load().(models.OIDCSetting)
ctx = clientCtx(ctx, setting.VerifyCert)
oauthToken, err := oauth.Exchange(ctx, code)
if err != nil {
return nil, err
@ -184,5 +177,36 @@ func VerifyToken(ctx context.Context, rawIDToken string) (*gooidc.IDToken, error
return nil, err
}
verifier := p.Verifier(&gooidc.Config{ClientID: provider.setting.Load().(models.OIDCSetting).ClientID})
setting := provider.setting.Load().(models.OIDCSetting)
ctx = clientCtx(ctx, setting.VerifyCert)
return verifier.Verify(ctx, rawIDToken)
}
func clientCtx(ctx context.Context, verifyCert bool) context.Context {
var client *http.Client
if !verifyCert {
client = &http.Client{
Transport: insecureTransport,
}
} else {
client = &http.Client{}
}
return gooidc.ClientContext(ctx, client)
}
// RefreshToken refreshes the token passed in parameter, and return the new token.
func RefreshToken(ctx context.Context, token *Token) (*Token, error) {
oauth, err := getOauthConf()
if err != nil {
log.Errorf("Failed to get OAuth configuration, error: %v", err)
return nil, err
}
setting := provider.setting.Load().(models.OIDCSetting)
ctx = clientCtx(ctx, setting.VerifyCert)
ts := oauth.TokenSource(ctx, token.Token)
t, err := ts.Token()
if err != nil {
return nil, err
}
return &Token{Token: t, IDToken: t.Extra("id_token").(string)}, nil
}

32
src/common/utils/oidc/helper_test.go
View File

@ -16,6 +16,7 @@ package oidc
import (
"github.com/goharbor/harbor/src/common"
config2 "github.com/goharbor/harbor/src/common/config"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/core/config"
"github.com/stretchr/testify/assert"
@ -28,16 +29,17 @@ import (
func TestMain(m *testing.M) {
conf := map[string]interface{}{
common.OIDCName: "test",
common.OIDCEndpoint: "https://accounts.google.com",
common.OIDCSkipCertVerify: "false",
common.OIDCScope: "openid, profile, offline_access",
common.OIDCCLientID: "client",
common.OIDCClientSecret: "secret",
common.ExtEndpoint: "https://harbor.test",
common.OIDCName: "test",
common.OIDCEndpoint: "https://accounts.google.com",
common.OIDCVerifyCert: "true",
common.OIDCScope: "openid, profile, offline_access",
common.OIDCCLientID: "client",
common.OIDCClientSecret: "secret",
common.ExtEndpoint: "https://harbor.test",
}
kp := &config2.PresetKeyProvider{Key: "naa4JtarA1Zsc3uY"}
config.InitWithSettings(conf)
config.InitWithSettings(conf, kp)
result := m.Run()
if result != 0 {
@ -71,13 +73,13 @@ func TestHelperGet(t *testing.T) {
assert.Equal(t, "https://oauth2.googleapis.com/token", p.Endpoint().TokenURL)
update := map[string]interface{}{
common.OIDCName: "test",
common.OIDCEndpoint: "https://accounts.google.com",
common.OIDCSkipCertVerify: "false",
common.OIDCScope: "openid, profile, offline_access",
common.OIDCCLientID: "client",
common.OIDCClientSecret: "new-secret",
common.ExtEndpoint: "https://harbor.test",
common.OIDCName: "test",
common.OIDCEndpoint: "https://accounts.google.com",
common.OIDCVerifyCert: "true",
common.OIDCScope: "openid, profile, offline_access",
common.OIDCCLientID: "client",
common.OIDCClientSecret: "new-secret",
common.ExtEndpoint: "https://harbor.test",
}
config.GetCfgManager().UpdateConfig(update)

130
src/common/utils/oidc/secret.go Normal file
View File

@ -0,0 +1,130 @@
package oidc
import (
"context"
"encoding/json"
"fmt"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
"github.com/pkg/errors"
"sync"
)
// SecretVerifyError wraps the different errors happened when verifying a secret for OIDC user. When seeing this error,
// the caller should consider this an authentication error.
type SecretVerifyError struct {
cause error
}
func (se *SecretVerifyError) Error() string {
return fmt.Sprintf("failed to verify the secret: %v", se.cause)
}
func verifyError(err error) error {
return &SecretVerifyError{err}
}
// SecretManager is the interface for store and verify the secret
type SecretManager interface {
// SetSecret sets the secret and token based on the ID of the user, when setting the secret the user has to be
// onboarded to Harbor DB.
SetSecret(userID int, secret string, token *Token) error
// VerifySecret verifies the secret and the token associated with it, it refreshes the token in the DB if it's
// refreshed during the verification
VerifySecret(ctx context.Context, userID int, secret string) error
}
type defaultManager struct {
sync.Mutex
key string
}
var m SecretManager = &defaultManager{}
func (dm *defaultManager) getEncryptKey() (string, error) {
if dm.key == "" {
dm.Lock()
defer dm.Unlock()
if dm.key == "" {
key, err := config.SecretKey()
if err != nil {
return "", err
}
dm.key = key
}
}
return dm.key, nil
}
// SetSecret sets the secret and token based on the ID of the user, when setting the secret the user has to be
// onboarded to Harbor DB.
func (dm *defaultManager) SetSecret(userID int, secret string, token *Token) error {
key, err := dm.getEncryptKey()
if err != nil {
return fmt.Errorf("failed to load the key for encryption/decryption： %v", err)
}
oidcUser, err := dao.GetOIDCUserByUserID(userID)
if oidcUser == nil {
return fmt.Errorf("failed to get oidc user info, error: %v", err)
}
encSecret, _ := utils.ReversibleEncrypt(secret, key)
tb, _ := json.Marshal(token)
encToken, _ := utils.ReversibleEncrypt(string(tb), key)
oidcUser.Secret = encSecret
oidcUser.Token = encToken
return dao.UpdateOIDCUser(oidcUser)
}
// VerifySecret verifies the secret and the token associated with it, it tries to update the token in the DB if it's
// refreshed during the verification
func (dm *defaultManager) VerifySecret(ctx context.Context, userID int, secret string) error {
oidcUser, err := dao.GetOIDCUserByUserID(userID)
if err != nil {
return fmt.Errorf("failed to get oidc user info, error: %v", err)
}
if oidcUser == nil {
return fmt.Errorf("user is not onboarded as OIDC user")
}
key, err := dm.getEncryptKey()
if err != nil {
return fmt.Errorf("failed to load the key for encryption/decryption： %v", err)
}
plainSecret, err := utils.ReversibleDecrypt(oidcUser.Secret, key)
if err != nil {
return fmt.Errorf("failed to decrypt secret from DB: %v", err)
}
if secret != plainSecret {
return verifyError(errors.New("secret mismatch"))
}
tokenStr, err := utils.ReversibleDecrypt(oidcUser.Token, key)
if err != nil {
return verifyError(err)
}
token := &Token{}
err = json.Unmarshal(([]byte)(tokenStr), token)
if err != nil {
return verifyError(err)
}
_, err = VerifyToken(ctx, token.IDToken)
if err == nil {
return nil
}
log.Infof("Failed to verify ID Token, error: %v, refreshing...", err)
t, err := RefreshToken(ctx, token)
if err != nil {
return verifyError(err)
}
err = dm.SetSecret(oidcUser.UserID, secret, t)
if err != nil {
log.Warningf("Failed to update the token in DB: %v, ignore this error.", err)
}
return nil
}
// VerifySecret verifies the secret and the token associated with it, it tries to update the token in the DB if it's
// refreshed during the verification
func VerifySecret(ctx context.Context, userID int, secret string) error {
return m.VerifySecret(ctx, userID, secret)
}

32
src/common/utils/oidc/secret_test.go Normal file
View File

@ -0,0 +1,32 @@
package oidc
import (
"context"
"fmt"
"github.com/stretchr/testify/assert"
"testing"
)
func TestSecretVerifyError(t *testing.T) {
sve := &SecretVerifyError{cause: fmt.Errorf("myerror")}
assert.Equal(t, "failed to verify the secret: myerror", sve.Error())
err := verifyError(fmt.Errorf("myerror"))
assert.Equal(t, sve, err)
}
func TestDefaultManagerGetEncryptKey(t *testing.T) {
d := &defaultManager{}
k, err := d.getEncryptKey()
assert.Nil(t, err)
assert.Equal(t, "naa4JtarA1Zsc3uY", k)
d2 := &defaultManager{key: "oldkey"}
k2, err := d2.getEncryptKey()
assert.Nil(t, err)
assert.Equal(t, "oldkey", k2)
}
func TestPkgVerifySecret(t *testing.T) {
SetHardcodeVerifierForTest("secret")
assert.Nil(t, VerifySecret(context.Background(), 1, "secret"))
assert.NotNil(t, VerifySecret(context.Background(), 1, "not-the-secret"))
}

26
src/common/utils/oidc/testutils.go Normal file
View File

@ -0,0 +1,26 @@
package oidc
import "context"
import "errors"
// This is for testing only
type fakeVerifier struct {
secret string
}
func (fv *fakeVerifier) SetSecret(uid int, s string, t *Token) error {
return nil
}
func (fv *fakeVerifier) VerifySecret(ctx context.Context, userID int, secret string) error {
if secret != fv.secret {
return verifyError(errors.New("mismatch"))
}
return nil
}
// SetHardcodeVerifierForTest overwrite the default secret manager for testing.
// Be reminded this is for testing only.
func SetHardcodeVerifierForTest(s string) {
m = &fakeVerifier{s}
}

7
src/common/utils/registry/auth/credential.go
View File

@ -38,7 +38,12 @@ func NewBasicAuthCredential(username, password string) Credential {
}
func (b *basicAuthCredential) AddAuthorization(req *http.Request) {
req.SetBasicAuth(b.username, b.password)
// only add the authentication info when the username isn't empty
// the logic is needed for requesting resources from docker hub's
// public repositories
if len(b.username) > 0 {
req.SetBasicAuth(b.username, b.password)
}
}
// implement github.com/goharbor/harbor/src/common/http/modifier.Modifier

2
src/common/utils/registry/auth/tokenauthorizer.go
View File

@ -278,7 +278,7 @@ func NewStandardTokenAuthorizer(client *http.Client, credential Credential,
// 1. performance issue
// 2. the realm field returned by registry is an IP which can not reachable
// inside Harbor
if len(customizedTokenService) > 0 {
if len(customizedTokenService) > 0 && len(customizedTokenService[0]) > 0 {
generator.realm = customizedTokenService[0]
}

18
src/common/utils/registry/registry.go
View File

@ -157,3 +157,21 @@ func (r *Registry) Ping() error {
Message: string(b),
}
}
// PingSimple checks whether the registry is available. It checks the connectivity and certificate (if TLS enabled)
// only, regardless of credential.
func (r *Registry) PingSimple() error {
err := r.Ping()
if err == nil {
return nil
}
httpErr, ok := err.(*commonhttp.Error)
if !ok {
return err
}
if httpErr.Code == http.StatusUnauthorized ||
httpErr.Code == http.StatusForbidden {
return nil
}
return httpErr
}

45
src/common/utils/test/policy_manager.go
View File

@ -1,45 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package test
import (
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/models"
)
type FakePolicyManager struct {
}
func (f *FakePolicyManager) GetPolicies(query models.QueryParameter) (*models.ReplicationPolicyQueryResult, error) {
return &models.ReplicationPolicyQueryResult{}, nil
}
func (f *FakePolicyManager) GetPolicy(id int64) (models.ReplicationPolicy, error) {
return models.ReplicationPolicy{
ID: 1,
Trigger: &models.Trigger{
Kind: replication.TriggerKindManual,
},
}, nil
}
func (f *FakePolicyManager) CreatePolicy(policy models.ReplicationPolicy) (int64, error) {
return 1, nil
}
func (f *FakePolicyManager) UpdatePolicy(models.ReplicationPolicy) error {
return nil
}
func (f *FakePolicyManager) RemovePolicy(int64) error {
return nil
}

65
src/common/utils/test/watch_item.go
View File

@ -1,65 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package test
import (
"github.com/goharbor/harbor/src/common/models"
)
// FakeWatchItemDAO is the fake implement for the dao.WatchItemDAO
type FakeWatchItemDAO struct {
items []models.WatchItem
}
// Add ...
func (f *FakeWatchItemDAO) Add(item *models.WatchItem) (int64, error) {
f.items = append(f.items, *item)
return int64(len(f.items) + 1), nil
}
// DeleteByPolicyID : delete the WatchItem specified by policy ID
func (f *FakeWatchItemDAO) DeleteByPolicyID(policyID int64) error {
for i, item := range f.items {
if item.PolicyID == policyID {
f.items = append(f.items[:i], f.items[i+1:]...)
break
}
}
return nil
}
// Get returns WatchItem list according to the namespace and operation
func (f *FakeWatchItemDAO) Get(namespace, operation string) ([]models.WatchItem, error) {
items := []models.WatchItem{}
for _, item := range f.items {
if item.Namespace != namespace {
continue
}
if operation == "push" {
if item.OnPush {
items = append(items, item)
}
}
if operation == "delete" {
if item.OnDeletion {
items = append(items, item)
}
}
}
return items, nil
}

49
src/core/api/admin_job.go
View File

@ -27,6 +27,7 @@ import (
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/api/models"
utils_core "github.com/goharbor/harbor/src/core/utils"
"github.com/pkg/errors"
)
// AJAPI manages the CRUD of admin job and its schedule, any API wants to handle manual and cron job like ScanAll and GC cloud reuse it.
@ -42,7 +43,7 @@ func (aj *AJAPI) Prepare() {
// updateSchedule update a schedule of admin job.
func (aj *AJAPI) updateSchedule(ajr models.AdminJobReq) {
if ajr.Schedule.Type == models.ScheduleManual {
aj.HandleInternalServerError(fmt.Sprintf("Fail to update admin job schedule as wrong schedule type: %s.", ajr.Schedule.Type))
aj.SendInternalServerError((fmt.Errorf("fail to update admin job schedule as wrong schedule type: %s", ajr.Schedule.Type)))
return
}
@ -52,24 +53,24 @@ func (aj *AJAPI) updateSchedule(ajr models.AdminJobReq) {
}
jobs, err := dao.GetAdminJobs(query)
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("%v", err))
aj.SendInternalServerError(err)
return
}
if len(jobs) != 1 {
aj.HandleInternalServerError("Fail to update admin job schedule as we found more than one schedule in system, please ensure that only one schedule left for your job .")
aj.SendInternalServerError(errors.New("fail to update admin job schedule as we found more than one schedule in system, please ensure that only one schedule left for your job"))
return
}
// stop the scheduled job and remove it.
if err = utils_core.GetJobServiceClient().PostAction(jobs[0].UUID, common_job.JobActionStop); err != nil {
if e, ok := err.(*common_http.Error); !ok || e.Code != http.StatusNotFound {
aj.HandleInternalServerError(fmt.Sprintf("%v", err))
aj.SendInternalServerError(err)
return
}
}
if err = dao.DeleteAdminJob(jobs[0].ID); err != nil {
aj.HandleInternalServerError(fmt.Sprintf("%v", err))
aj.SendInternalServerError(err)
return
}
@ -85,17 +86,17 @@ func (aj *AJAPI) get(id int64) {
ID: id,
})
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("failed to get admin jobs: %v", err))
aj.SendInternalServerError(fmt.Errorf("failed to get admin jobs: %v", err))
return
}
if len(jobs) == 0 {
aj.HandleNotFound("No admin job found.")
aj.SendNotFoundError(errors.New("no admin job found"))
return
}
adminJobRep, err := convertToAdminJobRep(jobs[0])
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("failed to convert admin job response: %v", err))
aj.SendInternalServerError(fmt.Errorf("failed to convert admin job response: %v", err))
return
}
@ -107,7 +108,7 @@ func (aj *AJAPI) get(id int64) {
func (aj *AJAPI) list(name string) {
jobs, err := dao.GetTop10AdminJobsOfName(name)
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("failed to get admin jobs: %v", err))
aj.SendInternalServerError(fmt.Errorf("failed to get admin jobs: %v", err))
return
}
@ -115,7 +116,7 @@ func (aj *AJAPI) list(name string) {
for _, job := range jobs {
AdminJobRep, err := convertToAdminJobRep(job)
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("failed to convert admin job response: %v", err))
aj.SendInternalServerError(fmt.Errorf("failed to convert admin job response: %v", err))
return
}
AdminJobReps = append(AdminJobReps, &AdminJobRep)
@ -134,18 +135,18 @@ func (aj *AJAPI) getSchedule(name string) {
Kind: common_job.JobKindPeriodic,
})
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("failed to get admin jobs: %v", err))
aj.SendInternalServerError(fmt.Errorf("failed to get admin jobs: %v", err))
return
}
if len(jobs) > 1 {
aj.HandleInternalServerError("Get more than one scheduled admin job, make sure there has only one.")
aj.SendInternalServerError(errors.New("get more than one scheduled admin job, make sure there has only one"))
return
}
if len(jobs) != 0 {
adminJobRep, err := convertToAdminJobRep(jobs[0])
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("failed to convert admin job response: %v", err))
aj.SendInternalServerError(fmt.Errorf("failed to convert admin job response: %v", err))
return
}
adminJobSchedule.Schedule = adminJobRep.Schedule
@ -160,11 +161,13 @@ func (aj *AJAPI) getLog(id int64) {
job, err := dao.GetAdminJob(id)
if err != nil {
log.Errorf("Failed to load job data for job: %d, error: %v", id, err)
aj.CustomAbort(http.StatusInternalServerError, "Failed to get Job data")
aj.SendInternalServerError(errors.New("Failed to get Job data"))
return
}
if job == nil {
log.Errorf("Failed to get admin job: %d", id)
aj.CustomAbort(http.StatusNotFound, "Failed to get Job")
aj.SendNotFoundError(errors.New("Failed to get Job"))
return
}
logBytes, err := utils_core.GetJobServiceClient().GetJobLog(job.UUID)
@ -175,14 +178,14 @@ func (aj *AJAPI) getLog(id int64) {
id, httpErr.Code, httpErr.Message))
return
}
aj.HandleInternalServerError(fmt.Sprintf("Failed to get job logs, uuid: %s, error: %v", job.UUID, err))
aj.SendInternalServerError(fmt.Errorf("Failed to get job logs, uuid: %s, error: %v", job.UUID, err))
return
}
aj.Ctx.ResponseWriter.Header().Set(http.CanonicalHeaderKey("Content-Length"), strconv.Itoa(len(logBytes)))
aj.Ctx.ResponseWriter.Header().Set(http.CanonicalHeaderKey("Content-Type"), "text/plain")
_, err = aj.Ctx.ResponseWriter.Write(logBytes)
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("Failed to write job logs, uuid: %s, error: %v", job.UUID, err))
aj.SendInternalServerError(fmt.Errorf("Failed to write job logs, uuid: %s, error: %v", job.UUID, err))
}
}
@ -195,11 +198,11 @@ func (aj *AJAPI) submit(ajr *models.AdminJobReq) {
Kind: common_job.JobKindPeriodic,
})
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("failed to get admin jobs: %v", err))
aj.SendInternalServerError(fmt.Errorf("failed to get admin jobs: %v", err))
return
}
if len(jobs) != 0 {
aj.HandleStatusPreconditionFailed("Fail to set schedule for admin job as always had one, please delete it firstly then to re-schedule.")
aj.SendPreconditionFailedError(errors.New("fail to set schedule for admin job as always had one, please delete it firstly then to re-schedule"))
return
}
}
@ -210,7 +213,7 @@ func (aj *AJAPI) submit(ajr *models.AdminJobReq) {
Cron: ajr.CronString(),
})
if err != nil {
aj.HandleInternalServerError(fmt.Sprintf("%v", err))
aj.SendInternalServerError(err)
return
}
ajr.ID = id
@ -224,14 +227,14 @@ func (aj *AJAPI) submit(ajr *models.AdminJobReq) {
log.Debugf("Failed to delete admin job, err: %v", err)
}
if httpErr, ok := err.(*common_http.Error); ok && httpErr.Code == http.StatusConflict {
aj.HandleConflict(fmt.Sprintf("Conflict when triggering %s, please try again later.", ajr.Name))
aj.SendConflictError(fmt.Errorf("conflict when triggering %s, please try again later", ajr.Name))
return
}
aj.HandleInternalServerError(fmt.Sprintf("%v", err))
aj.SendInternalServerError(err)
return
}
if err := dao.SetAdminJobUUID(id, uuid); err != nil {
aj.HandleInternalServerError(fmt.Sprintf("%v", err))
aj.SendInternalServerError(err)
return
}
}

47
src/core/api/base.go
View File

@ -17,14 +17,14 @@ package api
import (
"net/http"
yaml "github.com/ghodss/yaml"
"errors"
"github.com/ghodss/yaml"
"github.com/goharbor/harbor/src/common/api"
"github.com/goharbor/harbor/src/common/security"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/filter"
"github.com/goharbor/harbor/src/core/promgr"
"github.com/goharbor/harbor/src/core/utils"
)
const (
@ -54,55 +54,20 @@ func (b *BaseController) Prepare() {
ctx, err := filter.GetSecurityContext(b.Ctx.Request)
if err != nil {
log.Errorf("failed to get security context: %v", err)
b.CustomAbort(http.StatusInternalServerError, "")
b.SendInternalServerError(errors.New(""))
return
}
b.SecurityCtx = ctx
pm, err := filter.GetProjectManager(b.Ctx.Request)
if err != nil {
log.Errorf("failed to get project manager: %v", err)
b.CustomAbort(http.StatusInternalServerError, "")
b.SendInternalServerError(errors.New(""))
return
}
b.ProjectMgr = pm
}
// RenderFormatedError renders errors with well formted style `{"error": "This is an error"}`
func (b *BaseController) RenderFormatedError(code int, err error) {
formatedErr := utils.WrapError(err)
log.Errorf("%s %s failed with error: %s", b.Ctx.Request.Method, b.Ctx.Request.URL.String(), formatedErr.Error())
b.RenderError(code, formatedErr.Error())
}
// SendUnAuthorizedError sends unauthorized error to the client.
func (b *BaseController) SendUnAuthorizedError(err error) {
b.RenderFormatedError(http.StatusUnauthorized, err)
}
// SendConflictError sends conflict error to the client.
func (b *BaseController) SendConflictError(err error) {
b.RenderFormatedError(http.StatusConflict, err)
}
// SendNotFoundError sends not found error to the client.
func (b *BaseController) SendNotFoundError(err error) {
b.RenderFormatedError(http.StatusNotFound, err)
}
// SendBadRequestError sends bad request error to the client.
func (b *BaseController) SendBadRequestError(err error) {
b.RenderFormatedError(http.StatusBadRequest, err)
}
// SendInternalServerError sends internal server error to the client.
func (b *BaseController) SendInternalServerError(err error) {
b.RenderFormatedError(http.StatusInternalServerError, err)
}
// SendForbiddenError sends forbidden error to the client.
func (b *BaseController) SendForbiddenError(err error) {
b.RenderFormatedError(http.StatusForbidden, err)
}
// WriteJSONData writes the JSON data to the client.
func (b *BaseController) WriteJSONData(object interface{}) {
b.Data["json"] = object

7
src/core/api/chart_label.go
View File

@ -61,7 +61,7 @@ func (cla *ChartLabelAPI) requireAccess(action rbac.Action) bool {
resource := rbac.NewProjectNamespace(cla.project.ProjectID).Resource(rbac.ResourceHelmChartVersionLabel)
if !cla.SecurityCtx.Can(action, resource) {
cla.HandleForbidden(cla.SecurityCtx.GetUsername())
cla.SendForbiddenError(errors.New(cla.SecurityCtx.GetUsername()))
return false
}
@ -75,7 +75,10 @@ func (cla *ChartLabelAPI) MarkLabel() {
}
l := &models.Label{}
cla.DecodeJSONReq(l)
if err := cla.DecodeJSONReq(l); err != nil {
cla.SendBadRequestError(err)
return
}
label, ok := cla.validate(l.ID, cla.project.ProjectID)
if !ok {

29
src/core/api/chart_repository.go
View File

@ -2,6 +2,7 @@ package api
import (
"bytes"
"context"
"errors"
"fmt"
"io"
@ -95,7 +96,7 @@ func (cra *ChartRepositoryAPI) requireAccess(action rbac.Action, subresource ...
if !cra.SecurityCtx.IsAuthenticated() {
cra.SendUnAuthorizedError(errors.New("Unauthorized"))
} else {
cra.HandleForbidden(cra.SecurityCtx.GetUsername())
cra.SendForbiddenError(errors.New(cra.SecurityCtx.GetUsername()))
}
return false
@ -113,7 +114,7 @@ func (cra *ChartRepositoryAPI) GetHealthStatus() {
}
if !cra.SecurityCtx.IsSysAdmin() {
cra.HandleForbidden(cra.SecurityCtx.GetUsername())
cra.SendForbiddenError(errors.New(cra.SecurityCtx.GetUsername()))
return
}
@ -141,7 +142,7 @@ func (cra *ChartRepositoryAPI) GetIndex() {
}
if !cra.SecurityCtx.IsSysAdmin() {
cra.HandleForbidden(cra.SecurityCtx.GetUsername())
cra.SendForbiddenError(errors.New(cra.SecurityCtx.GetUsername()))
return
}
@ -297,8 +298,28 @@ func (cra *ChartRepositoryAPI) UploadChartVersion() {
}
}
// set namespace/repository/version for replication event.
_, header, err := cra.GetFile(formFieldNameForChart)
if err != nil {
cra.SendInternalServerError(err)
return
}
req := cra.Ctx.Request
charFileName := header.Filename
if !strings.HasSuffix(charFileName, ".tgz") {
cra.SendInternalServerError(fmt.Errorf("chart file expected %s to end with .tgz", charFileName))
return
}
charFileName = strings.TrimSuffix(charFileName, ".tgz")
// colon cannot be used as namespace
charFileName = strings.Replace(charFileName, "-", ":", -1)
// value sample: library:redis:4.0.3 (namespace:repository:version)
ctx := context.WithValue(cra.Ctx.Request.Context(), common.ChartUploadCtxKey, cra.namespace+":"+charFileName)
req = req.WithContext(ctx)
// Directly proxy to the backend
chartController.ProxyTraffic(cra.Ctx.ResponseWriter, cra.Ctx.Request)
chartController.ProxyTraffic(cra.Ctx.ResponseWriter, req)
}
// UploadChartProvFile handles POST /api/:repo/prov

29
src/core/api/config.go
View File

@ -16,9 +16,9 @@ package api
import (
"fmt"
"net/http"
"strings"
"errors"
"github.com/goharbor/harbor/src/common"
"github.com/goharbor/harbor/src/common/config"
"github.com/goharbor/harbor/src/common/config/metadata"
@ -41,20 +41,20 @@ func (c *ConfigAPI) Prepare() {
c.BaseController.Prepare()
c.cfgManager = corecfg.GetCfgManager()
if !c.SecurityCtx.IsAuthenticated() {
c.HandleUnauthorized()
c.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
// Only internal container can access /api/internal/configurations
if strings.EqualFold(c.Ctx.Request.RequestURI, "/api/internal/configurations") {
if _, ok := c.Ctx.Request.Context().Value(filter.SecurCtxKey).(*secret.SecurityContext); !ok {
c.HandleUnauthorized()
c.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
}
if !c.SecurityCtx.IsSysAdmin() && !c.SecurityCtx.IsSolutionUser() {
c.HandleForbidden(c.SecurityCtx.GetUsername())
c.SendForbiddenError(errors.New(c.SecurityCtx.GetUsername()))
return
}
@ -71,7 +71,8 @@ func (c *ConfigAPI) Get() {
m, err := convertForGet(configs)
if err != nil {
log.Errorf("failed to convert configurations: %v", err)
c.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
c.SendInternalServerError(errors.New(""))
return
}
c.Data["json"] = m
@ -89,25 +90,33 @@ func (c *ConfigAPI) GetInternalConfig() {
// Put updates configurations
func (c *ConfigAPI) Put() {
m := map[string]interface{}{}
c.DecodeJSONReq(&m)
if err := c.DecodeJSONReq(&m); err != nil {
c.SendBadRequestError(err)
return
}
err := c.cfgManager.Load()
if err != nil {
log.Errorf("failed to get configurations: %v", err)
c.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
c.SendInternalServerError(errors.New(""))
return
}
isSysErr, err := c.validateCfg(m)
if err != nil {
if isSysErr {
log.Errorf("failed to validate configurations: %v", err)
c.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
c.SendInternalServerError(errors.New(""))
return
}
c.CustomAbort(http.StatusBadRequest, err.Error())
c.SendBadRequestError(err)
return
}
if err := c.cfgManager.UpdateConfig(m); err != nil {
log.Errorf("failed to upload configurations: %v", err)
c.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
c.SendInternalServerError(errors.New(""))
return
}
}

42
src/core/api/dataprepare_test.go
View File

@ -19,18 +19,18 @@ import (
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
rep_dao "github.com/goharbor/harbor/src/replication/dao"
rep_models "github.com/goharbor/harbor/src/replication/dao/models"
)
const (
// Prepare Test info
TestUserName = "testUser0001"
TestUserPwd = "testUser0001"
TestUserEmail = "testUser0001@mydomain.com"
TestProName = "testProject0001"
TestTargetName = "testTarget0001"
TestRepoName = "testRepo0001"
AdminName = "admin"
DefaultProjectName = "library"
TestUserName = "testUser0001"
TestUserPwd = "testUser0001"
TestUserEmail = "testUser0001@mydomain.com"
TestProName = "testProject0001"
TestRegistryName = "testRegistry0001"
TestRepoName = "testRepo0001"
)
func CommonAddUser() {
@ -83,25 +83,25 @@ func CommonDelProject() {
_ = dao.DeleteProject(commonProject.ProjectID)
}
func CommonAddTarget() {
func CommonAddRegistry() {
endPoint := os.Getenv("REGISTRY_URL")
commonTarget := &models.RepTarget{
URL: endPoint,
Name: TestTargetName,
Username: adminName,
Password: adminPwd,
commonRegistry := &rep_models.Registry{
URL: endPoint,
Name: TestRegistryName,
AccessKey: adminName,
AccessSecret: adminPwd,
}
_, _ = dao.AddRepTarget(*commonTarget)
_, _ = rep_dao.AddRegistry(commonRegistry)
}
func CommonGetTarget() int {
target, _ := dao.GetRepTargetByName(TestTargetName)
return int(target.ID)
func CommonGetRegistry() int {
registry, _ := rep_dao.GetRegistryByName(TestRegistryName)
return int(registry.ID)
}
func CommonDelTarget() {
target, _ := dao.GetRepTargetByName(TestTargetName)
_ = dao.DeleteRepTarget(target.ID)
func CommonDelRegistry() {
registry, _ := rep_dao.GetRegistryByName(TestRegistryName)
_ = rep_dao.DeleteRegistry(registry.ID)
}
func CommonAddRepository() {

24
src/core/api/email.go
View File

@ -15,8 +15,8 @@
package api
import (
"errors"
"net"
"net/http"
"strconv"
"github.com/goharbor/harbor/src/common/utils/email"
@ -37,12 +37,12 @@ type EmailAPI struct {
func (e *EmailAPI) Prepare() {
e.BaseController.Prepare()
if !e.SecurityCtx.IsAuthenticated() {
e.HandleUnauthorized()
e.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
if !e.SecurityCtx.IsSysAdmin() {
e.HandleForbidden(e.SecurityCtx.GetUsername())
e.SendForbiddenError(errors.New(e.SecurityCtx.GetUsername()))
return
}
}
@ -57,8 +57,8 @@ func (e *EmailAPI) Ping() {
cfg, err := config.Email()
if err != nil {
log.Errorf("failed to get email configurations: %v", err)
e.CustomAbort(http.StatusInternalServerError,
http.StatusText(http.StatusInternalServerError))
e.SendInternalServerError(err)
return
}
host = cfg.Host
port = cfg.Port
@ -77,18 +77,22 @@ func (e *EmailAPI) Ping() {
Identity string `json:"email_identity"`
Insecure bool `json:"email_insecure"`
}{}
e.DecodeJSONReq(&settings)
if err := e.DecodeJSONReq(&settings); err != nil {
e.SendBadRequestError(err)
return
}
if len(settings.Host) == 0 || settings.Port == nil {
e.CustomAbort(http.StatusBadRequest, "empty host or port")
e.SendBadRequestError(errors.New("empty host or port"))
return
}
if settings.Password == nil {
cfg, err := config.Email()
if err != nil {
log.Errorf("failed to get email configurations: %v", err)
e.CustomAbort(http.StatusInternalServerError,
http.StatusText(http.StatusInternalServerError))
e.SendInternalServerError(err)
return
}
settings.Password = &cfg.Password
@ -108,7 +112,7 @@ func (e *EmailAPI) Ping() {
password, pingEmailTimeout, ssl, insecure); err != nil {
log.Errorf("failed to ping email server: %v", err)
// do not return any detail information of the error, or may cause SSRF security issue #3755
e.RenderError(http.StatusBadRequest, "failed to ping email server")
e.SendBadRequestError(errors.New("failed to ping email server"))
return
}
}

213
src/core/api/harborapi_test.go
View File

@ -27,25 +27,20 @@ import (
"runtime"
"strconv"
"github.com/astaxie/beego"
"github.com/dghubble/sling"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/job/test"
"github.com/goharbor/harbor/src/common/models"
testutils "github.com/goharbor/harbor/src/common/utils/test"
api_models "github.com/goharbor/harbor/src/core/api/models"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/filter"
"github.com/goharbor/harbor/tests/apitests/apilib"
// "strconv"
// "strings"
"github.com/astaxie/beego"
"github.com/dghubble/sling"
"github.com/goharbor/harbor/src/common/dao"
apimodels "github.com/goharbor/harbor/src/core/api/models"
_ "github.com/goharbor/harbor/src/core/auth/db"
_ "github.com/goharbor/harbor/src/core/auth/ldap"
"github.com/goharbor/harbor/src/replication/core"
_ "github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/filter"
"github.com/goharbor/harbor/src/replication/model"
"github.com/goharbor/harbor/tests/apitests/apilib"
)
const (
@ -128,14 +123,9 @@ func init() {
beego.Router("/api/repositories/*/tags/:tag/manifest", &RepositoryAPI{}, "get:GetManifests")
beego.Router("/api/repositories/*/signatures", &RepositoryAPI{}, "get:GetSignatures")
beego.Router("/api/repositories/top", &RepositoryAPI{}, "get:GetTopRepos")
beego.Router("/api/targets/", &TargetAPI{}, "get:List")
beego.Router("/api/targets/", &TargetAPI{}, "post:Post")
beego.Router("/api/targets/:id([0-9]+)", &TargetAPI{})
beego.Router("/api/targets/:id([0-9]+)/policies/", &TargetAPI{}, "get:ListPolicies")
beego.Router("/api/targets/ping", &TargetAPI{}, "post:Ping")
beego.Router("/api/policies/replication/:id([0-9]+)", &RepPolicyAPI{})
beego.Router("/api/policies/replication", &RepPolicyAPI{}, "get:List")
beego.Router("/api/policies/replication", &RepPolicyAPI{}, "post:Post;delete:Delete")
beego.Router("/api/registries", &RegistryAPI{}, "get:List;post:Post")
beego.Router("/api/registries/ping", &RegistryAPI{}, "post:Ping")
beego.Router("/api/registries/:id([0-9]+)", &RegistryAPI{}, "get:Get;put:Put;delete:Delete")
beego.Router("/api/systeminfo", &SystemInfoAPI{}, "get:GetGeneralInfo")
beego.Router("/api/systeminfo/volumes", &SystemInfoAPI{}, "get:GetVolumeInfo")
beego.Router("/api/systeminfo/getcert", &SystemInfoAPI{}, "get:GetCert")
@ -146,7 +136,6 @@ func init() {
beego.Router("/api/configurations", &ConfigAPI{})
beego.Router("/api/configs", &ConfigAPI{}, "get:GetInternalConfig")
beego.Router("/api/email/ping", &EmailAPI{}, "post:Ping")
beego.Router("/api/replications", &ReplicationAPI{})
beego.Router("/api/labels", &LabelAPI{}, "post:Post;get:List")
beego.Router("/api/labels/:id([0-9]+", &LabelAPI{}, "get:Get;put:Put;delete:Delete")
beego.Router("/api/labels/:id([0-9]+)/resources", &LabelAPI{}, "get:ListResources")
@ -159,6 +148,15 @@ func init() {
beego.Router("/api/projects/:pid([0-9]+)/robots/", &RobotAPI{}, "post:Post;get:List")
beego.Router("/api/projects/:pid([0-9]+)/robots/:id([0-9]+)", &RobotAPI{}, "get:Get;put:Put;delete:Delete")
beego.Router("/api/replication/adapters", &ReplicationAdapterAPI{}, "get:List")
beego.Router("/api/replication/executions", &ReplicationOperationAPI{}, "get:ListExecutions;post:CreateExecution")
beego.Router("/api/replication/executions/:id([0-9]+)", &ReplicationOperationAPI{}, "get:GetExecution;put:StopExecution")
beego.Router("/api/replication/executions/:id([0-9]+)/tasks", &ReplicationOperationAPI{}, "get:ListTasks")
beego.Router("/api/replication/executions/:id([0-9]+)/tasks/:tid([0-9]+)/log", &ReplicationOperationAPI{}, "get:GetTaskLog")
beego.Router("/api/replication/policies", &ReplicationPolicyAPI{}, "get:List;post:Create")
beego.Router("/api/replication/policies/:id([0-9]+)", &ReplicationPolicyAPI{}, "get:Get;put:Update;delete:Delete")
// Charts are controlled under projects
chartRepositoryAPIType := &ChartRepositoryAPI{}
beego.Router("/api/chartrepo/health", chartRepositoryAPIType, "get:GetHealthStatus")
@ -180,10 +178,6 @@ func init() {
beego.Router("/api/chartrepo/:repo/charts/:name/:version/labels", chartLabelAPIType, "get:GetLabels;post:MarkLabel")
beego.Router("/api/chartrepo/:repo/charts/:name/:version/labels/:id([0-9]+)", chartLabelAPIType, "delete:RemoveLabel")
if err := core.Init(); err != nil {
log.Fatalf("failed to initialize GlobalController: %v", err)
}
// syncRegistry
if err := SyncRegistry(config.GlobalProjectMgr); err != nil {
log.Fatalf("failed to sync repositories from registry: %v", err)
@ -659,103 +653,6 @@ func (a testapi) GetReposTop(authInfo usrInfo, count string) (int, interface{},
return http.StatusOK, result, nil
}
// -------------------------Targets Test---------------------------------------//
// Create a new replication target
func (a testapi) AddTargets(authInfo usrInfo, repTarget apilib.RepTargetPost) (int, string, error) {
_sling := sling.New().Post(a.basePath)
path := "/api/targets"
_sling = _sling.Path(path)
_sling = _sling.BodyJSON(repTarget)
httpStatusCode, body, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, string(body), err
}
// List filters targets by name
func (a testapi) ListTargets(authInfo usrInfo, targetName string) (int, []apilib.RepTarget, error) {
_sling := sling.New().Get(a.basePath)
path := "/api/targets?name=" + targetName
_sling = _sling.Path(path)
var successPayload []apilib.RepTarget
httpStatusCode, body, err := request(_sling, jsonAcceptHeader, authInfo)
if err == nil && httpStatusCode == 200 {
err = json.Unmarshal(body, &successPayload)
}
return httpStatusCode, successPayload, err
}
// Ping target
func (a testapi) PingTarget(authInfo usrInfo, body interface{}) (int, error) {
_sling := sling.New().Post(a.basePath)
path := "/api/targets/ping"
_sling = _sling.Path(path)
_sling = _sling.BodyJSON(body)
httpStatusCode, _, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, err
}
// Get target by targetID
func (a testapi) GetTargetByID(authInfo usrInfo, targetID string) (int, error) {
_sling := sling.New().Get(a.basePath)
path := "/api/targets/" + targetID
_sling = _sling.Path(path)
httpStatusCode, _, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, err
}
// Update target by targetID
func (a testapi) PutTargetByID(authInfo usrInfo, targetID string, repTarget apilib.RepTargetPost) (int, error) {
_sling := sling.New().Put(a.basePath)
path := "/api/targets/" + targetID
_sling = _sling.Path(path)
_sling = _sling.BodyJSON(repTarget)
httpStatusCode, _, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, err
}
// List the target relevant policies by targetID
func (a testapi) GetTargetPoliciesByID(authInfo usrInfo, targetID string) (int, error) {
_sling := sling.New().Get(a.basePath)
path := "/api/targets/" + targetID + "/policies/"
_sling = _sling.Path(path)
httpStatusCode, _, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, err
}
// Delete target by targetID
func (a testapi) DeleteTargetsByID(authInfo usrInfo, targetID string) (int, error) {
_sling := sling.New().Delete(a.basePath)
path := "/api/targets/" + targetID
_sling = _sling.Path(path)
httpStatusCode, _, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, err
}
// --------------------Replication_Policy Test--------------------------------//
// Create a new replication policy
@ -1244,3 +1141,73 @@ func (a testapi) ScanAllScheduleGet(authInfo usrInfo) (int, api_models.AdminJobS
return httpStatusCode, successPayLoad, err
}
func (a testapi) RegistryGet(authInfo usrInfo, registryID int64) (*model.Registry, int, error) {
_sling := sling.New().Base(a.basePath).Get(fmt.Sprintf("/api/registries/%d", registryID))
code, body, err := request(_sling, jsonAcceptHeader, authInfo)
if err == nil && code == http.StatusOK {
registry := model.Registry{}
if err := json.Unmarshal(body, &registry); err != nil {
return nil, code, err
}
return &registry, code, nil
}
return nil, code, err
}
func (a testapi) RegistryList(authInfo usrInfo) ([]*model.Registry, int, error) {
_sling := sling.New().Base(a.basePath).Get("/api/registries")
code, body, err := request(_sling, jsonAcceptHeader, authInfo)
if err != nil || code != http.StatusOK {
return nil, code, err
}
var registries []*model.Registry
if err := json.Unmarshal(body, &registries); err != nil {
return nil, code, err
}
return registries, code, nil
}
func (a testapi) RegistryCreate(authInfo usrInfo, registry *model.Registry) (int, error) {
_sling := sling.New().Base(a.basePath).Post("/api/registries").BodyJSON(registry)
code, _, err := request(_sling, jsonAcceptHeader, authInfo)
return code, err
}
type pingReq struct {
ID *int64 `json:"id"`
Type *string `json:"type"`
URL *string `json:"url"`
CredentialType *string `json:"credential_type"`
AccessKey *string `json:"access_key"`
AccessSecret *string `json:"access_secret"`
Insecure *bool `json:"insecure"`
}
func (a testapi) RegistryPing(authInfo usrInfo, registry *pingReq) (int, error) {
_sling := sling.New().Base(a.basePath).Post("/api/registries/ping").BodyJSON(registry)
code, _, err := request(_sling, jsonAcceptHeader, authInfo)
return code, err
}
func (a testapi) RegistryDelete(authInfo usrInfo, registryID int64) (int, error) {
_sling := sling.New().Base(a.basePath).Delete(fmt.Sprintf("/api/registries/%d", registryID))
code, _, err := request(_sling, jsonAcceptHeader, authInfo)
if err != nil || code != http.StatusOK {
return code, fmt.Errorf("delete registry error: %v", err)
}
return code, nil
}
func (a testapi) RegistryUpdate(authInfo usrInfo, registryID int64, req *apimodels.RegistryUpdateRequest) (int, error) {
_sling := sling.New().Base(a.basePath).Put(fmt.Sprintf("/api/registries/%d", registryID)).BodyJSON(req)
code, _, err := request(_sling, jsonAcceptHeader, authInfo)
if err != nil || code != http.StatusOK {
return code, fmt.Errorf("update registry error: %v", err)
}
return code, nil
}

14
src/core/api/internal.go
View File

@ -15,7 +15,7 @@
package api
import (
"net/http"
"errors"
"github.com/goharbor/harbor/src/common"
"github.com/goharbor/harbor/src/common/dao"
@ -32,11 +32,11 @@ type InternalAPI struct {
func (ia *InternalAPI) Prepare() {
ia.BaseController.Prepare()
if !ia.SecurityCtx.IsAuthenticated() {
ia.HandleUnauthorized()
ia.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
if !ia.SecurityCtx.IsSysAdmin() {
ia.HandleForbidden(ia.SecurityCtx.GetUsername())
ia.SendForbiddenError(errors.New(ia.SecurityCtx.GetUsername()))
return
}
}
@ -45,7 +45,7 @@ func (ia *InternalAPI) Prepare() {
func (ia *InternalAPI) SyncRegistry() {
err := SyncRegistry(ia.ProjectMgr)
if err != nil {
ia.HandleInternalServerError(err.Error())
ia.SendInternalServerError(err)
return
}
}
@ -54,7 +54,8 @@ func (ia *InternalAPI) SyncRegistry() {
func (ia *InternalAPI) RenameAdmin() {
if !dao.IsSuperUser(ia.SecurityCtx.GetUsername()) {
log.Errorf("User %s is not super user, not allow to rename admin.", ia.SecurityCtx.GetUsername())
ia.CustomAbort(http.StatusForbidden, "")
ia.SendForbiddenError(errors.New(ia.SecurityCtx.GetUsername()))
return
}
newName := common.NewHarborAdminName
if err := dao.ChangeUserProfile(models.User{
@ -62,7 +63,8 @@ func (ia *InternalAPI) RenameAdmin() {
Username: newName,
}, "username"); err != nil {
log.Errorf("Failed to change admin's username, error: %v", err)
ia.CustomAbort(http.StatusInternalServerError, "Failed to rename admin user.")
ia.SendInternalServerError(errors.New("failed to rename admin user"))
return
}
log.Debugf("The super user has been renamed to: %s", newName)
ia.DestroySession()

122
src/core/api/label.go
View File

@ -15,6 +15,7 @@
package api
import (
"errors"
"fmt"
"net/http"
"strconv"
@ -23,9 +24,6 @@ import (
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/rbac"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/core"
rep_models "github.com/goharbor/harbor/src/replication/models"
)
// LabelAPI handles requests for label management
@ -44,25 +42,25 @@ func (l *LabelAPI) Prepare() {
// POST, PUT, DELETE need login first
if !l.SecurityCtx.IsAuthenticated() {
l.HandleUnauthorized()
l.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
if method == http.MethodPut || method == http.MethodDelete {
id, err := l.GetInt64FromPath(":id")
if err != nil || id <= 0 {
l.HandleBadRequest("invalid label ID")
l.SendBadRequestError(errors.New("invalid lable ID"))
return
}
label, err := dao.GetLabel(id)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to get label %d: %v", id, err))
l.SendInternalServerError(fmt.Errorf("failed to get label %d: %v", id, err))
return
}
if label == nil || label.Deleted {
l.HandleNotFound(fmt.Sprintf("label %d not found", id))
l.SendNotFoundError(fmt.Errorf("label %d not found", id))
return
}
@ -86,9 +84,9 @@ func (l *LabelAPI) requireAccess(label *models.Label, action rbac.Action, subres
if !hasPermission {
if !l.SecurityCtx.IsAuthenticated() {
l.HandleUnauthorized()
l.SendUnAuthorizedError(errors.New("UnAuthorized"))
} else {
l.HandleForbidden(l.SecurityCtx.GetUsername())
l.SendForbiddenError(errors.New(l.SecurityCtx.GetUsername()))
}
return false
}
@ -99,7 +97,12 @@ func (l *LabelAPI) requireAccess(label *models.Label, action rbac.Action, subres
// Post creates a label
func (l *LabelAPI) Post() {
label := &models.Label{}
l.DecodeJSONReqAndValidate(label)
isValid, err := l.DecodeJSONReqAndValidate(label)
if !isValid {
l.SendBadRequestError(err)
return
}
label.Level = common.LabelLevelUser
switch label.Scope {
@ -108,12 +111,12 @@ func (l *LabelAPI) Post() {
case common.LabelScopeProject:
exist, err := l.ProjectMgr.Exists(label.ProjectID)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to check the existence of project %d: %v",
l.SendInternalServerError(fmt.Errorf("failed to check the existence of project %d: %v",
label.ProjectID, err))
return
}
if !exist {
l.HandleNotFound(fmt.Sprintf("project %d not found", label.ProjectID))
l.SendNotFoundError(fmt.Errorf("project %d not found", label.ProjectID))
return
}
}
@ -129,17 +132,17 @@ func (l *LabelAPI) Post() {
ProjectID: label.ProjectID,
})
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to list labels: %v", err))
l.SendInternalServerError(fmt.Errorf("failed to list labels: %v", err))
return
}
if len(labels) > 0 {
l.HandleConflict()
l.SendConflictError(errors.New("conflict label"))
return
}
id, err := dao.AddLabel(label)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to create label: %v", err))
l.SendInternalServerError(fmt.Errorf("failed to create label: %v", err))
return
}
@ -150,18 +153,18 @@ func (l *LabelAPI) Post() {
func (l *LabelAPI) Get() {
id, err := l.GetInt64FromPath(":id")
if err != nil || id <= 0 {
l.HandleBadRequest(fmt.Sprintf("invalid label ID: %s", l.GetStringFromPath(":id")))
l.SendBadRequestError(fmt.Errorf("invalid label ID: %s", l.GetStringFromPath(":id")))
return
}
label, err := dao.GetLabel(id)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to get label %d: %v", id, err))
l.SendInternalServerError(fmt.Errorf("failed to get label %d: %v", id, err))
return
}
if label == nil || label.Deleted {
l.HandleNotFound(fmt.Sprintf("label %d not found", id))
l.SendNotFoundError(fmt.Errorf("label %d not found", id))
return
}
@ -183,7 +186,7 @@ func (l *LabelAPI) List() {
scope := l.GetString("scope")
if scope != common.LabelScopeGlobal && scope != common.LabelScopeProject {
l.HandleBadRequest(fmt.Sprintf("invalid scope: %s", scope))
l.SendBadRequestError(fmt.Errorf("invalid scope: %s", scope))
return
}
query.Scope = scope
@ -191,22 +194,22 @@ func (l *LabelAPI) List() {
if scope == common.LabelScopeProject {
projectIDStr := l.GetString("project_id")
if len(projectIDStr) == 0 {
l.HandleBadRequest("project_id is required")
l.SendBadRequestError(errors.New("project_id is required"))
return
}
projectID, err := strconv.ParseInt(projectIDStr, 10, 64)
if err != nil || projectID <= 0 {
l.HandleBadRequest(fmt.Sprintf("invalid project_id: %s", projectIDStr))
l.SendBadRequestError(fmt.Errorf("invalid project_id: %s", projectIDStr))
return
}
resource := rbac.NewProjectNamespace(projectID).Resource(rbac.ResourceLabel)
if !l.SecurityCtx.Can(rbac.ActionList, resource) {
if !l.SecurityCtx.IsAuthenticated() {
l.HandleUnauthorized()
l.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
l.HandleForbidden(l.SecurityCtx.GetUsername())
l.SendForbiddenError(errors.New(l.SecurityCtx.GetUsername()))
return
}
query.ProjectID = projectID
@ -214,15 +217,19 @@ func (l *LabelAPI) List() {
total, err := dao.GetTotalOfLabels(query)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to get total count of labels: %v", err))
l.SendInternalServerError(fmt.Errorf("failed to get total count of labels: %v", err))
return
}
query.Page, query.Size = l.GetPaginationParams()
query.Page, query.Size, err = l.GetPaginationParams()
if err != nil {
l.SendBadRequestError(err)
return
}
labels, err := dao.ListLabels(query)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to list labels: %v", err))
l.SendInternalServerError(fmt.Errorf("failed to list labels: %v", err))
return
}
@ -238,7 +245,10 @@ func (l *LabelAPI) Put() {
}
label := &models.Label{}
l.DecodeJSONReq(label)
if err := l.DecodeJSONReq(label); err != nil {
l.SendBadRequestError(err)
return
}
oldName := l.label.Name
@ -247,7 +257,13 @@ func (l *LabelAPI) Put() {
l.label.Description = label.Description
l.label.Color = label.Color
l.Validate(l.label)
isValidate, err := l.Validate(l.label)
if !isValidate {
if err != nil {
l.SendBadRequestError(err)
return
}
}
if l.label.Name != oldName {
labels, err := dao.ListLabels(&models.LabelQuery{
@ -257,17 +273,17 @@ func (l *LabelAPI) Put() {
ProjectID: l.label.ProjectID,
})
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to list labels: %v", err))
l.SendInternalServerError(fmt.Errorf("failed to list labels: %v", err))
return
}
if len(labels) > 0 {
l.HandleConflict()
l.SendConflictError(errors.New("conflict label"))
return
}
}
if err := dao.UpdateLabel(l.label); err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to update label %d: %v", l.label.ID, err))
l.SendInternalServerError(fmt.Errorf("failed to update label %d: %v", l.label.ID, err))
return
}
@ -281,11 +297,11 @@ func (l *LabelAPI) Delete() {
id := l.label.ID
if err := dao.DeleteResourceLabelByLabel(id); err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to delete resource label mappings of label %d: %v", id, err))
l.SendInternalServerError(fmt.Errorf("failed to delete resource label mappings of label %d: %v", id, err))
return
}
if err := dao.DeleteLabel(id); err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to delete label %d: %v", id, err))
l.SendInternalServerError(fmt.Errorf("failed to delete label %d: %v", id, err))
return
}
}
@ -294,18 +310,18 @@ func (l *LabelAPI) Delete() {
func (l *LabelAPI) ListResources() {
id, err := l.GetInt64FromPath(":id")
if err != nil || id <= 0 {
l.HandleBadRequest("invalid label ID")
l.SendBadRequestError(errors.New("invalid label ID"))
return
}
label, err := dao.GetLabel(id)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to get label %d: %v", id, err))
l.SendInternalServerError(fmt.Errorf("failed to get label %d: %v", id, err))
return
}
if label == nil || label.Deleted {
l.HandleNotFound(fmt.Sprintf("label %d not found", id))
l.SendNotFoundError(fmt.Errorf("label %d not found", id))
return
}
@ -313,26 +329,28 @@ func (l *LabelAPI) ListResources() {
return
}
result, err := core.GlobalController.GetPolicies(rep_models.QueryParameter{})
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to get policies: %v", err))
return
}
policies := []*rep_models.ReplicationPolicy{}
if result != nil {
for _, policy := range result.Policies {
for _, filter := range policy.Filters {
if filter.Kind != replication.FilterItemKindLabel {
continue
}
if filter.Value.(int64) == label.ID {
policies = append(policies, policy)
/*
result, err := core.GlobalController.GetPolicies(rep_models.QueryParameter{})
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("failed to get policies: %v", err))
return
}
policies := []*rep_models.ReplicationPolicy{}
if result != nil {
for _, policy := range result.Policies {
for _, filter := range policy.Filters {
if filter.Kind != replication.FilterItemKindLabel {
continue
}
if filter.Value.(int64) == label.ID {
policies = append(policies, policy)
}
}
}
}
}
*/
resources := map[string]interface{}{}
resources["replication_policies"] = policies
resources["replication_policies"] = nil
l.Data["json"] = resources
l.ServeJSON()
}

105
src/core/api/label_test.go
View File

@ -21,10 +21,7 @@ import (
"testing"
"github.com/goharbor/harbor/src/common"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/replication"
rep_models "github.com/goharbor/harbor/src/replication/models"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@ -436,105 +433,3 @@ func TestLabelAPIDelete(t *testing.T) {
runCodeCheckingCases(t, cases...)
}
func TestListResources(t *testing.T) {
// global level label
globalLabelID, err := dao.AddLabel(&models.Label{
Name: "globel_level_label",
Scope: common.LabelScopeGlobal,
})
require.Nil(t, err)
defer dao.DeleteLabel(globalLabelID)
// project level label
projectLabelID, err := dao.AddLabel(&models.Label{
Name: "project_level_label",
Scope: common.LabelScopeProject,
ProjectID: 1,
})
require.Nil(t, err)
defer dao.DeleteLabel(projectLabelID)
targetID, err := dao.AddRepTarget(models.RepTarget{
Name: "target_for_testing_label_resource",
URL: "https://192.168.0.1",
})
require.Nil(t, err)
defer dao.DeleteRepTarget(targetID)
// create a policy references both global and project labels
policyID, err := dao.AddRepPolicy(models.RepPolicy{
Name: "policy_for_testing_label_resource",
ProjectID: 1,
TargetID: targetID,
Trigger: fmt.Sprintf(`{"kind":"%s"}`, replication.TriggerKindManual),
Filters: fmt.Sprintf(`[{"kind":"%s","value":%d}, {"kind":"%s","value":%d}]`,
replication.FilterItemKindLabel, globalLabelID,
replication.FilterItemKindLabel, projectLabelID),
})
require.Nil(t, err)
defer dao.DeleteRepPolicy(policyID)
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodGet,
url: fmt.Sprintf("%s/%d/resources", labelAPIBasePath, globalLabelID),
},
code: http.StatusUnauthorized,
},
// 404
{
request: &testingRequest{
method: http.MethodGet,
url: fmt.Sprintf("%s/%d/resources", labelAPIBasePath, 10000),
credential: sysAdmin,
},
code: http.StatusNotFound,
},
// 403: global level label
{
request: &testingRequest{
method: http.MethodGet,
url: fmt.Sprintf("%s/%d/resources", labelAPIBasePath, globalLabelID),
credential: projAdmin,
},
code: http.StatusForbidden,
},
// 403: project level label
{
request: &testingRequest{
method: http.MethodGet,
url: fmt.Sprintf("%s/%d/resources", labelAPIBasePath, projectLabelID),
credential: projDeveloper,
},
code: http.StatusForbidden,
},
}
runCodeCheckingCases(t, cases...)
// 200: global level label
resources := map[string][]rep_models.ReplicationPolicy{}
err = handleAndParse(&testingRequest{
method: http.MethodGet,
url: fmt.Sprintf("%s/%d/resources", labelAPIBasePath, globalLabelID),
credential: sysAdmin,
}, &resources)
require.Nil(t, err)
policies := resources["replication_policies"]
require.Equal(t, 1, len(policies))
assert.Equal(t, policyID, policies[0].ID)
// 200: project level label
resources = map[string][]rep_models.ReplicationPolicy{}
err = handleAndParse(&testingRequest{
method: http.MethodGet,
url: fmt.Sprintf("%s/%d/resources", labelAPIBasePath, projectLabelID),
credential: projAdmin,
}, &resources)
require.Nil(t, err)
policies = resources["replication_policies"]
require.Equal(t, 1, len(policies))
assert.Equal(t, policyID, policies[0].ID)
}

39
src/core/api/ldap.go
View File

@ -22,6 +22,7 @@ import (
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/auth"
"errors"
goldap "gopkg.in/ldap.v2"
)
@ -43,17 +44,17 @@ const (
func (l *LdapAPI) Prepare() {
l.BaseController.Prepare()
if !l.SecurityCtx.IsAuthenticated() {
l.HandleUnauthorized()
l.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
if !l.SecurityCtx.IsSysAdmin() {
l.HandleForbidden(l.SecurityCtx.GetUsername())
l.SendForbiddenError(errors.New(l.SecurityCtx.GetUsername()))
return
}
ldapCfg, err := ldapUtils.LoadSystemLdapConfig()
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("Can't load system configuration, error: %v", err))
l.SendInternalServerError(fmt.Errorf("Can't load system configuration, error: %v", err))
return
}
l.ldapConfig = ldapCfg
@ -73,12 +74,16 @@ func (l *LdapAPI) Ping() {
ldapSession := *l.ldapConfig
err = ldapSession.ConnectionTest()
} else {
l.DecodeJSONReqAndValidate(&ldapConfs)
isValid, err := l.DecodeJSONReqAndValidate(&ldapConfs)
if !isValid {
l.SendBadRequestError(err)
return
}
err = ldapUtils.ConnectionTestWithConfig(ldapConfs)
}
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("LDAP connect fail, error: %v", err))
l.SendInternalServerError(fmt.Errorf("LDAP connect fail, error: %v", err))
return
}
}
@ -89,7 +94,7 @@ func (l *LdapAPI) Search() {
var ldapUsers []models.LdapUser
ldapSession := *l.ldapConfig
if err = ldapSession.Open(); err != nil {
l.HandleInternalServerError(fmt.Sprintf("Can't Open LDAP session, error: %v", err))
l.SendInternalServerError(fmt.Errorf("can't Open LDAP session, error: %v", err))
return
}
defer ldapSession.Close()
@ -99,7 +104,7 @@ func (l *LdapAPI) Search() {
ldapUsers, err = ldapSession.SearchUser(searchName)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("LDAP search fail, error: %v", err))
l.SendInternalServerError(fmt.Errorf("LDAP search fail, error: %v", err))
return
}
@ -113,18 +118,22 @@ func (l *LdapAPI) ImportUser() {
var ldapImportUsers models.LdapImportUser
var ldapFailedImportUsers []models.LdapFailedImportUser
l.DecodeJSONReqAndValidate(&ldapImportUsers)
isValid, err := l.DecodeJSONReqAndValidate(&ldapImportUsers)
if !isValid {
l.SendBadRequestError(err)
return
}
ldapFailedImportUsers, err := importUsers(ldapImportUsers.LdapUIDList, l.ldapConfig)
ldapFailedImportUsers, err = importUsers(ldapImportUsers.LdapUIDList, l.ldapConfig)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("LDAP import user fail, error: %v", err))
l.SendInternalServerError(fmt.Errorf("LDAP import user fail, error: %v", err))
return
}
if len(ldapFailedImportUsers) > 0 {
// Some user require json format response.
l.HandleNotFound("")
l.SendNotFoundError(errors.New("ldap user is not found"))
l.Data["json"] = ldapFailedImportUsers
l.ServeJSON()
return
@ -206,23 +215,23 @@ func (l *LdapAPI) SearchGroup() {
if len(searchName) > 0 {
ldapGroups, err = ldapSession.SearchGroupByName(searchName)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf("Can't search LDAP group by name, error: %v", err))
l.SendInternalServerError(fmt.Errorf("can't search LDAP group by name, error: %v", err))
return
}
} else if len(groupDN) > 0 {
if _, err := goldap.ParseDN(groupDN); err != nil {
l.HandleBadRequest(fmt.Sprintf("Invalid DN: %v", err))
l.SendBadRequestError(fmt.Errorf("invalid DN: %v", err))
return
}
ldapGroups, err = ldapSession.SearchGroupByDN(groupDN)
if err != nil {
// OpenLDAP usually return an error if DN is not found
l.HandleNotFound(fmt.Sprintf("Search LDAP group fail, error: %v", err))
l.SendNotFoundError(fmt.Errorf("search LDAP group fail, error: %v", err))
return
}
}
if len(ldapGroups) == 0 {
l.HandleNotFound("No ldap group found")
l.SendNotFoundError(errors.New("No ldap group found"))
return
}
l.Data["json"] = ldapGroups

19
src/core/api/log.go
View File

@ -17,6 +17,7 @@ package api
import (
"fmt"
"errors"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils"
@ -33,7 +34,7 @@ type LogAPI struct {
func (l *LogAPI) Prepare() {
l.BaseController.Prepare()
if !l.SecurityCtx.IsAuthenticated() {
l.HandleUnauthorized()
l.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
l.username = l.SecurityCtx.GetUsername()
@ -42,7 +43,11 @@ func (l *LogAPI) Prepare() {
// Get returns the recent logs according to parameters
func (l *LogAPI) Get() {
page, size := l.GetPaginationParams()
page, size, err := l.GetPaginationParams()
if err != nil {
l.SendBadRequestError(err)
return
}
query := &models.LogQueryParam{
Username: l.GetString("username"),
Repository: l.GetString("repository"),
@ -58,7 +63,7 @@ func (l *LogAPI) Get() {
if len(timestamp) > 0 {
t, err := utils.ParseTimeStamp(timestamp)
if err != nil {
l.HandleBadRequest(fmt.Sprintf("invalid begin_timestamp: %s", timestamp))
l.SendBadRequestError(fmt.Errorf("invalid begin_timestamp: %s", timestamp))
return
}
query.BeginTime = t
@ -68,7 +73,7 @@ func (l *LogAPI) Get() {
if len(timestamp) > 0 {
t, err := utils.ParseTimeStamp(timestamp)
if err != nil {
l.HandleBadRequest(fmt.Sprintf("invalid end_timestamp: %s", timestamp))
l.SendBadRequestError(fmt.Errorf("invalid end_timestamp: %s", timestamp))
return
}
query.EndTime = t
@ -77,7 +82,7 @@ func (l *LogAPI) Get() {
if !l.isSysAdmin {
projects, err := l.SecurityCtx.GetMyProjects()
if err != nil {
l.HandleInternalServerError(fmt.Sprintf(
l.SendInternalServerError(fmt.Errorf(
"failed to get projects of user %s: %v", l.username, err))
return
}
@ -98,14 +103,14 @@ func (l *LogAPI) Get() {
total, err := dao.GetTotalOfAccessLogs(query)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf(
l.SendInternalServerError(fmt.Errorf(
"failed to get total of access logs: %v", err))
return
}
logs, err := dao.GetAccessLogs(query)
if err != nil {
l.HandleInternalServerError(fmt.Sprintf(
l.SendInternalServerError(fmt.Errorf(
"failed to get access logs: %v", err))
return
}

43
src/core/api/metadata.go
View File

@ -21,6 +21,7 @@ import (
"strconv"
"strings"
"errors"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/rbac"
"github.com/goharbor/harbor/src/common/utils/log"
@ -56,7 +57,7 @@ func (m *MetadataAPI) Prepare() {
} else {
text += fmt.Sprintf("%d", id)
}
m.HandleBadRequest(text)
m.SendBadRequestError(errors.New(text))
return
}
@ -67,7 +68,7 @@ func (m *MetadataAPI) Prepare() {
}
if project == nil {
m.HandleNotFound(fmt.Sprintf("project %d not found", id))
m.SendNotFoundError(fmt.Errorf("project %d not found", id))
return
}
@ -78,11 +79,11 @@ func (m *MetadataAPI) Prepare() {
m.name = name
metas, err := m.metaMgr.Get(project.ProjectID, name)
if err != nil {
m.HandleInternalServerError(fmt.Sprintf("failed to get metadata of project %d: %v", project.ProjectID, err))
m.SendInternalServerError(fmt.Errorf("failed to get metadata of project %d: %v", project.ProjectID, err))
return
}
if len(metas) == 0 {
m.HandleNotFound(fmt.Sprintf("metadata %s of project %d not found", name, project.ProjectID))
m.SendNotFoundError(fmt.Errorf("metadata %s of project %d not found", name, project.ProjectID))
return
}
}
@ -93,9 +94,9 @@ func (m *MetadataAPI) requireAccess(action rbac.Action) bool {
if !m.SecurityCtx.Can(action, resource) {
if !m.SecurityCtx.IsAuthenticated() {
m.HandleUnauthorized()
m.SendUnAuthorizedError(errors.New("Unauthorized"))
} else {
m.HandleForbidden(m.SecurityCtx.GetUsername())
m.SendForbiddenError(errors.New(m.SecurityCtx.GetUsername()))
}
return false
}
@ -118,7 +119,7 @@ func (m *MetadataAPI) Get() {
}
if err != nil {
m.HandleInternalServerError(fmt.Sprintf("failed to get metadata %s of project %d: %v", m.name, m.project.ProjectID, err))
m.SendInternalServerError(fmt.Errorf("failed to get metadata %s of project %d: %v", m.name, m.project.ProjectID, err))
return
}
m.Data["json"] = metas
@ -132,33 +133,36 @@ func (m *MetadataAPI) Post() {
}
var metas map[string]string
m.DecodeJSONReq(&metas)
if err := m.DecodeJSONReq(&metas); err != nil {
m.SendBadRequestError(err)
return
}
ms, err := validateProjectMetadata(metas)
if err != nil {
m.HandleBadRequest(err.Error())
m.SendBadRequestError(err)
return
}
if len(ms) != 1 {
m.HandleBadRequest("invalid request: has no valid key/value pairs or has more than one valid key/value pairs")
m.SendBadRequestError(errors.New("invalid request: has no valid key/value pairs or has more than one valid key/value pairs"))
return
}
keys := reflect.ValueOf(ms).MapKeys()
mts, err := m.metaMgr.Get(m.project.ProjectID, keys[0].String())
if err != nil {
m.HandleInternalServerError(fmt.Sprintf("failed to get metadata for project %d: %v", m.project.ProjectID, err))
m.SendInternalServerError(fmt.Errorf("failed to get metadata for project %d: %v", m.project.ProjectID, err))
return
}
if len(mts) != 0 {
m.HandleConflict()
m.SendConflictError(errors.New("conflict metadata"))
return
}
if err := m.metaMgr.Add(m.project.ProjectID, ms); err != nil {
m.HandleInternalServerError(fmt.Sprintf("failed to create metadata for project %d: %v", m.project.ProjectID, err))
m.SendInternalServerError(fmt.Errorf("failed to create metadata for project %d: %v", m.project.ProjectID, err))
return
}
@ -172,11 +176,14 @@ func (m *MetadataAPI) Put() {
}
var metas map[string]string
m.DecodeJSONReq(&metas)
if err := m.DecodeJSONReq(&metas); err != nil {
m.SendBadRequestError(err)
return
}
meta, exist := metas[m.name]
if !exist {
m.HandleBadRequest(fmt.Sprintf("must contains key %s", m.name))
m.SendBadRequestError(fmt.Errorf("must contains key %s", m.name))
return
}
@ -184,14 +191,14 @@ func (m *MetadataAPI) Put() {
m.name: meta,
})
if err != nil {
m.HandleBadRequest(err.Error())
m.SendBadRequestError(err)
return
}
if err := m.metaMgr.Update(m.project.ProjectID, map[string]string{
m.name: ms[m.name],
}); err != nil {
m.HandleInternalServerError(fmt.Sprintf("failed to update metadata %s of project %d: %v", m.name, m.project.ProjectID, err))
m.SendInternalServerError(fmt.Errorf("failed to update metadata %s of project %d: %v", m.name, m.project.ProjectID, err))
return
}
}
@ -203,7 +210,7 @@ func (m *MetadataAPI) Delete() {
}
if err := m.metaMgr.Delete(m.project.ProjectID, m.name); err != nil {
m.HandleInternalServerError(fmt.Sprintf("failed to delete metadata %s of project %d: %v", m.name, m.project.ProjectID, err))
m.SendInternalServerError(fmt.Errorf("failed to delete metadata %s of project %d: %v", m.name, m.project.ProjectID, err))
return
}
}

15
src/core/api/models/execution.go Normal file
View File

@ -0,0 +1,15 @@
package models
import (
"time"
)
// Execution defines the data model used in API level
type Execution struct {
ID int64 `json:"id"`
Status string `json:"status"`
TriggerMode string `json:"trigger_mode"`
Duration int `json:"duration"`
SuccessRate string `json:"success_rate"`
StartTime time.Time `json:"start_time"`
}

12
src/core/api/models/registry.go Normal file
View File

@ -0,0 +1,12 @@
package models
// RegistryUpdateRequest is request used to update a registry.
type RegistryUpdateRequest struct {
Name *string `json:"name"`
Description *string `json:"description"`
URL *string `json:"url"`
CredentialType *string `json:"credential_type"`
AccessKey *string `json:"access_key"`
AccessSecret *string `json:"access_secret"`
Insecure *bool `json:"insecure"`
}

68
src/core/api/models/replication_policy.go
View File

@ -1,68 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package models
import (
"time"
"github.com/astaxie/beego/validation"
common_models "github.com/goharbor/harbor/src/common/models"
rep_models "github.com/goharbor/harbor/src/replication/models"
)
// ReplicationPolicy defines the data model used in API level
type ReplicationPolicy struct {
ID int64 `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
Filters []rep_models.Filter `json:"filters"`
ReplicateDeletion bool `json:"replicate_deletion"`
Trigger *rep_models.Trigger `json:"trigger"`
Projects []*common_models.Project `json:"projects"`
Targets []*common_models.RepTarget `json:"targets"`
CreationTime time.Time `json:"creation_time"`
UpdateTime time.Time `json:"update_time"`
ReplicateExistingImageNow bool `json:"replicate_existing_image_now"`
ErrorJobCount int64 `json:"error_job_count"`
}
// Valid ...
func (r *ReplicationPolicy) Valid(v *validation.Validation) {
if len(r.Name) == 0 {
v.SetError("name", "can not be empty")
}
if len(r.Name) > 256 {
v.SetError("name", "max length is 256")
}
if len(r.Projects) == 0 {
v.SetError("projects", "can not be empty")
}
if len(r.Targets) == 0 {
v.SetError("targets", "can not be empty")
}
for i := range r.Filters {
r.Filters[i].Valid(v)
}
if r.Trigger == nil {
v.SetError("trigger", "can not be empty")
} else {
r.Trigger.Valid(v)
}
}

112
src/core/api/project.go
View File

@ -28,6 +28,7 @@ import (
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
"errors"
"strconv"
"time"
)
@ -59,7 +60,7 @@ func (p *ProjectAPI) Prepare() {
} else {
text += fmt.Sprintf("%d", id)
}
p.HandleBadRequest(text)
p.SendBadRequestError(errors.New(text))
return
}
@ -70,7 +71,7 @@ func (p *ProjectAPI) Prepare() {
}
if project == nil {
p.HandleNotFound(fmt.Sprintf("project %d not found", id))
p.SendNotFoundError(fmt.Errorf("project %d not found", id))
return
}
@ -86,9 +87,10 @@ func (p *ProjectAPI) requireAccess(action rbac.Action, subresource ...rbac.Resou
if !p.SecurityCtx.Can(action, resource) {
if !p.SecurityCtx.IsAuthenticated() {
p.HandleUnauthorized()
p.SendUnAuthorizedError(errors.New("Unauthorized"))
} else {
p.HandleForbidden(p.SecurityCtx.GetUsername())
p.SendForbiddenError(errors.New(p.SecurityCtx.GetUsername()))
}
return false
@ -100,7 +102,7 @@ func (p *ProjectAPI) requireAccess(action rbac.Action, subresource ...rbac.Resou
// Post ...
func (p *ProjectAPI) Post() {
if !p.SecurityCtx.IsAuthenticated() {
p.HandleUnauthorized()
p.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
var onlyAdmin bool
@ -111,21 +113,25 @@ func (p *ProjectAPI) Post() {
onlyAdmin, err = config.OnlyAdminCreateProject()
if err != nil {
log.Errorf("failed to determine whether only admin can create projects: %v", err)
p.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
p.SendInternalServerError(fmt.Errorf("failed to determine whether only admin can create projects: %v", err))
return
}
}
if onlyAdmin && !p.SecurityCtx.IsSysAdmin() {
if onlyAdmin && !(p.SecurityCtx.IsSysAdmin() || p.SecurityCtx.IsSolutionUser()) {
log.Errorf("Only sys admin can create project")
p.RenderError(http.StatusForbidden, "Only system admin can create project")
p.SendForbiddenError(errors.New("Only system admin can create project"))
return
}
var pro *models.ProjectRequest
p.DecodeJSONReq(&pro)
if err := p.DecodeJSONReq(&pro); err != nil {
p.SendBadRequestError(err)
return
}
err = validateProjectReq(pro)
if err != nil {
log.Errorf("Invalid project request, error: %v", err)
p.RenderError(http.StatusBadRequest, fmt.Sprintf("invalid request: %v", err))
p.SendBadRequestError(fmt.Errorf("invalid request: %v", err))
return
}
@ -136,7 +142,7 @@ func (p *ProjectAPI) Post() {
return
}
if exist {
p.RenderError(http.StatusConflict, "")
p.SendConflictError(errors.New("conflict project"))
return
}
@ -153,15 +159,29 @@ func (p *ProjectAPI) Post() {
pro.Metadata[models.ProMetaPublic] = strconv.FormatBool(false)
}
owner := p.SecurityCtx.GetUsername()
// set the owner as the system admin when the API being called by replication
// it's a solution to workaround the restriction of project creation API:
// only normal users can create projects
if p.SecurityCtx.IsSolutionUser() {
user, err := dao.GetUser(models.User{
UserID: 1,
})
if err != nil {
p.SendInternalServerError(fmt.Errorf("failed to get the user 1: %v", err))
return
}
owner = user.Username
}
projectID, err := p.ProjectMgr.Create(&models.Project{
Name: pro.Name,
OwnerName: p.SecurityCtx.GetUsername(),
OwnerName: owner,
Metadata: pro.Metadata,
})
if err != nil {
if err == errutil.ErrDupProject {
log.Debugf("conflict %s", pro.Name)
p.RenderError(http.StatusConflict, "")
p.SendConflictError(fmt.Errorf("conflict %s", pro.Name))
} else {
p.ParseAndHandleError("failed to add project", err)
}
@ -189,7 +209,7 @@ func (p *ProjectAPI) Post() {
func (p *ProjectAPI) Head() {
name := p.GetString("project_name")
if len(name) == 0 {
p.HandleBadRequest("project_name is needed")
p.SendBadRequestError(errors.New("project_name is needed"))
return
}
@ -200,7 +220,7 @@ func (p *ProjectAPI) Head() {
}
if project == nil {
p.HandleNotFound(fmt.Sprintf("project %s not found", name))
p.SendNotFoundError(fmt.Errorf("project %s not found", name))
return
}
}
@ -225,12 +245,13 @@ func (p *ProjectAPI) Delete() {
result, err := p.deletable(p.project.ProjectID)
if err != nil {
p.HandleInternalServerError(fmt.Sprintf(
p.SendInternalServerError(fmt.Errorf(
"failed to check the deletable of project %d: %v", p.project.ProjectID, err))
return
}
if !result.Deletable {
p.CustomAbort(http.StatusPreconditionFailed, result.Message)
p.SendPreconditionFailedError(errors.New(result.Message))
return
}
if err = p.ProjectMgr.Delete(p.project.ProjectID); err != nil {
@ -260,7 +281,7 @@ func (p *ProjectAPI) Deletable() {
result, err := p.deletable(p.project.ProjectID)
if err != nil {
p.HandleInternalServerError(fmt.Sprintf(
p.SendInternalServerError(fmt.Errorf(
"failed to check the deletable of project %d: %v", p.project.ProjectID, err))
return
}
@ -284,18 +305,6 @@ func (p *ProjectAPI) deletable(projectID int64) (*deletableResp, error) {
}, nil
}
policies, err := dao.GetRepPolicyByProject(projectID)
if err != nil {
return nil, err
}
if len(policies) > 0 {
return &deletableResp{
Deletable: false,
Message: "the project contains replication rules, can not be deleted",
}, nil
}
// Check helm charts number
if config.WithChartMuseum() {
charts, err := chartController.ListCharts(p.project.Name)
@ -319,7 +328,11 @@ func (p *ProjectAPI) deletable(projectID int64) (*deletableResp, error) {
// List ...
func (p *ProjectAPI) List() {
// query strings
page, size := p.GetPaginationParams()
page, size, err := p.GetPaginationParams()
if err != nil {
p.SendBadRequestError(err)
return
}
query := &models.ProjectQueryParam{
Name: p.GetString("name"),
Owner: p.GetString("owner"),
@ -333,7 +346,7 @@ func (p *ProjectAPI) List() {
if len(public) > 0 {
pub, err := strconv.ParseBool(public)
if err != nil {
p.HandleBadRequest(fmt.Sprintf("invalid public: %s", public))
p.SendBadRequestError(fmt.Errorf("invalid public: %s", public))
return
}
query.Public = &pub
@ -346,7 +359,7 @@ func (p *ProjectAPI) List() {
// not login, only get public projects
pros, err := p.ProjectMgr.GetPublic()
if err != nil {
p.HandleInternalServerError(fmt.Sprintf("failed to get public projects: %v", err))
p.SendInternalServerError(fmt.Errorf("failed to get public projects: %v", err))
return
}
projects = []*models.Project{}
@ -358,13 +371,13 @@ func (p *ProjectAPI) List() {
// projects that the user is member of
pros, err := p.ProjectMgr.GetPublic()
if err != nil {
p.HandleInternalServerError(fmt.Sprintf("failed to get public projects: %v", err))
p.SendInternalServerError(fmt.Errorf("failed to get public projects: %v", err))
return
}
projects = append(projects, pros...)
mps, err := p.SecurityCtx.GetMyProjects()
if err != nil {
p.HandleInternalServerError(fmt.Sprintf("failed to list projects: %v", err))
p.SendInternalServerError(fmt.Errorf("failed to list projects: %v", err))
return
}
projects = append(projects, mps...)
@ -414,7 +427,8 @@ func (p *ProjectAPI) populateProperties(project *models.Project) {
})
if err != nil {
log.Errorf("failed to get total of repositories of project %d: %v", project.ProjectID, err)
p.CustomAbort(http.StatusInternalServerError, "")
p.SendInternalServerError(errors.New(""))
return
}
project.RepoCount = total
@ -424,7 +438,8 @@ func (p *ProjectAPI) populateProperties(project *models.Project) {
count, err := chartController.GetCountOfCharts([]string{project.Name})
if err != nil {
log.Errorf("Failed to get total of charts under project %s: %v", project.Name, err)
p.CustomAbort(http.StatusInternalServerError, "")
p.SendInternalServerError(errors.New(""))
return
}
project.ChartCount = count
@ -438,7 +453,10 @@ func (p *ProjectAPI) Put() {
}
var req *models.ProjectRequest
p.DecodeJSONReq(&req)
if err := p.DecodeJSONReq(&req); err != nil {
p.SendBadRequestError(err)
return
}
if err := p.ProjectMgr.Update(p.project.ProjectID,
&models.Project{
@ -456,7 +474,11 @@ func (p *ProjectAPI) Logs() {
return
}
page, size := p.GetPaginationParams()
page, size, err := p.GetPaginationParams()
if err != nil {
p.SendBadRequestError(err)
return
}
query := &models.LogQueryParam{
ProjectIDs: []int64{p.project.ProjectID},
Username: p.GetString("username"),
@ -473,7 +495,7 @@ func (p *ProjectAPI) Logs() {
if len(timestamp) > 0 {
t, err := utils.ParseTimeStamp(timestamp)
if err != nil {
p.HandleBadRequest(fmt.Sprintf("invalid begin_timestamp: %s", timestamp))
p.SendBadRequestError(fmt.Errorf("invalid begin_timestamp: %s", timestamp))
return
}
query.BeginTime = t
@ -483,7 +505,7 @@ func (p *ProjectAPI) Logs() {
if len(timestamp) > 0 {
t, err := utils.ParseTimeStamp(timestamp)
if err != nil {
p.HandleBadRequest(fmt.Sprintf("invalid end_timestamp: %s", timestamp))
p.SendBadRequestError(fmt.Errorf("invalid end_timestamp: %s", timestamp))
return
}
query.EndTime = t
@ -491,14 +513,14 @@ func (p *ProjectAPI) Logs() {
total, err := dao.GetTotalOfAccessLogs(query)
if err != nil {
p.HandleInternalServerError(fmt.Sprintf(
p.SendInternalServerError(fmt.Errorf(
"failed to get total of access log: %v", err))
return
}
logs, err := dao.GetAccessLogs(query)
if err != nil {
p.HandleInternalServerError(fmt.Sprintf(
p.SendInternalServerError(fmt.Errorf(
"failed to get access log: %v", err))
return
}
@ -511,8 +533,8 @@ func (p *ProjectAPI) Logs() {
// TODO move this to pa ckage models
func validateProjectReq(req *models.ProjectRequest) error {
pn := req.Name
if utils.IsIllegalLength(req.Name, projectNameMinLen, projectNameMaxLen) {
return fmt.Errorf("Project name is illegal in length. (greater than %d or less than %d)", projectNameMaxLen, projectNameMinLen)
if utils.IsIllegalLength(pn, projectNameMinLen, projectNameMaxLen) {
return fmt.Errorf("Project name %s is illegal in length. (greater than %d or less than %d)", pn, projectNameMaxLen, projectNameMinLen)
}
validProjectName := regexp.MustCompile(`^` + restrictedNameChars + `$`)
legal := validProjectName.MatchString(pn)

46
src/core/api/projectmember.go
View File

@ -50,7 +50,7 @@ func (pma *ProjectMemberAPI) Prepare() {
pma.BaseController.Prepare()
if !pma.SecurityCtx.IsAuthenticated() {
pma.HandleUnauthorized()
pma.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
pid, err := pma.GetInt64FromPath(":pid")
@ -61,7 +61,7 @@ func (pma *ProjectMemberAPI) Prepare() {
} else {
text += fmt.Sprintf("%d", pid)
}
pma.HandleBadRequest(text)
pma.SendBadRequestError(errors.New(text))
return
}
project, err := pma.ProjectMgr.Get(pid)
@ -70,7 +70,7 @@ func (pma *ProjectMemberAPI) Prepare() {
return
}
if project == nil {
pma.HandleNotFound(fmt.Sprintf("project %d not found", pid))
pma.SendNotFoundError(fmt.Errorf("project %d not found", pid))
return
}
pma.project = project
@ -80,7 +80,7 @@ func (pma *ProjectMemberAPI) Prepare() {
log.Warningf("Failed to get pmid from path, error %v", err)
}
if pmid <= 0 && (pma.Ctx.Input.IsPut() || pma.Ctx.Input.IsDelete()) {
pma.HandleBadRequest(fmt.Sprintf("The project member id is invalid, pmid:%s", pma.GetStringFromPath(":pmid")))
pma.SendBadRequestError(fmt.Errorf("The project member id is invalid, pmid:%s", pma.GetStringFromPath(":pmid")))
return
}
pma.id = int(pmid)
@ -91,9 +91,9 @@ func (pma *ProjectMemberAPI) requireAccess(action rbac.Action) bool {
if !pma.SecurityCtx.Can(action, resource) {
if !pma.SecurityCtx.IsAuthenticated() {
pma.HandleUnauthorized()
pma.SendUnAuthorizedError(errors.New("Unauthorized"))
} else {
pma.HandleForbidden(pma.SecurityCtx.GetUsername())
pma.SendForbiddenError(errors.New(pma.SecurityCtx.GetUsername()))
}
return false
@ -115,7 +115,7 @@ func (pma *ProjectMemberAPI) Get() {
entityname := pma.GetString("entityname")
memberList, err := project.SearchMemberByName(projectID, entityname)
if err != nil {
pma.HandleInternalServerError(fmt.Sprintf("Failed to query database for member list, error: %v", err))
pma.SendInternalServerError(fmt.Errorf("Failed to query database for member list, error: %v", err))
return
}
if len(memberList) > 0 {
@ -127,11 +127,11 @@ func (pma *ProjectMemberAPI) Get() {
queryMember.ID = pma.id
memberList, err := project.GetProjectMember(queryMember)
if err != nil {
pma.HandleInternalServerError(fmt.Sprintf("Failed to query database for member list, error: %v", err))
pma.SendInternalServerError(fmt.Errorf("Failed to query database for member list, error: %v", err))
return
}
if len(memberList) == 0 {
pma.HandleNotFound(fmt.Sprintf("The project member does not exit, pmid:%v", pma.id))
pma.SendNotFoundError(fmt.Errorf("The project member does not exit, pmid:%v", pma.id))
return
}
@ -150,27 +150,30 @@ func (pma *ProjectMemberAPI) Post() {
}
projectID := pma.project.ProjectID
var request models.MemberReq
pma.DecodeJSONReq(&request)
if err := pma.DecodeJSONReq(&request); err != nil {
pma.SendBadRequestError(err)
return
}
request.MemberGroup.LdapGroupDN = strings.TrimSpace(request.MemberGroup.LdapGroupDN)
pmid, err := AddProjectMember(projectID, request)
if err == auth.ErrorGroupNotExist || err == auth.ErrorUserNotExist {
pma.HandleNotFound(fmt.Sprintf("Failed to add project member, error: %v", err))
pma.SendNotFoundError(fmt.Errorf("Failed to add project member, error: %v", err))
return
} else if err == auth.ErrDuplicateLDAPGroup {
pma.HandleConflict(fmt.Sprintf("Failed to add project member, already exist LDAP group or project member, groupDN:%v", request.MemberGroup.LdapGroupDN))
pma.SendConflictError(fmt.Errorf("Failed to add project member, already exist LDAP group or project member, groupDN:%v", request.MemberGroup.LdapGroupDN))
return
} else if err == ErrDuplicateProjectMember {
pma.HandleConflict(fmt.Sprintf("Failed to add project member, already exist LDAP group or project member, groupMemberID:%v", request.MemberGroup.ID))
pma.SendConflictError(fmt.Errorf("Failed to add project member, already exist LDAP group or project member, groupMemberID:%v", request.MemberGroup.ID))
return
} else if err == ErrInvalidRole {
pma.HandleBadRequest(fmt.Sprintf("Invalid role ID, role ID %v", request.Role))
pma.SendBadRequestError(fmt.Errorf("Invalid role ID, role ID %v", request.Role))
return
} else if err == auth.ErrInvalidLDAPGroupDN {
pma.HandleBadRequest(fmt.Sprintf("Invalid LDAP DN: %v", request.MemberGroup.LdapGroupDN))
pma.SendBadRequestError(fmt.Errorf("Invalid LDAP DN: %v", request.MemberGroup.LdapGroupDN))
return
} else if err != nil {
pma.HandleInternalServerError(fmt.Sprintf("Failed to add project member, error: %v", err))
pma.SendInternalServerError(fmt.Errorf("Failed to add project member, error: %v", err))
return
}
pma.Redirect(http.StatusCreated, strconv.FormatInt(int64(pmid), 10))
@ -184,14 +187,17 @@ func (pma *ProjectMemberAPI) Put() {
pid := pma.project.ProjectID
pmID := pma.id
var req models.Member
pma.DecodeJSONReq(&req)
if err := pma.DecodeJSONReq(&req); err != nil {
pma.SendBadRequestError(err)
return
}
if req.Role < 1 || req.Role > 4 {
pma.HandleBadRequest(fmt.Sprintf("Invalid role id %v", req.Role))
pma.SendBadRequestError(fmt.Errorf("Invalid role id %v", req.Role))
return
}
err := project.UpdateProjectMemberRole(pmID, req.Role)
if err != nil {
pma.HandleInternalServerError(fmt.Sprintf("Failed to update DB to add project user role, project id: %d, pmid : %d, role id: %d", pid, pmID, req.Role))
pma.SendInternalServerError(fmt.Errorf("Failed to update DB to add project user role, project id: %d, pmid : %d, role id: %d", pid, pmID, req.Role))
return
}
}
@ -204,7 +210,7 @@ func (pma *ProjectMemberAPI) Delete() {
pmid := pma.id
err := project.DeleteProjectMemberByID(pmid)
if err != nil {
pma.HandleInternalServerError(fmt.Sprintf("Failed to delete project roles for user, project member id: %d, error: %v", pmid, err))
pma.SendInternalServerError(fmt.Errorf("Failed to delete project roles for user, project member id: %d, error: %v", pmid, err))
return
}
}

22
src/core/api/reg_gc.go
View File

@ -15,7 +15,7 @@
package api
import (
"fmt"
"errors"
"net/http"
"os"
"strconv"
@ -33,11 +33,11 @@ type GCAPI struct {
func (gc *GCAPI) Prepare() {
gc.BaseController.Prepare()
if !gc.SecurityCtx.IsAuthenticated() {
gc.HandleUnauthorized()
gc.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
if !gc.SecurityCtx.IsSysAdmin() {
gc.HandleForbidden(gc.SecurityCtx.GetUsername())
gc.SendForbiddenError(errors.New(gc.SecurityCtx.GetUsername()))
return
}
}
@ -58,7 +58,11 @@ func (gc *GCAPI) Prepare() {
// }
func (gc *GCAPI) Post() {
ajr := models.AdminJobReq{}
gc.DecodeJSONReqAndValidate(&ajr)
isValid, err := gc.DecodeJSONReqAndValidate(&ajr)
if !isValid {
gc.SendBadRequestError(err)
return
}
ajr.Name = common_job.ImageGC
ajr.Parameters = map[string]interface{}{
"redis_url_reg": os.Getenv("_REDIS_URL_REG"),
@ -77,7 +81,11 @@ func (gc *GCAPI) Post() {
// }
func (gc *GCAPI) Put() {
ajr := models.AdminJobReq{}
gc.DecodeJSONReqAndValidate(&ajr)
isValid, err := gc.DecodeJSONReqAndValidate(&ajr)
if !isValid {
gc.SendBadRequestError(err)
return
}
ajr.Name = common_job.ImageGC
gc.updateSchedule(ajr)
}
@ -86,7 +94,7 @@ func (gc *GCAPI) Put() {
func (gc *GCAPI) GetGC() {
id, err := gc.GetInt64FromPath(":id")
if err != nil {
gc.HandleInternalServerError(fmt.Sprintf("need to specify gc id"))
gc.SendInternalServerError(errors.New("need to specify gc id"))
return
}
gc.get(id)
@ -106,7 +114,7 @@ func (gc *GCAPI) Get() {
func (gc *GCAPI) GetLog() {
id, err := gc.GetInt64FromPath(":id")
if err != nil {
gc.HandleBadRequest("invalid ID")
gc.SendBadRequestError(errors.New("invalid ID"))
return
}
gc.getLog(id)

496
src/core/api/registry.go Normal file
View File

@ -0,0 +1,496 @@
package api
import (
"errors"
"fmt"
"net/http"
"strconv"
common_http "github.com/goharbor/harbor/src/common/http"
"github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/api/models"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/adapter"
"github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/replication/model"
"github.com/goharbor/harbor/src/replication/policy"
"github.com/goharbor/harbor/src/replication/registry"
)
// RegistryAPI handles requests to /api/registries/{}. It manages registries integrated to Harbor.
type RegistryAPI struct {
BaseController
manager registry.Manager
policyCtl policy.Controller
}
// Prepare validates the user
func (t *RegistryAPI) Prepare() {
t.BaseController.Prepare()
if !t.SecurityCtx.IsAuthenticated() {
t.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
if !t.SecurityCtx.IsSysAdmin() {
t.SendForbiddenError(errors.New(t.SecurityCtx.GetUsername()))
return
}
t.manager = replication.RegistryMgr
t.policyCtl = replication.PolicyCtl
}
// Ping checks health status of a registry
func (t *RegistryAPI) Ping() {
req := struct {
ID *int64 `json:"id"`
Type *string `json:"type"`
URL *string `json:"url"`
CredentialType *string `json:"credential_type"`
AccessKey *string `json:"access_key"`
AccessSecret *string `json:"access_secret"`
Insecure *bool `json:"insecure"`
}{}
t.DecodeJSONReq(&req)
reg := &model.Registry{}
var err error
if req.ID != nil {
reg, err = t.manager.Get(*req.ID)
if err != nil {
t.SendInternalServerError(fmt.Errorf("failed to get registry %d: %v", *req.ID, err))
return
}
if reg == nil {
t.SendNotFoundError(fmt.Errorf("registry %d not found", *req.ID))
return
}
}
if req.Type != nil {
reg.Type = model.RegistryType(*req.Type)
}
if req.URL != nil {
url, err := utils.ParseEndpoint(*req.URL)
if err != nil {
t.SendBadRequestError(err)
return
}
// Prevent SSRF security issue #3755
reg.URL = url.Scheme + "://" + url.Host + url.Path
}
if req.CredentialType != nil {
if reg.Credential == nil {
reg.Credential = &model.Credential{}
}
reg.Credential.Type = model.CredentialType(*req.CredentialType)
}
if req.AccessKey != nil {
if reg.Credential == nil {
reg.Credential = &model.Credential{}
}
reg.Credential.AccessKey = *req.AccessKey
}
if req.AccessSecret != nil {
if reg.Credential == nil {
reg.Credential = &model.Credential{}
}
reg.Credential.AccessSecret = *req.AccessSecret
}
if req.Insecure != nil {
reg.Insecure = *req.Insecure
}
if len(reg.Type) == 0 || len(reg.URL) == 0 {
t.SendBadRequestError(errors.New("type or url cannot be empty"))
return
}
status, err := registry.CheckHealthStatus(reg)
if err != nil {
e, ok := err.(*common_http.Error)
if ok && e.Code == http.StatusUnauthorized {
t.SendBadRequestError(errors.New("invalid credential"))
return
}
t.SendInternalServerError(fmt.Errorf("failed to check health of registry %s: %v", reg.URL, err))
return
}
if status != model.Healthy {
t.SendBadRequestError(errors.New(""))
return
}
return
}
// Get gets a registry by id.
func (t *RegistryAPI) Get() {
id, err := t.GetIDFromURL()
if err != nil {
t.SendBadRequestError(err)
return
}
r, err := t.manager.Get(id)
if err != nil {
log.Errorf("failed to get registry %d: %v", id, err)
t.SendInternalServerError(err)
return
}
if r == nil {
t.SendNotFoundError(fmt.Errorf("registry %d not found", id))
return
}
// Hide access secret
if r.Credential != nil && len(r.Credential.AccessSecret) != 0 {
r.Credential.AccessSecret = "*****"
}
t.Data["json"] = r
t.ServeJSON()
}
// List lists all registries that match a given registry name.
func (t *RegistryAPI) List() {
name := t.GetString("name")
_, registries, err := t.manager.List(&model.RegistryQuery{
Name: name,
})
if err != nil {
log.Errorf("failed to list registries %s: %v", name, err)
t.SendInternalServerError(err)
return
}
// Hide passwords
for _, r := range registries {
if r.Credential != nil && len(r.Credential.AccessSecret) != 0 {
r.Credential.AccessSecret = "*****"
}
}
t.Data["json"] = registries
t.ServeJSON()
return
}
// Post creates a registry
func (t *RegistryAPI) Post() {
r := &model.Registry{}
isValid, err := t.DecodeJSONReqAndValidate(r)
if !isValid {
t.SendBadRequestError(err)
return
}
reg, err := t.manager.GetByName(r.Name)
if err != nil {
log.Errorf("failed to get registry %s: %v", r.Name, err)
t.SendInternalServerError(err)
return
}
if reg != nil {
t.SendConflictError(fmt.Errorf("name '%s' is already used", r.Name))
return
}
status, err := registry.CheckHealthStatus(r)
if err != nil {
t.SendBadRequestError(fmt.Errorf("health check to registry %s failed: %v", r.URL, err))
return
}
if status != model.Healthy {
t.SendBadRequestError(fmt.Errorf("registry %s is unhealthy: %s", r.URL, status))
return
}
id, err := t.manager.Add(r)
if err != nil {
log.Errorf("Add registry '%s' error: %v", r.URL, err)
t.SendInternalServerError(err)
return
}
t.Redirect(http.StatusCreated, strconv.FormatInt(id, 10))
}
// Put updates a registry
func (t *RegistryAPI) Put() {
id, err := t.GetIDFromURL()
if err != nil {
t.SendBadRequestError(err)
return
}
r, err := t.manager.Get(id)
if err != nil {
log.Errorf("Get registry by id %d error: %v", id, err)
t.SendInternalServerError(err)
return
}
if r == nil {
t.SendNotFoundError(fmt.Errorf("Registry %d not found", id))
return
}
req := models.RegistryUpdateRequest{}
if err := t.DecodeJSONReq(&req); err != nil {
t.SendBadRequestError(err)
return
}
originalName := r.Name
if req.Name != nil {
r.Name = *req.Name
}
if req.Description != nil {
r.Description = *req.Description
}
if req.URL != nil {
r.URL = *req.URL
}
if req.CredentialType != nil {
r.Credential.Type = (model.CredentialType)(*req.CredentialType)
}
if req.AccessKey != nil {
r.Credential.AccessKey = *req.AccessKey
}
if req.AccessSecret != nil {
r.Credential.AccessSecret = *req.AccessSecret
}
if req.Insecure != nil {
r.Insecure = *req.Insecure
}
t.Validate(r)
if r.Name != originalName {
reg, err := t.manager.GetByName(r.Name)
if err != nil {
log.Errorf("Get registry by name '%s' error: %v", r.Name, err)
t.SendInternalServerError(err)
return
}
if reg != nil {
t.SendConflictError(errors.New("name is already used"))
return
}
}
status, err := registry.CheckHealthStatus(r)
if err != nil {
t.SendBadRequestError(fmt.Errorf("health check to registry %s failed: %v", r.URL, err))
return
}
if status != model.Healthy {
t.SendBadRequestError(fmt.Errorf("registry %s is unhealthy: %s", r.URL, status))
return
}
if err := t.manager.Update(r); err != nil {
log.Errorf("Update registry %d error: %v", id, err)
t.SendInternalServerError(err)
return
}
}
// Delete deletes a registry
func (t *RegistryAPI) Delete() {
id, err := t.GetIDFromURL()
if err != nil {
t.SendBadRequestError(err)
return
}
registry, err := t.manager.Get(id)
if err != nil {
msg := fmt.Sprintf("Get registry %d error: %v", id, err)
log.Error(msg)
t.SendInternalServerError(errors.New(msg))
return
}
if registry == nil {
t.SendNotFoundError(fmt.Errorf("Registry %d not found", id))
return
}
// Check whether there are replication policies that use this registry as source registry.
total, _, err := t.policyCtl.List([]*model.PolicyQuery{
{
SrcRegistry: id,
},
}...)
if err != nil {
t.SendInternalServerError(fmt.Errorf("List replication policies with source registry %d error: %v", id, err))
return
}
if total > 0 {
msg := fmt.Sprintf("Can't delete registry %d, %d replication policies use it as source registry", id, total)
log.Error(msg)
t.SendPreconditionFailedError(errors.New(msg))
return
}
// Check whether there are replication policies that use this registry as destination registry.
total, _, err = t.policyCtl.List([]*model.PolicyQuery{
{
DestRegistry: id,
},
}...)
if err != nil {
t.SendInternalServerError(fmt.Errorf("List replication policies with destination registry %d error: %v", id, err))
return
}
if total > 0 {
msg := fmt.Sprintf("Can't delete registry %d, %d replication policies use it as destination registry", id, total)
log.Error(msg)
t.SendPreconditionFailedError(errors.New(msg))
return
}
if err := t.manager.Remove(id); err != nil {
msg := fmt.Sprintf("Delete registry %d error: %v", id, err)
log.Error(msg)
t.SendPreconditionFailedError(errors.New(msg))
return
}
}
// GetInfo returns the base info and capability declarations of the registry
func (t *RegistryAPI) GetInfo() {
id, err := t.GetInt64FromPath(":id")
// "0" is used for the ID of the local Harbor registry
if err != nil || id < 0 {
t.SendBadRequestError(fmt.Errorf("invalid registry ID %s", t.GetString(":id")))
return
}
var registry *model.Registry
if id == 0 {
registry = event.GetLocalRegistry()
} else {
registry, err = t.manager.Get(id)
if err != nil {
t.SendInternalServerError(fmt.Errorf("failed to get registry %d: %v", id, err))
return
}
if registry == nil {
t.SendNotFoundError(fmt.Errorf("registry %d not found", id))
return
}
}
factory, err := adapter.GetFactory(registry.Type)
if err != nil {
t.SendInternalServerError(fmt.Errorf("failed to get the adapter factory for registry type %s: %v", registry.Type, err))
return
}
adp, err := factory(registry)
if err != nil {
t.SendInternalServerError(fmt.Errorf("failed to create the adapter for registry %d: %v", registry.ID, err))
return
}
info, err := adp.Info()
if err != nil {
t.SendInternalServerError(fmt.Errorf("failed to get registry info %d: %v", id, err))
return
}
t.WriteJSONData(process(info))
}
// GetNamespace get the namespace of a registry
// TODO remove
func (t *RegistryAPI) GetNamespace() {
/*
var registry *model.Registry
var err error
id, err := t.GetInt64FromPath(":id")
if err != nil || id < 0 {
t.HandleBadRequest(fmt.Sprintf("invalid registry ID %s", t.GetString(":id")))
return
}
if id > 0 {
registry, err = t.manager.Get(id)
if err != nil {
t.HandleInternalServerError(fmt.Sprintf("failed to get registry %d: %v", id, err))
return
}
} else if id == 0 {
registry = event.GetLocalRegistry()
}
if registry == nil {
t.HandleNotFound(fmt.Sprintf("registry %d not found", id))
return
}
if !adapter.HasFactory(registry.Type) {
t.HandleInternalServerError(fmt.Sprintf("no adapter factory found for %s", registry.Type))
return
}
regFactory, err := adapter.GetFactory(registry.Type)
if err != nil {
t.HandleInternalServerError(fmt.Sprintf("fail to get adapter factory %s", registry.Type))
return
}
regAdapter, err := regFactory(registry)
if err != nil {
t.HandleInternalServerError(fmt.Sprintf("fail to get adapter %s", registry.Type))
return
}
query := &model.NamespaceQuery{
Name: t.GetString("name"),
}
npResults, err := regAdapter.ListNamespaces(query)
if err != nil {
t.HandleInternalServerError(fmt.Sprintf("fail to list namespaces %s %v", registry.Type, err))
return
}
t.Data["json"] = npResults
t.ServeJSON()
*/
}
// merge "SupportedResourceTypes" into "SupportedResourceFilters" for UI to render easier
func process(info *model.RegistryInfo) *model.RegistryInfo {
if info == nil {
return nil
}
in := &model.RegistryInfo{
Type: info.Type,
Description: info.Description,
SupportedTriggers: info.SupportedTriggers,
}
filters := []*model.FilterStyle{}
for _, filter := range info.SupportedResourceFilters {
if filter.Type != model.FilterTypeResource {
filters = append(filters, filter)
}
}
values := []string{}
for _, resourceType := range info.SupportedResourceTypes {
values = append(values, string(resourceType))
}
filters = append(filters, &model.FilterStyle{
Type: model.FilterTypeResource,
Style: model.FilterStyleTypeRadio,
Values: values,
})
in.SupportedResourceFilters = filters
return in
}

184
src/core/api/registry_test.go Normal file
View File

@ -0,0 +1,184 @@
package api
import (
"net/http"
"testing"
"github.com/goharbor/harbor/src/core/api/models"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/dao"
"github.com/goharbor/harbor/src/replication/model"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
)
var (
testRegistry = &model.Registry{
Name: "test1",
URL: "https://registry-1.docker.io",
Type: "harbor",
Credential: nil,
}
testRegistry2 = &model.Registry{
Name: "test2",
URL: "https://registry-1.docker.io",
Type: "harbor",
Credential: nil,
}
)
type RegistrySuite struct {
suite.Suite
testAPI *testapi
defaultRegistry model.Registry
}
func (suite *RegistrySuite) SetupSuite() {
assert := assert.New(suite.T())
assert.Nil(replication.Init(make(chan struct{})))
suite.testAPI = newHarborAPI()
code, err := suite.testAPI.RegistryCreate(*admin, testRegistry)
assert.Nil(err)
assert.Equal(http.StatusCreated, code)
tmp, err := dao.GetRegistryByName(testRegistry.Name)
assert.Nil(err)
assert.NotNil(tmp)
suite.defaultRegistry = *testRegistry
suite.defaultRegistry.ID = tmp.ID
CommonAddUser()
}
func (suite *RegistrySuite) TearDownSuite() {
assert := assert.New(suite.T())
code, err := suite.testAPI.RegistryDelete(*admin, suite.defaultRegistry.ID)
assert.Nil(err)
assert.Equal(http.StatusOK, code)
CommonDelUser()
}
func (suite *RegistrySuite) TestGet() {
assert := assert.New(suite.T())
// Get a non-existed registry
_, code, _ := suite.testAPI.RegistryGet(*admin, 0)
assert.Equal(http.StatusBadRequest, code)
// Get as admin, should succeed
retrieved, code, err := suite.testAPI.RegistryGet(*admin, suite.defaultRegistry.ID)
assert.Nil(err)
assert.NotNil(retrieved)
assert.Equal(http.StatusOK, code)
assert.Equal("test1", retrieved.Name)
// Get as user, should fail
_, code, _ = suite.testAPI.RegistryGet(*testUser, suite.defaultRegistry.ID)
assert.Equal(http.StatusForbidden, code)
}
func (suite *RegistrySuite) TestList() {
assert := assert.New(suite.T())
// List as admin, should succeed
registries, code, err := suite.testAPI.RegistryList(*admin)
assert.Nil(err)
assert.Equal(http.StatusOK, code)
assert.Equal(1, len(registries))
// List as user, should fail
registries, code, err = suite.testAPI.RegistryList(*testUser)
assert.Equal(http.StatusForbidden, code)
assert.Equal(0, len(registries))
}
func (suite *RegistrySuite) TestPost() {
assert := assert.New(suite.T())
// Should conflict when create exited registry
code, err := suite.testAPI.RegistryCreate(*admin, testRegistry)
assert.Nil(err)
assert.Equal(http.StatusConflict, code)
// Create as user, should fail
code, err = suite.testAPI.RegistryCreate(*testUser, testRegistry2)
assert.Nil(err)
assert.Equal(http.StatusForbidden, code)
}
func (suite *RegistrySuite) TestPing() {
assert := assert.New(suite.T())
code, err := suite.testAPI.RegistryPing(*admin, &pingReq{
ID: &suite.defaultRegistry.ID,
})
assert.Nil(err)
assert.Equal(http.StatusOK, code)
var id int64 = -1
code, err = suite.testAPI.RegistryPing(*admin, &pingReq{
ID: &id,
})
assert.Nil(err)
assert.Equal(http.StatusNotFound, code)
code, err = suite.testAPI.RegistryPing(*admin, nil)
assert.Nil(err)
assert.Equal(http.StatusBadRequest, code)
code, err = suite.testAPI.RegistryPing(*testUser, &pingReq{
ID: &suite.defaultRegistry.ID,
})
assert.Nil(err)
assert.Equal(http.StatusForbidden, code)
}
func (suite *RegistrySuite) TestRegistryPut() {
assert := assert.New(suite.T())
// Update as admin, should succeed
description := "foobar"
updateReq := &models.RegistryUpdateRequest{
Description: &description,
}
code, err := suite.testAPI.RegistryUpdate(*admin, suite.defaultRegistry.ID, updateReq)
assert.Nil(err)
assert.Equal(http.StatusOK, code)
updated, code, err := suite.testAPI.RegistryGet(*admin, suite.defaultRegistry.ID)
assert.Nil(err)
assert.Equal(http.StatusOK, code)
assert.Equal("foobar", updated.Description)
// Update as user, should fail
code, err = suite.testAPI.RegistryUpdate(*testUser, suite.defaultRegistry.ID, updateReq)
assert.NotNil(err)
assert.Equal(http.StatusForbidden, code)
}
func (suite *RegistrySuite) TestDelete() {
assert := assert.New(suite.T())
code, err := suite.testAPI.RegistryCreate(*admin, testRegistry2)
assert.Nil(err)
assert.Equal(http.StatusCreated, code)
tmp, err := dao.GetRegistryByName(testRegistry2.Name)
assert.Nil(err)
assert.NotNil(tmp)
// Delete as user, should fail
code, err = suite.testAPI.RegistryDelete(*testUser, tmp.ID)
assert.NotNil(err)
assert.Equal(http.StatusForbidden, code)
// Delete as admin, should succeed
code, err = suite.testAPI.RegistryDelete(*admin, tmp.ID)
assert.Nil(err)
assert.Equal(http.StatusOK, code)
}
func TestRegistrySuite(t *testing.T) {
suite.Run(t, new(RegistrySuite))
}

107
src/core/api/replication.go
View File

@ -1,107 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"fmt"
"net/http"
"strings"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils/log"
api_models "github.com/goharbor/harbor/src/core/api/models"
"github.com/goharbor/harbor/src/core/notifier"
"github.com/goharbor/harbor/src/replication/core"
"github.com/goharbor/harbor/src/replication/event/notification"
"github.com/goharbor/harbor/src/replication/event/topic"
"github.com/docker/distribution/uuid"
)
// ReplicationAPI handles API calls for replication
type ReplicationAPI struct {
BaseController
}
// Prepare does authentication and authorization works
func (r *ReplicationAPI) Prepare() {
r.BaseController.Prepare()
if !r.SecurityCtx.IsAuthenticated() {
r.HandleUnauthorized()
return
}
if !r.SecurityCtx.IsSysAdmin() && !r.SecurityCtx.IsSolutionUser() {
r.HandleForbidden(r.SecurityCtx.GetUsername())
return
}
}
// Post trigger a replication according to the specified policy
func (r *ReplicationAPI) Post() {
replication := &api_models.Replication{}
r.DecodeJSONReqAndValidate(replication)
policy, err := core.GlobalController.GetPolicy(replication.PolicyID)
if err != nil {
r.HandleInternalServerError(fmt.Sprintf("failed to get replication policy %d: %v", replication.PolicyID, err))
return
}
if policy.ID == 0 {
r.HandleNotFound(fmt.Sprintf("replication policy %d not found", replication.PolicyID))
return
}
count, err := dao.GetTotalCountOfRepJobs(&models.RepJobQuery{
PolicyID: replication.PolicyID,
Statuses: []string{models.JobPending, models.JobRunning},
Operations: []string{models.RepOpTransfer, models.RepOpDelete},
})
if err != nil {
r.HandleInternalServerError(fmt.Sprintf("failed to filter jobs of policy %d: %v",
replication.PolicyID, err))
return
}
if count > 0 {
r.RenderError(http.StatusPreconditionFailed, "policy has running/pending jobs, new replication can not be triggered")
return
}
opUUID, err := startReplication(replication.PolicyID)
if err != nil {
r.HandleInternalServerError(fmt.Sprintf("failed to publish replication topic for policy %d: %v", replication.PolicyID, err))
return
}
log.Infof("replication signal for policy %d sent", replication.PolicyID)
r.Data["json"] = api_models.ReplicationResponse{
UUID: opUUID,
}
r.ServeJSON()
}
// startReplication triggers a replication and return the uuid of this replication.
func startReplication(policyID int64) (string, error) {
opUUID := strings.Replace(uuid.Generate().String(), "-", "", -1)
return opUUID, notifier.Publish(topic.StartReplicationTopic,
notification.StartReplicationNotification{
PolicyID: policyID,
Metadata: map[string]interface{}{
"op_uuid": opUUID,
},
})
}

46
src/core/api/replication_adapter.go Normal file
View File

@ -0,0 +1,46 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"errors"
"github.com/goharbor/harbor/src/replication/adapter"
"github.com/goharbor/harbor/src/replication/model"
)
// ReplicationAdapterAPI handles the replication adapter requests
type ReplicationAdapterAPI struct {
BaseController
}
// Prepare ...
func (r *ReplicationAdapterAPI) Prepare() {
r.BaseController.Prepare()
if !r.SecurityCtx.IsSysAdmin() {
if !r.SecurityCtx.IsAuthenticated() {
r.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
r.SendForbiddenError(errors.New(r.SecurityCtx.GetUsername()))
return
}
}
// List the replication adapters
func (r *ReplicationAdapterAPI) List() {
types := []model.RegistryType{}
types = append(types, adapter.ListRegisteredAdapterTypes()...)
r.WriteJSONData(types)
}

63
src/core/api/replication_adapter_test.go Normal file
View File

@ -0,0 +1,63 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"net/http"
"testing"
"github.com/goharbor/harbor/src/replication/adapter"
"github.com/goharbor/harbor/src/replication/model"
"github.com/stretchr/testify/require"
)
func fakedFactory(*model.Registry) (adapter.Adapter, error) {
return nil, nil
}
func TestReplicationAdapterAPIList(t *testing.T) {
err := adapter.RegisterFactory("test", fakedFactory)
require.Nil(t, err)
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/adapters",
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/adapters",
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 200
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/adapters",
credential: sysAdmin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}

277
src/core/api/replication_execution.go Normal file
View File

@ -0,0 +1,277 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"errors"
"fmt"
"net/http"
"strconv"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/dao/models"
"github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/replication/model"
)
// ReplicationOperationAPI handles the replication operation requests
type ReplicationOperationAPI struct {
BaseController
}
// Prepare ...
func (r *ReplicationOperationAPI) Prepare() {
r.BaseController.Prepare()
// As we delegate the jobservice to trigger the scheduled replication,
// we need to allow the jobservice to call the API
if !(r.SecurityCtx.IsSysAdmin() || r.SecurityCtx.IsSolutionUser()) {
if !r.SecurityCtx.IsAuthenticated() {
r.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
r.SendForbiddenError(errors.New(r.SecurityCtx.GetUsername()))
return
}
}
// The API is open only for system admin currently, we can use
// the code commentted below to make the API available to the
// users who have permission for all projects that the policy
// refers
/*
func (r *ReplicationOperationAPI) authorized(policy *model.Policy, resource rbac.Resource, action rbac.Action) bool {
projects := []string{}
// pull mode
if policy.SrcRegistryID != 0 {
projects = append(projects, policy.DestNamespace)
} else {
// push mode
projects = append(projects, policy.SrcNamespaces...)
}
for _, project := range projects {
resource := rbac.NewProjectNamespace(project).Resource(resource)
if !r.SecurityCtx.Can(action, resource) {
r.HandleForbidden(r.SecurityCtx.GetUsername())
return false
}
}
return true
}
*/
// ListExecutions ...
func (r *ReplicationOperationAPI) ListExecutions() {
query := &models.ExecutionQuery{
Trigger: r.GetString("trigger"),
}
if len(r.GetString("status")) > 0 {
query.Statuses = []string{r.GetString("status")}
}
if len(r.GetString("policy_id")) > 0 {
policyID, err := r.GetInt64("policy_id")
if err != nil || policyID <= 0 {
r.SendBadRequestError(fmt.Errorf("invalid policy_id %s", r.GetString("policy_id")))
return
}
query.PolicyID = policyID
}
page, size, err := r.GetPaginationParams()
if err != nil {
r.SendBadRequestError(err)
return
}
query.Page = page
query.Size = size
total, executions, err := replication.OperationCtl.ListExecutions(query)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to list executions: %v", err))
return
}
r.SetPaginationHeader(total, query.Page, query.Size)
r.WriteJSONData(executions)
}
// CreateExecution starts a replication
func (r *ReplicationOperationAPI) CreateExecution() {
execution := &models.Execution{}
if err := r.DecodeJSONReq(execution); err != nil {
r.SendBadRequestError(err)
return
}
policy, err := replication.PolicyCtl.Get(execution.PolicyID)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get policy %d: %v", execution.PolicyID, err))
return
}
if policy == nil {
r.SendNotFoundError(fmt.Errorf("policy %d not found", execution.PolicyID))
return
}
if !policy.Enabled {
r.SendBadRequestError(fmt.Errorf("the policy %d is disabled", execution.PolicyID))
return
}
if err = event.PopulateRegistries(replication.RegistryMgr, policy); err != nil {
r.SendInternalServerError(fmt.Errorf("failed to populate registries for policy %d: %v", execution.PolicyID, err))
return
}
trigger := r.GetString("trigger", string(model.TriggerTypeManual))
executionID, err := replication.OperationCtl.StartReplication(policy, nil, model.TriggerType(trigger))
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to start replication for policy %d: %v", execution.PolicyID, err))
return
}
r.Redirect(http.StatusCreated, strconv.FormatInt(executionID, 10))
}
// GetExecution gets one execution of the replication
func (r *ReplicationOperationAPI) GetExecution() {
executionID, err := r.GetInt64FromPath(":id")
if err != nil || executionID <= 0 {
r.SendBadRequestError(errors.New("invalid execution ID"))
return
}
execution, err := replication.OperationCtl.GetExecution(executionID)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get execution %d: %v", executionID, err))
return
}
if execution == nil {
r.SendNotFoundError(fmt.Errorf("execution %d not found", executionID))
return
}
r.WriteJSONData(execution)
}
// StopExecution stops one execution of the replication
func (r *ReplicationOperationAPI) StopExecution() {
executionID, err := r.GetInt64FromPath(":id")
if err != nil || executionID <= 0 {
r.SendBadRequestError(errors.New("invalid execution ID"))
return
}
execution, err := replication.OperationCtl.GetExecution(executionID)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get execution %d: %v", executionID, err))
return
}
if execution == nil {
r.SendNotFoundError(fmt.Errorf("execution %d not found", executionID))
return
}
if err := replication.OperationCtl.StopReplication(executionID); err != nil {
r.SendInternalServerError(fmt.Errorf("failed to stop execution %d: %v", executionID, err))
return
}
}
// ListTasks ...
func (r *ReplicationOperationAPI) ListTasks() {
executionID, err := r.GetInt64FromPath(":id")
if err != nil || executionID <= 0 {
r.SendBadRequestError(errors.New("invalid execution ID"))
return
}
execution, err := replication.OperationCtl.GetExecution(executionID)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get execution %d: %v", executionID, err))
return
}
if execution == nil {
r.SendNotFoundError(fmt.Errorf("execution %d not found", executionID))
return
}
query := &models.TaskQuery{
ExecutionID: executionID,
ResourceType: r.GetString("resource_type"),
}
status := r.GetString("status")
if len(status) > 0 {
query.Statuses = []string{status}
}
page, size, err := r.GetPaginationParams()
if err != nil {
r.SendBadRequestError(err)
return
}
query.Page = page
query.Size = size
total, tasks, err := replication.OperationCtl.ListTasks(query)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to list tasks: %v", err))
return
}
r.SetPaginationHeader(total, query.Page, query.Size)
r.WriteJSONData(tasks)
}
// GetTaskLog ...
func (r *ReplicationOperationAPI) GetTaskLog() {
executionID, err := r.GetInt64FromPath(":id")
if err != nil || executionID <= 0 {
r.SendBadRequestError(errors.New("invalid execution ID"))
return
}
execution, err := replication.OperationCtl.GetExecution(executionID)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get execution %d: %v", executionID, err))
return
}
if execution == nil {
r.SendNotFoundError(fmt.Errorf("execution %d not found", executionID))
return
}
taskID, err := r.GetInt64FromPath(":tid")
if err != nil || taskID <= 0 {
r.SendBadRequestError(errors.New("invalid task ID"))
return
}
task, err := replication.OperationCtl.GetTask(taskID)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get task %d: %v", taskID, err))
return
}
if task == nil {
r.SendNotFoundError(fmt.Errorf("task %d not found", taskID))
return
}
logBytes, err := replication.OperationCtl.GetTaskLog(taskID)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get log of task %d: %v", taskID, err))
return
}
r.Ctx.ResponseWriter.Header().Set(http.CanonicalHeaderKey("Content-Length"), strconv.Itoa(len(logBytes)))
r.Ctx.ResponseWriter.Header().Set(http.CanonicalHeaderKey("Content-Type"), "text/plain")
_, err = r.Ctx.ResponseWriter.Write(logBytes)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to write log of task %d: %v", taskID, err))
return
}
}

429
src/core/api/replication_execution_test.go Normal file
View File

@ -0,0 +1,429 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"net/http"
"testing"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/dao/models"
"github.com/goharbor/harbor/src/replication/model"
)
type fakedOperationController struct{}
func (f *fakedOperationController) StartReplication(policy *model.Policy, resource *model.Resource, trigger model.TriggerType) (int64, error) {
return 1, nil
}
func (f *fakedOperationController) StopReplication(int64) error {
return nil
}
func (f *fakedOperationController) ListExecutions(...*models.ExecutionQuery) (int64, []*models.Execution, error) {
return 1, []*models.Execution{
{
ID: 1,
PolicyID: 1,
},
}, nil
}
func (f *fakedOperationController) GetExecution(id int64) (*models.Execution, error) {
if id == 1 {
return &models.Execution{
ID: 1,
PolicyID: 1,
}, nil
}
return nil, nil
}
func (f *fakedOperationController) ListTasks(...*models.TaskQuery) (int64, []*models.Task, error) {
return 1, []*models.Task{
{
ID: 1,
ExecutionID: 1,
},
}, nil
}
func (f *fakedOperationController) GetTask(id int64) (*models.Task, error) {
if id == 1 {
return &models.Task{
ID: 1,
ExecutionID: 1,
}, nil
}
return nil, nil
}
func (f *fakedOperationController) UpdateTaskStatus(id int64, status string, statusCondition ...string) error {
return nil
}
func (f *fakedOperationController) GetTaskLog(int64) ([]byte, error) {
return []byte("success"), nil
}
type fakedPolicyManager struct{}
func (f *fakedPolicyManager) Create(*model.Policy) (int64, error) {
return 0, nil
}
func (f *fakedPolicyManager) List(...*model.PolicyQuery) (int64, []*model.Policy, error) {
return 0, nil, nil
}
func (f *fakedPolicyManager) Get(id int64) (*model.Policy, error) {
if id == 1 {
return &model.Policy{
ID: 1,
Enabled: true,
SrcRegistry: &model.Registry{
ID: 1,
},
}, nil
}
if id == 2 {
return &model.Policy{
ID: 2,
Enabled: false,
SrcRegistry: &model.Registry{
ID: 1,
},
}, nil
}
return nil, nil
}
func (f *fakedPolicyManager) GetByName(name string) (*model.Policy, error) {
if name == "duplicate_name" {
return &model.Policy{
Name: "duplicate_name",
}, nil
}
return nil, nil
}
func (f *fakedPolicyManager) Update(*model.Policy) error {
return nil
}
func (f *fakedPolicyManager) Remove(int64) error {
return nil
}
func TestListExecutions(t *testing.T) {
operationCtl := replication.OperationCtl
defer func() {
replication.OperationCtl = operationCtl
}()
replication.OperationCtl = &fakedOperationController{}
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions",
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions",
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 200
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions",
credential: sysAdmin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}
func TestCreateExecution(t *testing.T) {
operationCtl := replication.OperationCtl
policyMgr := replication.PolicyCtl
registryMgr := replication.RegistryMgr
defer func() {
replication.OperationCtl = operationCtl
replication.PolicyCtl = policyMgr
replication.RegistryMgr = registryMgr
}()
replication.OperationCtl = &fakedOperationController{}
replication.PolicyCtl = &fakedPolicyManager{}
replication.RegistryMgr = &fakedRegistryManager{}
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodPost,
url: "/api/replication/executions",
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
method: http.MethodPost,
url: "/api/replication/executions",
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 404
{
request: &testingRequest{
method: http.MethodPost,
url: "/api/replication/executions",
bodyJSON: &models.Execution{
PolicyID: 3,
},
credential: sysAdmin,
},
code: http.StatusNotFound,
},
// 400
{
request: &testingRequest{
method: http.MethodPost,
url: "/api/replication/executions",
bodyJSON: &models.Execution{
PolicyID: 2,
},
credential: sysAdmin,
},
code: http.StatusBadRequest,
},
// 201
{
request: &testingRequest{
method: http.MethodPost,
url: "/api/replication/executions",
bodyJSON: &models.Execution{
PolicyID: 1,
},
credential: sysAdmin,
},
code: http.StatusCreated,
},
}
runCodeCheckingCases(t, cases...)
}
func TestGetExecution(t *testing.T) {
operationCtl := replication.OperationCtl
defer func() {
replication.OperationCtl = operationCtl
}()
replication.OperationCtl = &fakedOperationController{}
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1",
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1",
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 404
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/2",
credential: sysAdmin,
},
code: http.StatusNotFound,
},
// 200
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1",
credential: sysAdmin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}
func TestStopExecution(t *testing.T) {
operationCtl := replication.OperationCtl
defer func() {
replication.OperationCtl = operationCtl
}()
replication.OperationCtl = &fakedOperationController{}
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodPut,
url: "/api/replication/executions/1",
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
method: http.MethodPut,
url: "/api/replication/executions/1",
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 404
{
request: &testingRequest{
method: http.MethodPut,
url: "/api/replication/executions/2",
credential: sysAdmin,
},
code: http.StatusNotFound,
},
// 200
{
request: &testingRequest{
method: http.MethodPut,
url: "/api/replication/executions/1",
credential: sysAdmin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}
func TestListTasks(t *testing.T) {
operationCtl := replication.OperationCtl
defer func() {
replication.OperationCtl = operationCtl
}()
replication.OperationCtl = &fakedOperationController{}
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1/tasks",
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1/tasks",
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 404
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/2/tasks",
credential: sysAdmin,
},
code: http.StatusNotFound,
},
// 200
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1/tasks",
credential: sysAdmin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}
func TestGetTaskLog(t *testing.T) {
operationCtl := replication.OperationCtl
defer func() {
replication.OperationCtl = operationCtl
}()
replication.OperationCtl = &fakedOperationController{}
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1/tasks/1/log",
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1/tasks/1/log",
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 404, execution not found
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/2/tasks/1/log",
credential: sysAdmin,
},
code: http.StatusNotFound,
},
// 404, task not found
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1/tasks/2/log",
credential: sysAdmin,
},
code: http.StatusNotFound,
},
// 200
{
request: &testingRequest{
method: http.MethodGet,
url: "/api/replication/executions/1/tasks/1/log",
credential: sysAdmin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}

254
src/core/api/replication_job.go
View File

@ -1,254 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"fmt"
"net/http"
"strconv"
"time"
"github.com/goharbor/harbor/src/common/dao"
common_http "github.com/goharbor/harbor/src/common/http"
common_job "github.com/goharbor/harbor/src/common/job"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/rbac"
"github.com/goharbor/harbor/src/common/utils/log"
api_models "github.com/goharbor/harbor/src/core/api/models"
"github.com/goharbor/harbor/src/core/utils"
"github.com/goharbor/harbor/src/replication/core"
)
// RepJobAPI handles request to /api/replicationJobs /api/replicationJobs/:id/log
type RepJobAPI struct {
BaseController
jobID int64
}
// Prepare validates that whether user has system admin role
func (ra *RepJobAPI) Prepare() {
ra.BaseController.Prepare()
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
return
}
if !(ra.Ctx.Request.Method == http.MethodGet || ra.SecurityCtx.IsSysAdmin()) {
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
return
}
if len(ra.GetStringFromPath(":id")) != 0 {
id, err := ra.GetInt64FromPath(":id")
if err != nil {
ra.HandleBadRequest(fmt.Sprintf("invalid ID: %s", ra.GetStringFromPath(":id")))
return
}
ra.jobID = id
}
}
// List filters jobs according to the parameters
func (ra *RepJobAPI) List() {
policyID, err := ra.GetInt64("policy_id")
if err != nil || policyID <= 0 {
ra.HandleBadRequest(fmt.Sprintf("invalid policy_id: %s", ra.GetString("policy_id")))
return
}
policy, err := core.GlobalController.GetPolicy(policyID)
if err != nil {
log.Errorf("failed to get policy %d: %v", policyID, err)
ra.CustomAbort(http.StatusInternalServerError, "")
}
if policy.ID == 0 {
ra.HandleNotFound(fmt.Sprintf("policy %d not found", policyID))
return
}
resource := rbac.NewProjectNamespace(policy.ProjectIDs[0]).Resource(rbac.ResourceReplicationJob)
if !ra.SecurityCtx.Can(rbac.ActionList, resource) {
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
return
}
query := &models.RepJobQuery{
PolicyID: policyID,
// hide the schedule job, the schedule job is used to trigger replication
// for scheduled policy
Operations: []string{models.RepOpTransfer, models.RepOpDelete},
}
query.Repository = ra.GetString("repository")
query.Statuses = ra.GetStrings("status")
query.OpUUID = ra.GetString("op_uuid")
startTimeStr := ra.GetString("start_time")
if len(startTimeStr) != 0 {
i, err := strconv.ParseInt(startTimeStr, 10, 64)
if err != nil {
ra.HandleBadRequest(fmt.Sprintf("invalid start_time: %s", startTimeStr))
return
}
t := time.Unix(i, 0)
query.StartTime = &t
}
endTimeStr := ra.GetString("end_time")
if len(endTimeStr) != 0 {
i, err := strconv.ParseInt(endTimeStr, 10, 64)
if err != nil {
ra.HandleBadRequest(fmt.Sprintf("invalid end_time: %s", endTimeStr))
return
}
t := time.Unix(i, 0)
query.EndTime = &t
}
query.Page, query.Size = ra.GetPaginationParams()
total, err := dao.GetTotalCountOfRepJobs(query)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get total count of repository jobs of policy %d: %v", policyID, err))
return
}
jobs, err := dao.GetRepJobs(query)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get repository jobs, query: %v :%v", query, err))
return
}
ra.SetPaginationHeader(total, query.Page, query.Size)
ra.Data["json"] = jobs
ra.ServeJSON()
}
// Delete ...
func (ra *RepJobAPI) Delete() {
if ra.jobID == 0 {
ra.HandleBadRequest("ID is nil")
return
}
job, err := dao.GetRepJob(ra.jobID)
if err != nil {
log.Errorf("failed to get job %d: %v", ra.jobID, err)
ra.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
}
if job == nil {
ra.HandleNotFound(fmt.Sprintf("job %d not found", ra.jobID))
return
}
if job.Status == models.JobPending || job.Status == models.JobRunning {
ra.HandleBadRequest(fmt.Sprintf("job is %s, can not be deleted", job.Status))
return
}
if err = dao.DeleteRepJob(ra.jobID); err != nil {
log.Errorf("failed to deleted job %d: %v", ra.jobID, err)
ra.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
}
}
// GetLog ...
func (ra *RepJobAPI) GetLog() {
if ra.jobID == 0 {
ra.HandleBadRequest("ID is nil")
return
}
job, err := dao.GetRepJob(ra.jobID)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get replication job %d: %v", ra.jobID, err))
return
}
if job == nil {
ra.HandleNotFound(fmt.Sprintf("replication job %d not found", ra.jobID))
return
}
policy, err := core.GlobalController.GetPolicy(job.PolicyID)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get policy %d: %v", job.PolicyID, err))
return
}
resource := rbac.NewProjectNamespace(policy.ProjectIDs[0]).Resource(rbac.ResourceReplicationJob)
if !ra.SecurityCtx.Can(rbac.ActionRead, resource) {
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
return
}
logBytes, err := utils.GetJobServiceClient().GetJobLog(job.UUID)
if err != nil {
if httpErr, ok := err.(*common_http.Error); ok {
ra.RenderError(httpErr.Code, "")
log.Errorf(fmt.Sprintf("failed to get log of job %d: %d %s",
ra.jobID, httpErr.Code, httpErr.Message))
return
}
ra.HandleInternalServerError(fmt.Sprintf("failed to get log of job %s: %v",
job.UUID, err))
return
}
ra.Ctx.ResponseWriter.Header().Set(http.CanonicalHeaderKey("Content-Length"), strconv.Itoa(len(logBytes)))
ra.Ctx.ResponseWriter.Header().Set(http.CanonicalHeaderKey("Content-Type"), "text/plain")
_, err = ra.Ctx.ResponseWriter.Write(logBytes)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to write log of job %s: %v", job.UUID, err))
return
}
}
// StopJobs stop replication jobs for the policy
func (ra *RepJobAPI) StopJobs() {
req := &api_models.StopJobsReq{}
ra.DecodeJSONReqAndValidate(req)
policy, err := core.GlobalController.GetPolicy(req.PolicyID)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get policy %d: %v", req.PolicyID, err))
return
}
if policy.ID == 0 {
ra.HandleNotFound(fmt.Sprintf("policy %d not found", req.PolicyID))
return
}
jobs, err := dao.GetRepJobs(&models.RepJobQuery{
PolicyID: policy.ID,
Operations: []string{models.RepOpTransfer, models.RepOpDelete},
})
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to list jobs of policy %d: %v", policy.ID, err))
return
}
for _, job := range jobs {
if err = utils.GetJobServiceClient().PostAction(job.UUID, common_job.JobActionStop); err != nil {
log.Errorf("failed to stop job id-%d uuid-%s: %v", job.ID, job.UUID, err)
continue
}
}
}
// TODO:add Post handler to call job service API to submit jobs by policy

537
src/core/api/replication_policy.go
View File

@ -15,428 +15,243 @@
package api
import (
"errors"
"fmt"
"net/http"
"strconv"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/rbac"
"github.com/goharbor/harbor/src/common/utils/log"
api_models "github.com/goharbor/harbor/src/core/api/models"
"github.com/goharbor/harbor/src/core/promgr"
common_model "github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/core"
rep_models "github.com/goharbor/harbor/src/replication/models"
"github.com/goharbor/harbor/src/replication/dao/models"
"github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/replication/model"
"github.com/goharbor/harbor/src/replication/registry"
)
// RepPolicyAPI handles /api/replicationPolicies /api/replicationPolicies/:id/enablement
type RepPolicyAPI struct {
// TODO rename the file to "replication.go"
// ReplicationPolicyAPI handles the replication policy requests
type ReplicationPolicyAPI struct {
BaseController
}
// Prepare validates whether the user has system admin role
func (pa *RepPolicyAPI) Prepare() {
pa.BaseController.Prepare()
if !pa.SecurityCtx.IsAuthenticated() {
pa.HandleUnauthorized()
return
}
if !(pa.Ctx.Request.Method == http.MethodGet || pa.SecurityCtx.IsSysAdmin()) {
pa.HandleForbidden(pa.SecurityCtx.GetUsername())
// Prepare ...
func (r *ReplicationPolicyAPI) Prepare() {
r.BaseController.Prepare()
if !r.SecurityCtx.IsSysAdmin() {
if !r.SecurityCtx.IsAuthenticated() {
r.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
r.SendForbiddenError(errors.New(r.SecurityCtx.GetUsername()))
return
}
}
// Get ...
func (pa *RepPolicyAPI) Get() {
id := pa.GetIDFromURL()
policy, err := core.GlobalController.GetPolicy(id)
// List the replication policies
func (r *ReplicationPolicyAPI) List() {
page, size, err := r.GetPaginationParams()
if err != nil {
log.Errorf("failed to get policy %d: %v", id, err)
pa.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
}
if policy.ID == 0 {
pa.HandleNotFound(fmt.Sprintf("policy %d not found", id))
r.SendInternalServerError(err)
return
}
resource := rbac.NewProjectNamespace(policy.ProjectIDs[0]).Resource(rbac.ResourceReplication)
if !pa.SecurityCtx.Can(rbac.ActionRead, resource) {
pa.HandleForbidden(pa.SecurityCtx.GetUsername())
return
// TODO: support more query
query := &model.PolicyQuery{
Name: r.GetString("name"),
Pagination: common_model.Pagination{
Page: page,
Size: size,
},
}
ply, err := convertFromRepPolicy(pa.ProjectMgr, policy)
total, policies, err := replication.PolicyCtl.List(query)
if err != nil {
pa.ParseAndHandleError(fmt.Sprintf("failed to convert from replication policy"), err)
r.SendInternalServerError(fmt.Errorf("failed to list policies: %v", err))
return
}
pa.Data["json"] = ply
pa.ServeJSON()
for _, policy := range policies {
if err = populateRegistries(replication.RegistryMgr, policy); err != nil {
r.SendInternalServerError(fmt.Errorf("failed to populate registries for policy %d: %v", policy.ID, err))
return
}
}
r.SetPaginationHeader(total, query.Page, query.Size)
r.WriteJSONData(policies)
}
// List ...
func (pa *RepPolicyAPI) List() {
queryParam := rep_models.QueryParameter{
Name: pa.GetString("name"),
// Create the replication policy
func (r *ReplicationPolicyAPI) Create() {
policy := &model.Policy{}
isValid, err := r.DecodeJSONReqAndValidate(policy)
if !isValid {
r.SendBadRequestError(err)
return
}
projectIDStr := pa.GetString("project_id")
if len(projectIDStr) > 0 {
projectID, err := strconv.ParseInt(projectIDStr, 10, 64)
if err != nil || projectID <= 0 {
pa.HandleBadRequest(fmt.Sprintf("invalid project ID: %s", projectIDStr))
return
}
queryParam.ProjectID = projectID
}
queryParam.Page, queryParam.PageSize = pa.GetPaginationParams()
result, err := core.GlobalController.GetPolicies(queryParam)
if !r.validateName(policy) {
return
}
if !r.validateRegistry(policy) {
return
}
id, err := replication.PolicyCtl.Create(policy)
if err != nil {
log.Errorf("failed to get policies: %v, query parameters: %v", err, queryParam)
pa.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
r.SendInternalServerError(fmt.Errorf("failed to create the policy: %v", err))
return
}
var total int64
policies := []*api_models.ReplicationPolicy{}
if result != nil {
total = result.Total
for _, policy := range result.Policies {
resource := rbac.NewProjectNamespace(policy.ProjectIDs[0]).Resource(rbac.ResourceReplication)
if !pa.SecurityCtx.Can(rbac.ActionRead, resource) {
continue
}
ply, err := convertFromRepPolicy(pa.ProjectMgr, *policy)
if err != nil {
pa.ParseAndHandleError(fmt.Sprintf("failed to convert from replication policy"), err)
return
}
policies = append(policies, ply)
}
}
pa.SetPaginationHeader(total, queryParam.Page, queryParam.PageSize)
pa.Data["json"] = policies
pa.ServeJSON()
r.Redirect(http.StatusCreated, strconv.FormatInt(id, 10))
}
// Post creates a replicartion policy
func (pa *RepPolicyAPI) Post() {
policy := &api_models.ReplicationPolicy{}
pa.DecodeJSONReqAndValidate(policy)
// check the name
exist, err := exist(policy.Name)
// make sure the policy name doesn't exist
func (r *ReplicationPolicyAPI) validateName(policy *model.Policy) bool {
p, err := replication.PolicyCtl.GetByName(policy.Name)
if err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to check the existence of policy %s: %v", policy.Name, err))
return
r.SendInternalServerError(fmt.Errorf("failed to get policy %s: %v", policy.Name, err))
return false
}
if exist {
pa.HandleConflict(fmt.Sprintf("name %s is already used", policy.Name))
return
if p != nil {
r.SendConflictError(fmt.Errorf("policy %s already exists", policy.Name))
return false
}
// check the existence of projects
for _, project := range policy.Projects {
pro, err := pa.ProjectMgr.Get(project.ProjectID)
if err != nil {
pa.ParseAndHandleError(fmt.Sprintf("failed to check the existence of project %d", project.ProjectID), err)
return
}
if pro == nil {
pa.HandleNotFound(fmt.Sprintf("project %d not found", project.ProjectID))
return
}
project.Name = pro.Name
}
// check the existence of targets
for _, target := range policy.Targets {
t, err := dao.GetRepTarget(target.ID)
if err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to get target %d: %v", target.ID, err))
return
}
if t == nil {
pa.HandleNotFound(fmt.Sprintf("target %d not found", target.ID))
return
}
}
// check the existence of labels
for _, filter := range policy.Filters {
if filter.Kind == replication.FilterItemKindLabel {
labelID := filter.Value.(int64)
label, err := dao.GetLabel(labelID)
if err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to get label %d: %v", labelID, err))
return
}
if label == nil || label.Deleted {
pa.HandleNotFound(fmt.Sprintf("label %d not found", labelID))
return
}
}
}
id, err := core.GlobalController.CreatePolicy(convertToRepPolicy(policy))
if err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to create policy: %v", err))
return
}
if policy.ReplicateExistingImageNow {
go func() {
if _, err = startReplication(id); err != nil {
log.Errorf("failed to send replication signal for policy %d: %v", id, err)
return
}
log.Infof("replication signal for policy %d sent", id)
}()
}
pa.Redirect(http.StatusCreated, strconv.FormatInt(id, 10))
return true
}
func exist(name string) (bool, error) {
result, err := core.GlobalController.GetPolicies(rep_models.QueryParameter{
Name: name,
})
// make sure the registry referred exists
func (r *ReplicationPolicyAPI) validateRegistry(policy *model.Policy) bool {
var registryID int64
if policy.SrcRegistry != nil && policy.SrcRegistry.ID > 0 {
registryID = policy.SrcRegistry.ID
} else {
registryID = policy.DestRegistry.ID
}
registry, err := replication.RegistryMgr.Get(registryID)
if err != nil {
return false, err
r.SendConflictError(fmt.Errorf("failed to get registry %d: %v", registryID, err))
return false
}
for _, policy := range result.Policies {
if policy.Name == name {
return true, nil
}
if registry == nil {
r.SendNotFoundError(fmt.Errorf("registry %d not found", registryID))
return false
}
return false, nil
return true
}
// Put updates the replication policy
func (pa *RepPolicyAPI) Put() {
id := pa.GetIDFromURL()
originalPolicy, err := core.GlobalController.GetPolicy(id)
if err != nil {
log.Errorf("failed to get policy %d: %v", id, err)
pa.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
}
if originalPolicy.ID == 0 {
pa.HandleNotFound(fmt.Sprintf("policy %d not found", id))
// Get the specified replication policy
func (r *ReplicationPolicyAPI) Get() {
id, err := r.GetInt64FromPath(":id")
if id <= 0 || err != nil {
r.SendBadRequestError(errors.New("invalid policy ID"))
return
}
policy := &api_models.ReplicationPolicy{}
pa.DecodeJSONReqAndValidate(policy)
policy, err := replication.PolicyCtl.Get(id)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get the policy %d: %v", id, err))
return
}
if policy == nil {
r.SendNotFoundError(fmt.Errorf("policy %d not found", id))
return
}
if err = populateRegistries(replication.RegistryMgr, policy); err != nil {
r.SendInternalServerError(fmt.Errorf("failed to populate registries for policy %d: %v", policy.ID, err))
return
}
r.WriteJSONData(policy)
}
// Update the replication policy
func (r *ReplicationPolicyAPI) Update() {
id, err := r.GetInt64FromPath(":id")
if id <= 0 || err != nil {
r.SendBadRequestError(errors.New("invalid policy ID"))
return
}
originalPolicy, err := replication.PolicyCtl.Get(id)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get the policy %d: %v", id, err))
return
}
if originalPolicy == nil {
r.SendNotFoundError(fmt.Errorf("policy %d not found", id))
return
}
policy := &model.Policy{}
isValid, err := r.DecodeJSONReqAndValidate(policy)
if !isValid {
r.SendBadRequestError(err)
return
}
if policy.Name != originalPolicy.Name &&
!r.validateName(policy) {
return
}
if !r.validateRegistry(policy) {
return
}
policy.ID = id
// check the name
if policy.Name != originalPolicy.Name {
exist, err := exist(policy.Name)
if err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to check the existence of policy %s: %v", policy.Name, err))
return
}
if exist {
pa.HandleConflict(fmt.Sprintf("name %s is already used", policy.Name))
return
}
}
// check the existence of projects
for _, project := range policy.Projects {
pro, err := pa.ProjectMgr.Get(project.ProjectID)
if err != nil {
pa.ParseAndHandleError(fmt.Sprintf("failed to check the existence of project %d", project.ProjectID), err)
return
}
if pro == nil {
pa.HandleNotFound(fmt.Sprintf("project %d not found", project.ProjectID))
return
}
project.Name = pro.Name
}
// check the existence of targets
for _, target := range policy.Targets {
t, err := dao.GetRepTarget(target.ID)
if err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to get target %d: %v", target.ID, err))
return
}
if t == nil {
pa.HandleNotFound(fmt.Sprintf("target %d not found", target.ID))
return
}
}
// check the existence of labels
for _, filter := range policy.Filters {
if filter.Kind == replication.FilterItemKindLabel {
labelID := filter.Value.(int64)
label, err := dao.GetLabel(labelID)
if err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to get label %d: %v", labelID, err))
return
}
if label == nil || label.Deleted {
pa.HandleNotFound(fmt.Sprintf("label %d not found", labelID))
return
}
}
}
if err = core.GlobalController.UpdatePolicy(convertToRepPolicy(policy)); err != nil {
pa.HandleInternalServerError(fmt.Sprintf("failed to update policy %d: %v", id, err))
if err := replication.PolicyCtl.Update(policy); err != nil {
r.SendInternalServerError(fmt.Errorf("failed to update the policy %d: %v", id, err))
return
}
if policy.ReplicateExistingImageNow {
go func() {
if _, err = startReplication(id); err != nil {
log.Errorf("failed to send replication signal for policy %d: %v", id, err)
return
}
log.Infof("replication signal for policy %d sent", id)
}()
}
}
// Delete the replication policy
func (pa *RepPolicyAPI) Delete() {
id := pa.GetIDFromURL()
policy, err := core.GlobalController.GetPolicy(id)
if err != nil {
log.Errorf("failed to get policy %d: %v", id, err)
pa.CustomAbort(http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
}
if policy.ID == 0 {
pa.HandleNotFound(fmt.Sprintf("policy %d not found", id))
func (r *ReplicationPolicyAPI) Delete() {
id, err := r.GetInt64FromPath(":id")
if id <= 0 || err != nil {
r.SendBadRequestError(errors.New("invalid policy ID"))
return
}
count, err := dao.GetTotalCountOfRepJobs(&models.RepJobQuery{
PolicyID: id,
Statuses: []string{models.JobRunning, models.JobRetrying, models.JobPending},
// only get the transfer and delete jobs, do not get schedule job
Operations: []string{models.RepOpTransfer, models.RepOpDelete},
})
policy, err := replication.PolicyCtl.Get(id)
if err != nil {
log.Errorf("failed to filter jobs of policy %d: %v", id, err)
pa.CustomAbort(http.StatusInternalServerError, "")
r.SendInternalServerError(fmt.Errorf("failed to get the policy %d: %v", id, err))
return
}
if count > 0 {
pa.CustomAbort(http.StatusPreconditionFailed, "policy has running/retrying/pending jobs, can not be deleted")
}
if err = core.GlobalController.RemovePolicy(id); err != nil {
log.Errorf("failed to delete policy %d: %v", id, err)
pa.CustomAbort(http.StatusInternalServerError, "")
}
}
func convertFromRepPolicy(projectMgr promgr.ProjectManager, policy rep_models.ReplicationPolicy) (*api_models.ReplicationPolicy, error) {
if policy.ID == 0 {
return nil, nil
}
// populate simple properties
ply := &api_models.ReplicationPolicy{
ID: policy.ID,
Name: policy.Name,
Description: policy.Description,
ReplicateDeletion: policy.ReplicateDeletion,
Trigger: policy.Trigger,
CreationTime: policy.CreationTime,
UpdateTime: policy.UpdateTime,
}
// populate projects
for _, projectID := range policy.ProjectIDs {
project, err := projectMgr.Get(projectID)
if err != nil {
return nil, err
}
ply.Projects = append(ply.Projects, project)
}
// populate targets
for _, targetID := range policy.TargetIDs {
target, err := dao.GetRepTarget(targetID)
if err != nil {
return nil, err
}
target.Password = ""
ply.Targets = append(ply.Targets, target)
}
// populate label used in label filter
for _, filter := range policy.Filters {
if filter.Kind == replication.FilterItemKindLabel {
labelID := filter.Value.(int64)
label, err := dao.GetLabel(labelID)
if err != nil {
return nil, err
}
filter.Value = label
}
ply.Filters = append(ply.Filters, filter)
}
// TODO call the method from replication controller
errJobCount, err := dao.GetTotalCountOfRepJobs(&models.RepJobQuery{
PolicyID: policy.ID,
Statuses: []string{models.JobError},
})
if err != nil {
return nil, err
}
ply.ErrorJobCount = errJobCount
return ply, nil
}
func convertToRepPolicy(policy *api_models.ReplicationPolicy) rep_models.ReplicationPolicy {
if policy == nil {
return rep_models.ReplicationPolicy{}
r.SendNotFoundError(fmt.Errorf("policy %d not found", id))
return
}
ply := rep_models.ReplicationPolicy{
ID: policy.ID,
Name: policy.Name,
Description: policy.Description,
Filters: policy.Filters,
ReplicateDeletion: policy.ReplicateDeletion,
Trigger: policy.Trigger,
CreationTime: policy.CreationTime,
UpdateTime: policy.UpdateTime,
_, executions, err := replication.OperationCtl.ListExecutions(&models.ExecutionQuery{
PolicyID: id,
})
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get the executions of policy %d: %v", id, err))
return
}
for _, project := range policy.Projects {
ply.ProjectIDs = append(ply.ProjectIDs, project.ProjectID)
ply.Namespaces = append(ply.Namespaces, project.Name)
for _, execution := range executions {
if execution.Status == models.ExecutionStatusInProgress {
r.SendInternalServerError(fmt.Errorf("the policy %d has running executions, can not be deleted", id))
return
}
}
for _, target := range policy.Targets {
ply.TargetIDs = append(ply.TargetIDs, target.ID)
if err := replication.PolicyCtl.Remove(id); err != nil {
r.SendInternalServerError(fmt.Errorf("failed to delete the policy %d: %v", id, err))
return
}
return ply
}
// ignore the credential for the registries
func populateRegistries(registryMgr registry.Manager, policy *model.Policy) error {
if err := event.PopulateRegistries(registryMgr, policy); err != nil {
return err
}
if policy.SrcRegistry != nil {
policy.SrcRegistry.Credential = nil
}
if policy.DestRegistry != nil {
policy.DestRegistry.Credential = nil
}
return nil
}

943
src/core/api/replication_policy_test.go
View File

File diff suppressed because it is too large Load Diff

92
src/core/api/replication_test.go
View File

@ -1,92 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"fmt"
"net/http"
"testing"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
api_models "github.com/goharbor/harbor/src/core/api/models"
"github.com/goharbor/harbor/src/replication"
"github.com/stretchr/testify/require"
)
const (
replicationAPIBaseURL = "/api/replications"
)
func TestReplicationAPIPost(t *testing.T) {
targetID, err := dao.AddRepTarget(
models.RepTarget{
Name: "test_replication_target",
URL: "127.0.0.1",
Username: "username",
Password: "password",
})
require.Nil(t, err)
defer dao.DeleteRepTarget(targetID)
policyID, err := dao.AddRepPolicy(
models.RepPolicy{
Name: "test_replication_policy",
ProjectID: 1,
TargetID: targetID,
Trigger: fmt.Sprintf("{\"kind\":\"%s\"}", replication.TriggerKindManual),
})
require.Nil(t, err)
defer dao.DeleteRepPolicy(policyID)
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
method: http.MethodPost,
url: replicationAPIBaseURL,
bodyJSON: &api_models.Replication{
PolicyID: policyID,
},
},
code: http.StatusUnauthorized,
},
// 404
{
request: &testingRequest{
method: http.MethodPost,
url: replicationAPIBaseURL,
bodyJSON: &api_models.Replication{
PolicyID: 10000,
},
credential: admin,
},
code: http.StatusNotFound,
},
// 200
{
request: &testingRequest{
method: http.MethodPost,
url: replicationAPIBaseURL,
bodyJSON: &api_models.Replication{
PolicyID: policyID,
},
credential: admin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}

230
src/core/api/repository.go
View File

@ -24,6 +24,7 @@ import (
"strings"
"time"
"errors"
"github.com/docker/distribution/manifest/schema1"
"github.com/docker/distribution/manifest/schema2"
"github.com/goharbor/harbor/src/common"
@ -37,10 +38,10 @@ import (
"github.com/goharbor/harbor/src/common/utils/notary"
"github.com/goharbor/harbor/src/common/utils/registry"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/notifier"
coreutils "github.com/goharbor/harbor/src/core/utils"
"github.com/goharbor/harbor/src/replication/event/notification"
"github.com/goharbor/harbor/src/replication/event/topic"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/replication/model"
)
// RepositoryAPI handles request to /api/repositories /api/repositories/tags /api/repositories/manifests, the parm has to be put
@ -110,13 +111,13 @@ type manifestResp struct {
func (ra *RepositoryAPI) Get() {
projectID, err := ra.GetInt64("project_id")
if err != nil || projectID <= 0 {
ra.HandleBadRequest(fmt.Sprintf("invalid project_id %s", ra.GetString("project_id")))
ra.SendBadRequestError(fmt.Errorf("invalid project_id %s", ra.GetString("project_id")))
return
}
labelID, err := ra.GetInt64("label_id", 0)
if err != nil {
ra.HandleBadRequest(fmt.Sprintf("invalid label_id: %s", ra.GetString("label_id")))
ra.SendBadRequestError(fmt.Errorf("invalid label_id: %s", ra.GetString("label_id")))
return
}
@ -128,17 +129,17 @@ func (ra *RepositoryAPI) Get() {
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("project %d not found", projectID))
ra.SendNotFoundError(fmt.Errorf("project %d not found", projectID))
return
}
resource := rbac.NewProjectNamespace(projectID).Resource(rbac.ResourceRepository)
if !ra.SecurityCtx.Can(rbac.ActionList, resource) {
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
@ -147,19 +148,24 @@ func (ra *RepositoryAPI) Get() {
Name: ra.GetString("q"),
LabelID: labelID,
}
query.Page, query.Size = ra.GetPaginationParams()
query.Page, query.Size, err = ra.GetPaginationParams()
if err != nil {
ra.SendBadRequestError(err)
return
}
query.Sort = ra.GetString("sort")
total, err := dao.GetTotalOfRepositories(query)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get total of repositories of project %d: %v",
ra.SendInternalServerError(fmt.Errorf("failed to get total of repositories of project %d: %v",
projectID, err))
return
}
repositories, err := getRepositories(query)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get repository: %v", err))
ra.SendInternalServerError(fmt.Errorf("failed to get repository: %v", err))
return
}
@ -240,25 +246,26 @@ func (ra *RepositoryAPI) Delete() {
}
if project == nil {
ra.HandleNotFound(fmt.Sprintf("project %s not found", projectName))
ra.SendNotFoundError(fmt.Errorf("project %s not found", projectName))
return
}
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
resource := rbac.NewProjectNamespace(project.ProjectID).Resource(rbac.ResourceRepository)
if !ra.SecurityCtx.Can(rbac.ActionDelete, resource) {
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
rc, err := coreutils.NewRepositoryClientForUI(ra.SecurityCtx.GetUsername(), repoName)
if err != nil {
log.Errorf("error occurred while initializing repository client for %s: %v", repoName, err)
ra.CustomAbort(http.StatusInternalServerError, "internal error")
ra.SendInternalServerError(errors.New("internal error"))
return
}
tags := []string{}
@ -266,18 +273,13 @@ func (ra *RepositoryAPI) Delete() {
if len(tag) == 0 {
tagList, err := rc.ListTag()
if err != nil {
log.Errorf("error occurred while listing tags of %s: %v", repoName, err)
if regErr, ok := err.(*commonhttp.Error); ok {
ra.CustomAbort(regErr.Code, regErr.Message)
}
ra.CustomAbort(http.StatusInternalServerError, "internal error")
ra.ParseAndHandleError(fmt.Sprintf("error occurred while listing tags of %s", repoName), err)
return
}
// TODO remove the logic if the bug of registry is fixed
if len(tagList) == 0 {
ra.HandleNotFound(fmt.Sprintf("no tags found for repository %s", repoName))
ra.SendNotFoundError(fmt.Errorf("no tags found for repository %s", repoName))
return
}
@ -289,7 +291,7 @@ func (ra *RepositoryAPI) Delete() {
if config.WithNotary() {
signedTags, err := getSignatures(ra.SecurityCtx.GetUsername(), repoName)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf(
ra.SendInternalServerError(fmt.Errorf(
"failed to get signatures for repository %s: %v", repoName, err))
return
}
@ -298,12 +300,14 @@ func (ra *RepositoryAPI) Delete() {
digest, _, err := rc.ManifestExist(t)
if err != nil {
log.Errorf("Failed to Check the digest of tag: %s, error: %v", t, err.Error())
ra.CustomAbort(http.StatusInternalServerError, err.Error())
ra.SendInternalServerError(err)
return
}
log.Debugf("Tag: %s, digest: %s", t, digest)
if _, ok := signedTags[digest]; ok {
log.Errorf("Found signed tag, repostory: %s, tag: %s, deletion will be canceled", repoName, t)
ra.CustomAbort(http.StatusPreconditionFailed, fmt.Sprintf("tag %s is signed", t))
ra.SendPreconditionFailedError(fmt.Errorf("tag %s is signed", t))
return
}
}
}
@ -311,7 +315,7 @@ func (ra *RepositoryAPI) Delete() {
for _, t := range tags {
image := fmt.Sprintf("%s:%s", repoName, t)
if err = dao.DeleteLabelsOfResource(common.ResourceTypeImage, image); err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to delete labels of image %s: %v", image, err))
ra.SendInternalServerError(fmt.Errorf("failed to delete labels of image %s: %v", image, err))
return
}
if err = rc.DeleteTag(t); err != nil {
@ -319,24 +323,29 @@ func (ra *RepositoryAPI) Delete() {
if regErr.Code == http.StatusNotFound {
continue
}
log.Errorf("failed to delete tag %s: %v", t, err)
ra.CustomAbort(regErr.Code, regErr.Message)
}
log.Errorf("error occurred while deleting tag %s:%s: %v", repoName, t, err)
ra.CustomAbort(http.StatusInternalServerError, "internal error")
ra.ParseAndHandleError(fmt.Sprintf("failed to delete tag %s", t), err)
return
}
log.Infof("delete tag: %s:%s", repoName, t)
go func(tag string) {
image := repoName + ":" + tag
err := notifier.Publish(topic.ReplicationEventTopicOnDeletion, notification.OnDeletionNotification{
Image: image,
})
if err != nil {
log.Errorf("failed to publish on deletion topic for resource %s: %v", image, err)
return
e := &event.Event{
Type: event.EventTypeImagePush,
Resource: &model.Resource{
Type: model.ResourceTypeImage,
Metadata: &model.ResourceMetadata{
Repository: &model.Repository{
Name: repoName,
},
Vtags: []string{tag},
},
Deleted: true,
},
}
if err := replication.EventHandler.Handle(e); err != nil {
log.Errorf("failed to handle event: %v", err)
}
log.Debugf("the on deletion topic for resource %s published", image)
}(t)
go func(tag string) {
@ -356,12 +365,13 @@ func (ra *RepositoryAPI) Delete() {
exist, err := repositoryExist(repoName, rc)
if err != nil {
log.Errorf("failed to check the existence of repository %s: %v", repoName, err)
ra.CustomAbort(http.StatusInternalServerError, "")
ra.SendInternalServerError(fmt.Errorf("failed to check the existence of repository %s: %v", repoName, err))
return
}
if !exist {
repository, err := dao.GetRepositoryByName(repoName)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get repository %s: %v", repoName, err))
ra.SendInternalServerError(fmt.Errorf("failed to get repository %s: %v", repoName, err))
return
}
if repository == nil {
@ -371,13 +381,14 @@ func (ra *RepositoryAPI) Delete() {
if err = dao.DeleteLabelsOfResource(common.ResourceTypeRepository,
strconv.FormatInt(repository.RepositoryID, 10)); err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to delete labels of repository %s: %v",
ra.SendInternalServerError(fmt.Errorf("failed to delete labels of repository %s: %v",
repoName, err))
return
}
if err = dao.DeleteRepository(repoName); err != nil {
log.Errorf("failed to delete repository %s: %v", repoName, err)
ra.CustomAbort(http.StatusInternalServerError, "")
ra.SendInternalServerError(fmt.Errorf("failed to delete repository %s: %v", repoName, err))
return
}
}
}
@ -388,38 +399,38 @@ func (ra *RepositoryAPI) GetTag() {
tag := ra.GetString(":tag")
exist, _, err := ra.checkExistence(repository, tag)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to check the existence of resource, error: %v", err))
ra.SendInternalServerError(fmt.Errorf("failed to check the existence of resource, error: %v", err))
return
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("resource: %s:%s not found", repository, tag))
ra.SendNotFoundError(fmt.Errorf("resource: %s:%s not found", repository, tag))
return
}
project, _ := utils.ParseRepository(repository)
resource := rbac.NewProjectNamespace(project).Resource(rbac.ResourceRepositoryTag)
if !ra.SecurityCtx.Can(rbac.ActionRead, resource) {
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
client, err := coreutils.NewRepositoryClientForUI(ra.SecurityCtx.GetUsername(), repository)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to initialize the client for %s: %v",
ra.SendInternalServerError(fmt.Errorf("failed to initialize the client for %s: %v",
repository, err))
return
}
_, exist, err = client.ManifestExist(tag)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to check the existence of %s:%s: %v", repository, tag, err))
ra.SendInternalServerError(fmt.Errorf("failed to check the existence of %s:%s: %v", repository, tag, err))
return
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("%s not found", tag))
ra.SendNotFoundError(fmt.Errorf("%s not found", tag))
return
}
@ -432,38 +443,41 @@ func (ra *RepositoryAPI) GetTag() {
// Retag tags an existing image to another tag in this repo, the source image is specified by request body.
func (ra *RepositoryAPI) Retag() {
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
repoName := ra.GetString(":splat")
project, repo := utils.ParseRepository(repoName)
if !utils.ValidateRepo(repo) {
ra.HandleBadRequest(fmt.Sprintf("invalid repo '%s'", repo))
ra.SendBadRequestError(fmt.Errorf("invalid repo '%s'", repo))
return
}
request := models.RetagRequest{}
ra.DecodeJSONReq(&request)
if err := ra.DecodeJSONReq(&request); err != nil {
ra.SendBadRequestError(err)
return
}
srcImage, err := models.ParseImage(request.SrcImage)
if err != nil {
ra.HandleBadRequest(fmt.Sprintf("invalid src image string '%s', should in format '<project>/<repo>:<tag>'", request.SrcImage))
ra.SendBadRequestError(fmt.Errorf("invalid src image string '%s', should in format '<project>/<repo>:<tag>'", request.SrcImage))
return
}
if !utils.ValidateTag(request.Tag) {
ra.HandleBadRequest(fmt.Sprintf("invalid tag '%s'", request.Tag))
ra.SendBadRequestError(fmt.Errorf("invalid tag '%s'", request.Tag))
return
}
// Check whether source image exists
exist, _, err := ra.checkExistence(fmt.Sprintf("%s/%s", srcImage.Project, srcImage.Repo), srcImage.Tag)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("check existence of %s error: %v", request.SrcImage, err))
ra.SendInternalServerError(fmt.Errorf("check existence of %s error: %v", request.SrcImage, err))
return
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("image %s not exist", request.SrcImage))
ra.SendNotFoundError(fmt.Errorf("image %s not exist", request.SrcImage))
return
}
@ -474,7 +488,7 @@ func (ra *RepositoryAPI) Retag() {
return
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("project %s not found", project))
ra.SendNotFoundError(fmt.Errorf("project %s not found", project))
return
}
@ -482,11 +496,11 @@ func (ra *RepositoryAPI) Retag() {
if !request.Override {
exist, _, err := ra.checkExistence(repoName, request.Tag)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("check existence of %s:%s error: %v", repoName, request.Tag, err))
ra.SendInternalServerError(fmt.Errorf("check existence of %s:%s error: %v", repoName, request.Tag, err))
return
}
if exist {
ra.HandleConflict(fmt.Sprintf("tag '%s' already existed for '%s'", request.Tag, repoName))
ra.SendConflictError(fmt.Errorf("tag '%s' already existed for '%s'", request.Tag, repoName))
return
}
}
@ -495,7 +509,7 @@ func (ra *RepositoryAPI) Retag() {
srcResource := rbac.NewProjectNamespace(srcImage.Project).Resource(rbac.ResourceRepository)
if !ra.SecurityCtx.Can(rbac.ActionPull, srcResource) {
log.Errorf("user has no read permission to project '%s'", srcImage.Project)
ra.HandleForbidden(fmt.Sprintf("%s has no read permission to project %s", ra.SecurityCtx.GetUsername(), srcImage.Project))
ra.SendForbiddenError(fmt.Errorf("%s has no read permission to project %s", ra.SecurityCtx.GetUsername(), srcImage.Project))
return
}
@ -503,7 +517,7 @@ func (ra *RepositoryAPI) Retag() {
destResource := rbac.NewProjectNamespace(project).Resource(rbac.ResourceRepository)
if !ra.SecurityCtx.Can(rbac.ActionPush, destResource) {
log.Errorf("user has no write permission to project '%s'", project)
ra.HandleForbidden(fmt.Sprintf("%s has no write permission to project %s", ra.SecurityCtx.GetUsername(), project))
ra.SendForbiddenError(fmt.Errorf("%s has no write permission to project %s", ra.SecurityCtx.GetUsername(), project))
return
}
@ -513,7 +527,7 @@ func (ra *RepositoryAPI) Retag() {
Repo: repo,
Tag: request.Tag,
}); err != nil {
ra.HandleInternalServerError(fmt.Sprintf("%v", err))
ra.SendInternalServerError(fmt.Errorf("%v", err))
}
}
@ -522,7 +536,7 @@ func (ra *RepositoryAPI) GetTags() {
repoName := ra.GetString(":splat")
labelID, err := ra.GetInt64("label_id", 0)
if err != nil {
ra.HandleBadRequest(fmt.Sprintf("invalid label_id: %s", ra.GetString("label_id")))
ra.SendBadRequestError(fmt.Errorf("invalid label_id: %s", ra.GetString("label_id")))
return
}
@ -535,29 +549,30 @@ func (ra *RepositoryAPI) GetTags() {
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("project %s not found", projectName))
ra.SendNotFoundError(fmt.Errorf("project %s not found", projectName))
return
}
resource := rbac.NewProjectNamespace(projectName).Resource(rbac.ResourceRepositoryTag)
if !ra.SecurityCtx.Can(rbac.ActionList, resource) {
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
client, err := coreutils.NewRepositoryClientForUI(ra.SecurityCtx.GetUsername(), repoName)
if err != nil {
log.Errorf("error occurred while initializing repository client for %s: %v", repoName, err)
ra.CustomAbort(http.StatusInternalServerError, "internal error")
ra.SendInternalServerError(errors.New("internal error"))
return
}
tags, err := client.ListTag()
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get tag of %s: %v", repoName, err))
ra.SendInternalServerError(fmt.Errorf("failed to get tag of %s: %v", repoName, err))
return
}
@ -568,7 +583,7 @@ func (ra *RepositoryAPI) GetTags() {
ResourceType: common.ResourceTypeImage,
})
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to list resource labels: %v", err))
ra.SendInternalServerError(fmt.Errorf("failed to list resource labels: %v", err))
return
}
labeledTags := map[string]struct{}{}
@ -731,7 +746,7 @@ func (ra *RepositoryAPI) GetManifests() {
}
if version != "v1" && version != "v2" {
ra.HandleBadRequest("version should be v1 or v2")
ra.SendBadRequestError(errors.New("version should be v1 or v2"))
return
}
@ -744,36 +759,32 @@ func (ra *RepositoryAPI) GetManifests() {
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("project %s not found", projectName))
ra.SendNotFoundError(fmt.Errorf("project %s not found", projectName))
return
}
resource := rbac.NewProjectNamespace(projectName).Resource(rbac.ResourceRepositoryTagManifest)
if !ra.SecurityCtx.Can(rbac.ActionRead, resource) {
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
rc, err := coreutils.NewRepositoryClientForUI(ra.SecurityCtx.GetUsername(), repoName)
if err != nil {
log.Errorf("error occurred while initializing repository client for %s: %v", repoName, err)
ra.CustomAbort(http.StatusInternalServerError, "internal error")
ra.SendInternalServerError(errors.New("internal error"))
return
}
manifest, err := getManifest(rc, tag, version)
if err != nil {
log.Errorf("error occurred while getting manifest of %s:%s: %v", repoName, tag, err)
if regErr, ok := err.(*commonhttp.Error); ok {
ra.CustomAbort(regErr.Code, regErr.Message)
}
ra.CustomAbort(http.StatusInternalServerError, "internal error")
ra.ParseAndHandleError(fmt.Sprintf("error occurred while getting manifest of %s:%s", repoName, tag), err)
return
}
ra.Data["json"] = manifest
@ -826,7 +837,7 @@ func getManifest(client *registry.Repository,
func (ra *RepositoryAPI) GetTopRepos() {
count, err := ra.GetInt("count", 10)
if err != nil || count <= 0 {
ra.HandleBadRequest(fmt.Sprintf("invalid count: %s", ra.GetString("count")))
ra.SendBadRequestError(fmt.Errorf("invalid count: %s", ra.GetString("count")))
return
}
@ -839,7 +850,7 @@ func (ra *RepositoryAPI) GetTopRepos() {
if ra.SecurityCtx.IsAuthenticated() {
list, err := ra.SecurityCtx.GetMyProjects()
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get projects which the user %s is a member of: %v",
ra.SendInternalServerError(fmt.Errorf("failed to get projects which the user %s is a member of: %v",
ra.SecurityCtx.GetUsername(), err))
return
}
@ -853,7 +864,8 @@ func (ra *RepositoryAPI) GetTopRepos() {
repos, err := dao.GetTopRepos(projectIDs, count)
if err != nil {
log.Errorf("failed to get top repos: %v", err)
ra.CustomAbort(http.StatusInternalServerError, "internal server error")
ra.SendInternalServerError(errors.New("internal server error"))
return
}
ra.Data["json"] = assembleReposInParallel(repos)
@ -865,35 +877,38 @@ func (ra *RepositoryAPI) Put() {
name := ra.GetString(":splat")
repository, err := dao.GetRepositoryByName(name)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get repository %s: %v", name, err))
ra.SendInternalServerError(fmt.Errorf("failed to get repository %s: %v", name, err))
return
}
if repository == nil {
ra.HandleNotFound(fmt.Sprintf("repository %s not found", name))
ra.SendNotFoundError(fmt.Errorf("repository %s not found", name))
return
}
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
project, _ := utils.ParseRepository(name)
resource := rbac.NewProjectNamespace(project).Resource(rbac.ResourceRepository)
if !ra.SecurityCtx.Can(rbac.ActionUpdate, resource) {
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
desc := struct {
Description string `json:"description"`
}{}
ra.DecodeJSONReq(&desc)
if err := ra.DecodeJSONReq(&desc); err != nil {
ra.SendBadRequestError(err)
return
}
repository.Description = desc.Description
if err = dao.UpdateRepository(*repository); err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to update repository %s: %v", name, err))
ra.SendInternalServerError(fmt.Errorf("failed to update repository %s: %v", name, err))
return
}
}
@ -911,17 +926,17 @@ func (ra *RepositoryAPI) GetSignatures() {
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("project %s not found", projectName))
ra.SendNotFoundError(fmt.Errorf("project %s not found", projectName))
return
}
resource := rbac.NewProjectNamespace(projectName).Resource(rbac.ResourceRepository)
if !ra.SecurityCtx.Can(rbac.ActionRead, resource) {
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
@ -929,7 +944,8 @@ func (ra *RepositoryAPI) GetSignatures() {
ra.SecurityCtx.GetUsername(), repoName)
if err != nil {
log.Errorf("Error while fetching signature from notary: %v", err)
ra.CustomAbort(http.StatusInternalServerError, "internal error")
ra.SendInternalServerError(errors.New("internal error"))
return
}
ra.Data["json"] = targets
ra.ServeJSON()
@ -939,7 +955,7 @@ func (ra *RepositoryAPI) GetSignatures() {
func (ra *RepositoryAPI) ScanImage() {
if !config.WithClair() {
log.Warningf("Harbor is not deployed with Clair, scan is disabled.")
ra.RenderError(http.StatusServiceUnavailable, "")
ra.SendInternalServerError(errors.New("harbor is not deployed with Clair, scan is disabled"))
return
}
repoName := ra.GetString(":splat")
@ -952,23 +968,23 @@ func (ra *RepositoryAPI) ScanImage() {
return
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("project %s not found", projectName))
ra.SendNotFoundError(fmt.Errorf("project %s not found", projectName))
return
}
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
resource := rbac.NewProjectNamespace(projectName).Resource(rbac.ResourceRepositoryTagScanJob)
if !ra.SecurityCtx.Can(rbac.ActionCreate, resource) {
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
err = coreutils.TriggerImageScan(repoName, tag)
if err != nil {
log.Errorf("Error while calling job service to trigger image scan: %v", err)
ra.HandleInternalServerError("Failed to scan image, please check log for details")
ra.SendInternalServerError(errors.New("Failed to scan image, please check log for details"))
return
}
}
@ -977,18 +993,18 @@ func (ra *RepositoryAPI) ScanImage() {
func (ra *RepositoryAPI) VulnerabilityDetails() {
if !config.WithClair() {
log.Warningf("Harbor is not deployed with Clair, it's not impossible to get vulnerability details.")
ra.RenderError(http.StatusServiceUnavailable, "")
ra.SendInternalServerError(errors.New("harbor is not deployed with Clair, it's not impossible to get vulnerability details"))
return
}
repository := ra.GetString(":splat")
tag := ra.GetString(":tag")
exist, digest, err := ra.checkExistence(repository, tag)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to check the existence of resource, error: %v", err))
ra.SendInternalServerError(fmt.Errorf("failed to check the existence of resource, error: %v", err))
return
}
if !exist {
ra.HandleNotFound(fmt.Sprintf("resource: %s:%s not found", repository, tag))
ra.SendNotFoundError(fmt.Errorf("resource: %s:%s not found", repository, tag))
return
}
project, _ := utils.ParseRepository(repository)
@ -996,16 +1012,16 @@ func (ra *RepositoryAPI) VulnerabilityDetails() {
resource := rbac.NewProjectNamespace(project).Resource(rbac.ResourceRepositoryTagVulnerability)
if !ra.SecurityCtx.Can(rbac.ActionList, resource) {
if !ra.SecurityCtx.IsAuthenticated() {
ra.HandleUnauthorized()
ra.SendUnAuthorizedError(errors.New("Unauthorized"))
return
}
ra.HandleForbidden(ra.SecurityCtx.GetUsername())
ra.SendForbiddenError(errors.New(ra.SecurityCtx.GetUsername()))
return
}
res := []*models.VulnerabilityItem{}
overview, err := dao.GetImgScanOverview(digest)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("failed to get the scan overview, error: %v", err))
ra.SendInternalServerError(fmt.Errorf("failed to get the scan overview, error: %v", err))
return
}
if overview != nil && len(overview.DetailsKey) > 0 {
@ -1013,7 +1029,7 @@ func (ra *RepositoryAPI) VulnerabilityDetails() {
log.Debugf("The key for getting details: %s", overview.DetailsKey)
details, err := clairClient.GetResult(overview.DetailsKey)
if err != nil {
ra.HandleInternalServerError(fmt.Sprintf("Failed to get scan details from Clair, error: %v", err))
ra.SendInternalServerError(fmt.Errorf("Failed to get scan details from Clair, error: %v", err))
return
}
res = transformVulnerabilities(details)

Some files were not shown because too many files have changed in this diff Show More