Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-11-22 10:15:35 +01:00
Code Issues Projects Releases Wiki Activity

Remove dead code

Browse Source 
Remove dead code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
This commit is contained in:
Wenkai Yin 2020-02-13 14:28:36 +08:00
parent f36152a560
commit bd204464f3
53 changed files with 32 additions and 7207 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

401
api/v2.0/legacy_swagger.yaml
View File

@ -1066,31 +1066,6 @@ paths:
'500':
description: Unexpected internal errors.
'/repositories/{repo_name}':
delete:
summary: Delete a repository.
description: |
This endpoint let user delete a repository with name.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository which will be deleted.
tags:
- Products
responses:
'200':
description: Delete successfully.
'400':
description: Invalid repo_name.
'401':
description: Unauthorized.
'403':
description: Forbidden.
'404':
description: Repository not found.
'412':
description: Precondition Failed.
put:
summary: Update description of the repository.
description: |
@ -1118,382 +1093,6 @@ paths:
description: Forbidden.
'404':
description: Repository not found.
'/repositories/{repo_name}/labels':
get:
summary: Get labels of a repository.
description: |
Get labels of a repository specified by the repo_name.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository.
tags:
- Products
responses:
'200':
description: Successfully.
schema:
type: array
items:
$ref: '#/definitions/Label'
'401':
description: Unauthorized.
'403':
description: Forbidden. User should have read permisson for the repository to perform the action.
'404':
description: Repository not found.
post:
summary: Add a label to the repository.
description: |
Add a label to the repository.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository.
- name: label
in: body
description: Only the ID property is required.
required: true
schema:
$ref: '#/definitions/Label'
tags:
- Products
responses:
'200':
description: Successfully.
'401':
description: Unauthorized.
'403':
description: Forbidden. User should have write permisson for the repository to perform the action.
'404':
description: Resource not found.
'/repositories/{repo_name}/labels/{label_id}':
delete:
summary: Delete label from the repository.
description: |
Delete the label from the repository specified by the repo_name.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository.
- name: label_id
in: path
type: integer
required: true
description: The ID of label.
tags:
- Products
responses:
'200':
description: Successfully.
'401':
description: Unauthorized.
'403':
description: Forbidden. User should have write permisson for the repository to perform the action.
'404':
description: Resource not found.
'/repositories/{repo_name}/tags/{tag}':
get:
summary: Get the tag of the repository.
description: |
This endpoint aims to retrieve the tag of the repository. If deployed with Notary, the signature property of response represents whether the image is singed or not. If the property is null, the image is unsigned.
parameters:
- name: repo_name
in: path
type: string
required: true
description: Relevant repository name.
- name: tag
in: path
type: string
required: true
description: Tag of the repository.
tags:
- Products
responses:
'200':
description: Get tag successfully.
schema:
$ref: '#/definitions/DetailedTag'
'500':
description: Unexpected internal errors.
delete:
summary: Delete a tag in a repository.
description: |
This endpoint let user delete tags with repo name and tag.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository which will be deleted.
- name: tag
in: path
type: string
required: true
description: Tag of a repository.
tags:
- Products
responses:
'200':
description: Delete tag successfully.
'400':
description: Invalid repo_name.
'401':
description: Unauthorized.
'403':
description: Forbidden.
'404':
description: Repository or tag not found.
'/repositories/{repo_name}/tags':
get:
summary: Get tags of a relevant repository.
description: |
This endpoint aims to retrieve tags from a relevant repository. If deployed with Notary, the signature property of response represents whether the image is singed or not. If the property is null, the image is unsigned.
parameters:
- name: repo_name
in: path
type: string
required: true
description: Relevant repository name.
- name: label_id
in: query
type: string
required: false
description: A label ID.
- name: detail
in: query
type: boolean
required: false
description: Bool value indicating whether return detailed information of the tag, such as vulnerability scan info, if set to false, only tag name is returned.
tags:
- Products
responses:
'200':
description: Get tags successfully.
schema:
type: array
items:
$ref: '#/definitions/DetailedTag'
'500':
description: Unexpected internal errors.
post:
summary: Retag an image
description: >
This endpoint tags an existing image with another tag in this repo, source images
can be in different repos or projects.
parameters:
- name: repo_name
in: path
type: string
required: true
description: Relevant repository name.
- name: request
in: body
description: Request to give source image and target tag.
required: true
schema:
$ref: '#/definitions/RetagReq'
tags:
- Products
responses:
'200':
description: Image retag successfully.
'400':
description: Invalid image values provided.
'401':
description: User has no permission to the source project or destination project.
'403':
description: Forbiden as quota exceeded.
'404':
description: Project or repository not found.
'409':
description: Target tag already exists.
'500':
description: Unexpected internal errors.
'/repositories/{repo_name}/tags/{tag}/labels':
get:
summary: Get labels of an image.
description: |
Get labels of an image specified by the repo_name and tag.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository.
- name: tag
in: path
type: string
required: true
description: The tag of the image.
tags:
- Products
responses:
'200':
description: Successfully.
schema:
type: array
items:
$ref: '#/definitions/Label'
'401':
description: Unauthorized.
'403':
description: Forbidden. User should have read permisson for the image to perform the action.
'404':
description: Resource not found.
post:
summary: Add a label to image.
description: |
Add a label to the image.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository.
- name: tag
in: path
type: string
required: true
description: The tag of the image.
- name: label
in: body
description: Only the ID property is required.
required: true
schema:
$ref: '#/definitions/Label'
tags:
- Products
responses:
'200':
description: Successfully.
'401':
description: Unauthorized.
'403':
description: Forbidden. User should have write permisson for the image to perform the action.
'404':
description: Resource not found.
'/repositories/{repo_name}/tags/{tag}/labels/{label_id}':
delete:
summary: Delete label from the image.
description: |
Delete the label from the image specified by the repo_name and tag.
parameters:
- name: repo_name
in: path
type: string
required: true
description: The name of repository.
- name: tag
in: path
type: string
required: true
description: The tag of the image.
- name: label_id
in: path
type: integer
required: true
description: The ID of label.
tags:
- Products
responses:
'200':
description: Successfully.
'401':
description: Unauthorized.
'403':
description: Forbidden. User should have write permisson for the image to perform the action.
'404':
description: Resource not found.
'/repositories/{repo_name}/tags/{tag}/manifest':
get:
summary: Get manifests of a relevant repository.
description: |
This endpoint aims to retreive manifests from a relevant repository.
parameters:
- name: repo_name
in: path
type: string
required: true
description: Repository name
- name: tag
in: path
type: string
required: true
description: Tag name
- name: version
in: query
type: string
required: false
description: 'The version of manifest, valid value are "v1" and "v2", default is "v2"'
tags:
- Products
responses:
'200':
description: Retrieved manifests from a relevant repository successfully.
schema:
$ref: '#/definitions/Manifest'
'404':
description: Retrieved manifests from a relevant repository not found.
'500':
description: Unexpected internal errors.
'/repositories/{repo_name}/signatures':
get:
summary: Get signature information of a repository
description: |
This endpoint aims to retrieve signature information of a repository, the data is
from the nested notary instance of Harbor.
If the repository does not have any signature information in notary, this API will
return an empty list with response code 200, instead of 404
parameters:
- name: repo_name
in: path
type: string
required: true
description: repository name.
tags:
- Products
responses:
'200':
description: Retrieved signatures.
schema:
type: array
items:
$ref: '#/definitions/RepoSignature'
'500':
description: Server side error.
/repositories/top:
get:
summary: Get public repositories which are accessed most.
description: |
This endpoint aims to let users see the most popular public repositories
parameters:
- name: count
in: query
type: integer
format: int32
required: false
description: 'The number of the requested public repositories, default is 10 if not provided.'
tags:
- Products
responses:
'200':
description: Get popular repositories successfully.
schema:
type: array
items:
$ref: '#/definitions/Repository'
'400':
description: Bad request because of invalid count.
'500':
description: Unexpected internal errors.
/logs:
get:
summary: Get recent logs of the projects which the user is a member of

1
make/photon/prepare/templates/core/env.jinja
View File

@ -1,7 +1,6 @@
CONFIG_PATH=/etc/core/app.conf
UAA_CA_ROOT=/etc/core/certificates/uaa_ca.pem
_REDIS_URL={{redis_host}}:{{redis_port}},100,{{redis_password}}
SYNC_REGISTRY=false
SYNC_QUOTA=true
CHART_CACHE_DRIVER={{chart_cache_driver}}
_REDIS_URL_REG={{redis_url_reg}}

23
src/common/dao/dao_test.go
View File

@ -751,29 +751,6 @@ func TestGetRepositoryByName(t *testing.T) {
}
}
func TestIncreasePullCount(t *testing.T) {
if err := IncreasePullCount(currentRepository.Name); err != nil {
log.Errorf("Error happens when increasing pull count: %v", currentRepository.Name)
}
repository, err := GetRepositoryByName(currentRepository.Name)
if err != nil {
t.Errorf("Error occurred in GetRepositoryByName: %v", err)
}
if repository.PullCount != 1 {
t.Errorf("repository pull count is not 1 after IncreasePullCount, expected: 1, actual: %d", repository.PullCount)
}
}
func TestRepositoryExists(t *testing.T) {
var exists bool
exists = RepositoryExists(currentRepository.Name)
if !exists {
t.Errorf("The repository with name: %s, does not exist", currentRepository.Name)
}
}
func TestDeleteRepository(t *testing.T) {
err := DeleteRepository(currentRepository.Name)
if err != nil {

42
src/common/dao/repository.go
View File

@ -69,54 +69,12 @@ func DeleteRepository(name string) error {
return err
}
// UpdateRepository ...
func UpdateRepository(repo models.RepoRecord) error {
o := GetOrmer()
repo.UpdateTime = time.Now()
_, err := o.Update(&repo)
return err
}
// IncreasePullCount ...
func IncreasePullCount(name string) (err error) {
o := GetOrmer()
num, err := o.QueryTable("repository").Filter("name", name).Update(
orm.Params{
"pull_count": orm.ColValue(orm.ColAdd, 1),
"update_time": time.Now(),
})
if err != nil {
return err
}
if num == 0 {
return fmt.Errorf("Failed to increase repository pull count with name: %s", name)
}
return nil
}
// RepositoryExists returns whether the repository exists according to its name.
func RepositoryExists(name string) bool {
o := GetOrmer()
return o.QueryTable("repository").Filter("name", name).Exist()
}
// GetTopRepos returns the most popular repositories whose project ID is
// in projectIDs
func GetTopRepos(projectIDs []int64, n int) ([]*models.RepoRecord, error) {
repositories := []*models.RepoRecord{}
if len(projectIDs) == 0 {
return repositories, nil
}
_, err := GetOrmer().QueryTable(&models.RepoRecord{}).
Filter("project_id__in", projectIDs).
OrderBy("-pull_count").
Limit(n).
All(&repositories)
return repositories, err
}
// GetTotalOfRepositories ...
func GetTotalOfRepositories(query ...*models.RepositoryQuery) (int64, error) {
sql, params := repositoryQueryConditions(query...)

61
src/common/dao/repository_test.go
View File

@ -15,7 +15,6 @@
package dao
import (
"fmt"
"testing"
"github.com/goharbor/harbor/src/common"
@ -122,66 +121,6 @@ func TestGetRepositories(t *testing.T) {
assert.Equal(t, name, repositories[0].Name)
}
func TestGetTopRepos(t *testing.T) {
var err error
require := require.New(t)
require.NoError(GetOrmer().Begin())
defer func() {
require.NoError(GetOrmer().Rollback())
}()
projectIDs := []int64{}
project1 := models.Project{
OwnerID: 1,
Name: "project1",
}
project1.ProjectID, err = AddProject(project1)
require.NoError(err)
projectIDs = append(projectIDs, project1.ProjectID)
project2 := models.Project{
OwnerID: 1,
Name: "project2",
}
project2.ProjectID, err = AddProject(project2)
require.NoError(err)
projectIDs = append(projectIDs, project2.ProjectID)
repository1 := &models.RepoRecord{
Name: fmt.Sprintf("%v/repository1", project1.Name),
ProjectID: project1.ProjectID,
}
err = AddRepository(*repository1)
require.NoError(err)
require.NoError(IncreasePullCount(repository1.Name))
repository2 := &models.RepoRecord{
Name: fmt.Sprintf("%v/repository2", project1.Name),
ProjectID: project1.ProjectID,
}
err = AddRepository(*repository2)
require.NoError(err)
require.NoError(IncreasePullCount(repository2.Name))
require.NoError(IncreasePullCount(repository2.Name))
repository3 := &models.RepoRecord{
Name: fmt.Sprintf("%v/repository3", project2.Name),
ProjectID: project2.ProjectID,
}
err = AddRepository(*repository3)
require.NoError(err)
require.NoError(IncreasePullCount(repository3.Name))
require.NoError(IncreasePullCount(repository3.Name))
require.NoError(IncreasePullCount(repository3.Name))
topRepos, err := GetTopRepos(projectIDs, 100)
require.NoError(err)
require.Len(topRepos, 3)
require.Equal(topRepos[0].Name, repository3.Name)
}
func addRepository(repository *models.RepoRecord) error {
return AddRepository(*repository)
}

52
src/common/models/retag.go
View File

@ -1,52 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package models
import (
"fmt"
"strings"
)
// RetagRequest gives the source image and target image of retag
type RetagRequest struct {
Tag string `json:"tag"` // The new tag
SrcImage string `json:"src_image"` // Source images in format <project>/<repo>:<reference>
Override bool `json:"override"` // If target tag exists, whether override it
}
// Image holds each part (project, repo, tag) of an image name
type Image struct {
Project string
Repo string
Tag string
}
// ParseImage parses an image name such as 'library/app:v1.0' to a structure with
// project, repo, and tag fields
func ParseImage(image string) (*Image, error) {
repo := strings.SplitN(image, "/", 2)
if len(repo) < 2 {
return nil, fmt.Errorf("unable to parse image from string: %s", image)
}
i := strings.SplitN(repo[1], ":", 2)
res := &Image{
Project: repo[0],
Repo: i[0],
}
if len(i) == 2 {
res.Tag = i[1]
}
return res, nil
}

75
src/common/models/retag_test.go
View File

@ -1,75 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package models
import (
"reflect"
"testing"
"github.com/stretchr/testify/assert"
)
func TestParseImage(t *testing.T) {
cases := []struct {
Input string
Expected *Image
Valid bool
}{
{
Input: "library/busybox",
Expected: &Image{
Project: "library",
Repo: "busybox",
Tag: "",
},
Valid: true,
},
{
Input: "library/busybox:v1.0",
Expected: &Image{
Project: "library",
Repo: "busybox",
Tag: "v1.0",
},
Valid: true,
},
{
Input: "library/busybox:sha256:9e2c9d5f44efbb6ee83aecd17a120c513047d289d142ec5738c9f02f9b24ad07",
Expected: &Image{
Project: "library",
Repo: "busybox",
Tag: "sha256:9e2c9d5f44efbb6ee83aecd17a120c513047d289d142ec5738c9f02f9b24ad07",
},
Valid: true,
},
{
Input: "busybox/v1.0",
Valid: false,
},
}
for _, c := range cases {
output, err := ParseImage(c.Input)
if c.Valid {
if !reflect.DeepEqual(output, c.Expected) {
assert.Equal(t, c.Expected, output)
}
} else {
if err != nil {
t.Errorf("failed to parse image %s", c.Input)
}
}
}
}

240
src/core/api/harborapi_test.go
View File

@ -113,21 +113,10 @@ func init() {
beego.Router("/api/projects/:id([0-9]+)/metadatas/", &MetadataAPI{}, "post:Post")
beego.Router("/api/projects/:id([0-9]+)/metadatas/:name", &MetadataAPI{}, "put:Put;delete:Delete")
beego.Router("/api/projects/:pid([0-9]+)/members/?:pmid([0-9]+)", &ProjectMemberAPI{})
beego.Router("/api/repositories", &RepositoryAPI{})
beego.Router("/api/statistics", &StatisticAPI{})
beego.Router("/api/users/?:id", &UserAPI{})
beego.Router("/api/usergroups/?:ugid([0-9]+)", &UserGroupAPI{})
beego.Router("/api/logs", &LogAPI{})
beego.Router("/api/repositories/*", &RepositoryAPI{}, "put:Put")
beego.Router("/api/repositories/*/labels", &RepositoryLabelAPI{}, "get:GetOfRepository;post:AddToRepository")
beego.Router("/api/repositories/*/labels/:id([0-9]+", &RepositoryLabelAPI{}, "delete:RemoveFromRepository")
beego.Router("/api/repositories/*/tags/:tag/labels", &RepositoryLabelAPI{}, "get:GetOfImage;post:AddToImage")
beego.Router("/api/repositories/*/tags/:tag/labels/:id([0-9]+", &RepositoryLabelAPI{}, "delete:RemoveFromImage")
beego.Router("/api/repositories/*/tags/:tag", &RepositoryAPI{}, "delete:Delete;get:GetTag")
beego.Router("/api/repositories/*/tags", &RepositoryAPI{}, "get:GetTags;post:Retag")
beego.Router("/api/repositories/*/tags/:tag/manifest", &RepositoryAPI{}, "get:GetManifests")
beego.Router("/api/repositories/*/signatures", &RepositoryAPI{}, "get:GetSignatures")
beego.Router("/api/repositories/top", &RepositoryAPI{}, "get:GetTopRepos")
beego.Router("/api/registries", &RegistryAPI{}, "get:List;post:Post")
beego.Router("/api/registries/ping", &RegistryAPI{}, "post:Ping")
beego.Router("/api/registries/:id([0-9]+)", &RegistryAPI{}, "get:Get;put:Put;delete:Delete")
@ -226,11 +215,6 @@ func init() {
beego.Router("/api/repositories/*/tags/:tag/scan", scanAPI, "post:Scan;get:Report")
beego.Router("/api/repositories/*/tags/:tag/scan/:uuid/log", scanAPI, "get:Log")
// syncRegistry
if err := SyncRegistry(config.GlobalProjectMgr); err != nil {
log.Fatalf("failed to sync repositories from registry: %v", err)
}
if err := quota.Sync(config.GlobalProjectMgr, false); err != nil {
log.Fatalf("failed to sync quota from backend: %v", err)
}
@ -357,48 +341,6 @@ func (a testapi) LogGet(user usrInfo) (int, []apilib.AccessLog, error) {
return code, successPayload, err
}
// // Delete a repository or a tag in a repository.
// // Delete a repository or a tag in a repository.
// // This endpoint let user delete repositories and tags with repo name and tag.\n
// // @param repoName The name of repository which will be deleted.
// // @param tag Tag of a repository.
// // @return void
// // func (a testapi) RepositoriesDelete(prjUsr UsrInfo, repoName string, tag string) (int, error) {
// func (a testapi) RepositoriesDelete(prjUsr UsrInfo, repoName string, tag string) (int, error) {
// _sling := sling.New().Delete(a.basePath)
// // create path and map variables
// path := "/api/repositories"
// _sling = _sling.Path(path)
// type QueryParams struct {
// RepoName string `url:"repo_name,omitempty"`
// Tag string `url:"tag,omitempty"`
// }
// _sling = _sling.QueryStruct(&QueryParams{RepoName: repoName, Tag: tag})
// // accept header
// accepts := []string{"application/json", "text/plain"}
// for key := range accepts {
// _sling = _sling.Set("Accept", accepts[key])
// break // only use the first Accept
// }
// req, err := _sling.Request()
// req.SetBasicAuth(prjUsr.Name, prjUsr.Passwd)
// // fmt.Printf("request %+v", req)
// client := &http.Client{}
// httpResponse, err := client.Do(req)
// defer httpResponse.Body.Close()
// if err != nil {
// // handle error
// }
// return httpResponse.StatusCode, err
// }
// Delete project by projectID
func (a testapi) ProjectsDelete(prjUsr usrInfo, projectID string) (int, error) {
_sling := sling.New().Delete(a.basePath)
@ -609,140 +551,6 @@ func (a testapi) PutProjectMember(authInfo usrInfo, projectID string, userID str
return httpStatusCode, err
}
// -------------------------Repositories Test---------------------------------------//
// Return relevant repos of projectID
func (a testapi) GetRepos(authInfo usrInfo, projectID, keyword string) (
int, interface{}, error) {
_sling := sling.New().Get(a.basePath)
path := "/api/repositories/"
_sling = _sling.Path(path)
type QueryParams struct {
ProjectID string `url:"project_id"`
Keyword string `url:"q"`
}
_sling = _sling.QueryStruct(&QueryParams{
ProjectID: projectID,
Keyword: keyword,
})
code, body, err := request(_sling, jsonAcceptHeader, authInfo)
if err != nil {
return 0, nil, err
}
if code == http.StatusOK {
repositories := []repoResp{}
if err = json.Unmarshal(body, &repositories); err != nil {
return 0, nil, err
}
return code, repositories, nil
}
return code, nil, nil
}
func (a testapi) GetTag(authInfo usrInfo, repository string, tag string) (int, *models.TagResp, error) {
_sling := sling.New().Get(a.basePath).Path(fmt.Sprintf("/api/repositories/%s/tags/%s", repository, tag))
code, data, err := request(_sling, jsonAcceptHeader, authInfo)
if err != nil {
return 0, nil, err
}
if code != http.StatusOK {
log.Printf("failed to get tag of %s:%s: %d %s \n", repository, tag, code, string(data))
return code, nil, nil
}
result := models.TagResp{}
if err := json.Unmarshal(data, &result); err != nil {
return 0, nil, err
}
return http.StatusOK, &result, nil
}
// Get tags of a relevant repository
func (a testapi) GetReposTags(authInfo usrInfo, repoName string) (int, interface{}, error) {
_sling := sling.New().Get(a.basePath)
path := fmt.Sprintf("/api/repositories/%s/tags", repoName)
_sling = _sling.Path(path)
httpStatusCode, body, err := request(_sling, jsonAcceptHeader, authInfo)
if err != nil {
return 0, nil, err
}
if httpStatusCode != http.StatusOK {
return httpStatusCode, body, nil
}
result := []models.TagResp{}
if err := json.Unmarshal(body, &result); err != nil {
return 0, nil, err
}
return http.StatusOK, result, nil
}
// RetagImage retag image to another tag
func (a testapi) RetagImage(authInfo usrInfo, repoName string, retag *apilib.Retag) (int, error) {
_sling := sling.New().Post(a.basePath)
path := fmt.Sprintf("/api/repositories/%s/tags", repoName)
_sling = _sling.Path(path)
_sling = _sling.BodyJSON(retag)
httpStatusCode, _, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, err
}
// Get manifests of a relevant repository
func (a testapi) GetReposManifests(authInfo usrInfo, repoName string, tag string) (int, error) {
_sling := sling.New().Get(a.basePath)
path := fmt.Sprintf("/api/repositories/%s/tags/%s/manifest", repoName, tag)
_sling = _sling.Path(path)
httpStatusCode, _, err := request(_sling, jsonAcceptHeader, authInfo)
return httpStatusCode, err
}
// Get public repositories which are accessed most
func (a testapi) GetReposTop(authInfo usrInfo, count string) (int, interface{}, error) {
_sling := sling.New().Get(a.basePath)
path := "/api/repositories/top"
_sling = _sling.Path(path)
type QueryParams struct {
Count string `url:"count"`
}
_sling = _sling.QueryStruct(&QueryParams{
Count: count,
})
code, body, err := request(_sling, jsonAcceptHeader, authInfo)
if err != nil {
return 0, nil, err
}
if code != http.StatusOK {
return code, body, err
}
result := []*repoResp{}
if err = json.Unmarshal(body, &result); err != nil {
return 0, nil, err
}
return http.StatusOK, result, nil
}
// --------------------Replication_Policy Test--------------------------------//
// Create a new replication policy
@ -836,54 +644,6 @@ func (a testapi) DeletePolicyByID(authInfo usrInfo, policyID string) (int, error
return httpStatusCode, err
}
// Return projects created by Harbor
// func (a HarborApi) ProjectsGet (projectName string, isPublic int32) ([]Project, error) {
// }
// Check if the project name user provided already exists.
// func (a HarborApi) ProjectsHead (projectName string) (error) {
// }
// Get access logs accompany with a relevant project.
// func (a HarborApi) ProjectsProjectIdLogsFilterPost (projectID int32, accessLog AccessLog) ([]AccessLog, error) {
// }
// Return a project&#39;s relevant role members.
// func (a HarborApi) ProjectsProjectIdMembersGet (projectID int32) ([]Role, error) {
// }
// Add project role member accompany with relevant project and user.
// func (a HarborApi) ProjectsProjectIdMembersPost (projectID int32, roles RoleParam) (error) {
// }
// Delete project role members accompany with relevant project and user.
// func (a HarborApi) ProjectsProjectIdMembersUserIdDelete (projectID int32, userId int32) (error) {
// }
// Return role members accompany with relevant project and user.
// func (a HarborApi) ProjectsProjectIdMembersUserIdGet (projectID int32, userId int32) ([]Role, error) {
// }
// Update project role members accompany with relevant project and user.
// func (a HarborApi) ProjectsProjectIdMembersUserIdPut (projectID int32, userId int32, roles RoleParam) (error) {
// }
// Update properties for a selected project.
// func (a HarborApi) ProjectsProjectIdPut (projectID int32, project Project) (error) {
// }
// Get repositories accompany with relevant project and repo name.
// func (a HarborApi) RepositoriesGet (projectID int32, q string) ([]Repository, error) {
// }
// Get manifests of a relevant repository.
// func (a HarborApi) RepositoriesManifestGet (repoName string, tag string) (error) {
// }
// Get tags of a relevant repository.
// func (a HarborApi) RepositoriesTagsGet (repoName string) (error) {
// }
// Get registered users of Harbor.
func (a testapi) UsersGet(userName string, authInfo usrInfo) (int, []apilib.User, error) {
_sling := sling.New().Get(a.basePath)

9
src/core/api/internal.go
View File

@ -47,15 +47,6 @@ func (ia *InternalAPI) Prepare() {
}
}
// SyncRegistry ...
func (ia *InternalAPI) SyncRegistry() {
err := SyncRegistry(ia.ProjectMgr)
if err != nil {
ia.SendInternalServerError(err)
return
}
}
// RenameAdmin we don't provide flexibility in this API, as this is a workaround.
func (ia *InternalAPI) RenameAdmin() {
if !dao.IsSuperUser(ia.SecurityCtx.GetUsername()) {

1100
src/core/api/repository.go
View File

File diff suppressed because it is too large Load Diff

193
src/core/api/repository_label.go
View File

@ -1,193 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"errors"
"fmt"
"net/http"
"github.com/goharbor/harbor/src/common"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/rbac"
coreutils "github.com/goharbor/harbor/src/core/utils"
)
// RepositoryLabelAPI handles requests for adding/removing label to/from repositories and images
type RepositoryLabelAPI struct {
LabelResourceAPI
repository *models.RepoRecord
tag string
label *models.Label
}
// Prepare ...
func (r *RepositoryLabelAPI) Prepare() {
// Super
r.LabelResourceAPI.Prepare()
if !r.SecurityCtx.IsAuthenticated() {
r.SendUnAuthorizedError(errors.New("UnAuthorized"))
return
}
repository := r.GetString(":splat")
repo, err := dao.GetRepositoryByName(repository)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get repository %s: %v", repository, err))
return
}
if repo == nil {
r.SendNotFoundError(fmt.Errorf("repository %s not found", repository))
return
}
r.repository = repo
tag := r.GetString(":tag")
if len(tag) > 0 {
exist, err := imageExist(r.SecurityCtx.GetUsername(), repository, tag)
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to check the existence of image %s:%s: %v", repository, tag, err))
return
}
if !exist {
r.SendNotFoundError(fmt.Errorf("image %s:%s not found", repository, tag))
return
}
r.tag = tag
}
if r.Ctx.Request.Method == http.MethodDelete {
labelID, err := r.GetInt64FromPath(":id")
if err != nil {
r.SendInternalServerError(fmt.Errorf("failed to get ID parameter from path: %v", err))
return
}
label, ok := r.exists(labelID)
if !ok {
return
}
r.label = label
}
}
func (r *RepositoryLabelAPI) requireAccess(action rbac.Action, subresource ...rbac.Resource) bool {
if len(subresource) == 0 {
subresource = append(subresource, rbac.ResourceRepositoryLabel)
}
return r.RequireProjectAccess(r.repository.ProjectID, action, subresource...)
}
func (r *RepositoryLabelAPI) isValidLabelReq() bool {
p, err := r.ProjectMgr.Get(r.repository.ProjectID)
if err != nil {
r.SendInternalServerError(err)
return false
}
l := &models.Label{}
if err := r.DecodeJSONReq(l); err != nil {
r.SendBadRequestError(err)
return false
}
label, ok := r.validate(l.ID, p.ProjectID)
if !ok {
return false
}
r.label = label
return true
}
// GetOfImage returns labels of an image
func (r *RepositoryLabelAPI) GetOfImage() {
if !r.requireAccess(rbac.ActionList, rbac.ResourceRepositoryTagLabel) {
return
}
r.getLabelsOfResource(common.ResourceTypeImage, fmt.Sprintf("%s:%s", r.repository.Name, r.tag))
}
// AddToImage adds the label to an image
func (r *RepositoryLabelAPI) AddToImage() {
if !r.requireAccess(rbac.ActionCreate, rbac.ResourceRepositoryTagLabel) || !r.isValidLabelReq() {
return
}
rl := &models.ResourceLabel{
LabelID: r.label.ID,
ResourceType: common.ResourceTypeImage,
ResourceName: fmt.Sprintf("%s:%s", r.repository.Name, r.tag),
}
r.markLabelToResource(rl)
}
// RemoveFromImage removes the label from an image
func (r *RepositoryLabelAPI) RemoveFromImage() {
if !r.requireAccess(rbac.ActionDelete, rbac.ResourceRepositoryTagLabel) {
return
}
r.removeLabelFromResource(common.ResourceTypeImage,
fmt.Sprintf("%s:%s", r.repository.Name, r.tag), r.label.ID)
}
// GetOfRepository returns labels of a repository
func (r *RepositoryLabelAPI) GetOfRepository() {
if !r.requireAccess(rbac.ActionList) {
return
}
r.getLabelsOfResource(common.ResourceTypeRepository, r.repository.RepositoryID)
}
// AddToRepository adds the label to a repository
func (r *RepositoryLabelAPI) AddToRepository() {
if !r.requireAccess(rbac.ActionCreate) || !r.isValidLabelReq() {
return
}
rl := &models.ResourceLabel{
LabelID: r.label.ID,
ResourceType: common.ResourceTypeRepository,
ResourceID: r.repository.RepositoryID,
}
r.markLabelToResource(rl)
}
// RemoveFromRepository removes the label from a repository
func (r *RepositoryLabelAPI) RemoveFromRepository() {
if !r.requireAccess(rbac.ActionDelete) {
return
}
r.removeLabelFromResource(common.ResourceTypeRepository, r.repository.RepositoryID, r.label.ID)
}
func imageExist(username, repository, tag string) (bool, error) {
client, err := coreutils.NewRepositoryClientForUI(username, repository)
if err != nil {
return false, err
}
_, exist, err := client.ManifestExist(tag)
return exist, err
}

255
src/core/api/repository_label_test.go
View File

@ -1,255 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"fmt"
"net/http"
"testing"
"github.com/goharbor/harbor/src/common"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
var (
resourceLabelAPIBasePath = "/api/repositories"
repo = "library/hello-world"
tag = "latest"
proLibraryLabelID int64
)
func TestAddToImage(t *testing.T) {
sysLevelLabelID, err := dao.AddLabel(&models.Label{
Name: "sys_level_label",
Level: common.LabelLevelSystem,
})
require.Nil(t, err)
defer dao.DeleteLabel(sysLevelLabelID)
proTestLabelID, err := dao.AddLabel(&models.Label{
Name: "pro_test_label",
Level: common.LabelLevelUser,
Scope: common.LabelScopeProject,
ProjectID: 100,
})
require.Nil(t, err)
defer dao.DeleteLabel(proTestLabelID)
proLibraryLabelID, err = dao.AddLabel(&models.Label{
Name: "pro_library_label",
Level: common.LabelLevelUser,
Scope: common.LabelScopeProject,
ProjectID: 1,
})
require.Nil(t, err)
cases := []*codeCheckingCase{
// 401
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath,
repo, tag),
method: http.MethodPost,
},
code: http.StatusUnauthorized,
},
// 403
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath,
repo, tag),
method: http.MethodPost,
credential: projGuest,
},
code: http.StatusForbidden,
},
// 404 repo doesn't exist
{
request: &testingRequest{
url: fmt.Sprintf("%s/library/non-exist-repo/tags/%s/labels", resourceLabelAPIBasePath, tag),
method: http.MethodPost,
credential: projDeveloper,
},
code: http.StatusNotFound,
},
// 404 image doesn't exist
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/non-exist-tag/labels", resourceLabelAPIBasePath, repo),
method: http.MethodPost,
credential: projDeveloper,
},
code: http.StatusNotFound,
},
// 400
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath, repo, tag),
method: http.MethodPost,
credential: projDeveloper,
},
code: http.StatusBadRequest,
},
// 404 label doesn't exist
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath,
repo, tag),
method: http.MethodPost,
credential: projDeveloper,
bodyJSON: struct {
ID int64
}{
ID: 1000,
},
},
code: http.StatusNotFound,
},
// 400 system level label
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath,
repo, tag),
method: http.MethodPost,
credential: projDeveloper,
bodyJSON: struct {
ID int64
}{
ID: sysLevelLabelID,
},
},
code: http.StatusBadRequest,
},
// 400 try to add the label of project1 to the image under project2
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath,
repo, tag),
method: http.MethodPost,
credential: projDeveloper,
bodyJSON: struct {
ID int64
}{
ID: proTestLabelID,
},
},
code: http.StatusBadRequest,
},
// 200
{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath,
repo, tag),
method: http.MethodPost,
credential: projDeveloper,
bodyJSON: struct {
ID int64
}{
ID: proLibraryLabelID,
},
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
}
func TestGetOfImage(t *testing.T) {
labels := []*models.Label{}
err := handleAndParse(&testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath, repo, tag),
method: http.MethodGet,
credential: projDeveloper,
}, &labels)
require.Nil(t, err)
require.Equal(t, 1, len(labels))
assert.Equal(t, proLibraryLabelID, labels[0].ID)
}
func TestRemoveFromImage(t *testing.T) {
runCodeCheckingCases(t, &codeCheckingCase{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels/%d", resourceLabelAPIBasePath,
repo, tag, proLibraryLabelID),
method: http.MethodDelete,
credential: projDeveloper,
},
code: http.StatusOK,
})
labels := []*models.Label{}
err := handleAndParse(&testingRequest{
url: fmt.Sprintf("%s/%s/tags/%s/labels", resourceLabelAPIBasePath,
repo, tag),
method: http.MethodGet,
credential: projDeveloper,
}, &labels)
require.Nil(t, err)
require.Equal(t, 0, len(labels))
}
func TestAddToRepository(t *testing.T) {
runCodeCheckingCases(t, &codeCheckingCase{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/labels", resourceLabelAPIBasePath, repo),
method: http.MethodPost,
bodyJSON: struct {
ID int64
}{
ID: proLibraryLabelID,
},
credential: projDeveloper,
},
code: http.StatusOK,
})
}
func TestGetOfRepository(t *testing.T) {
labels := []*models.Label{}
err := handleAndParse(&testingRequest{
url: fmt.Sprintf("%s/%s/labels", resourceLabelAPIBasePath, repo),
method: http.MethodGet,
credential: projDeveloper,
}, &labels)
require.Nil(t, err)
require.Equal(t, 1, len(labels))
assert.Equal(t, proLibraryLabelID, labels[0].ID)
}
func TestRemoveFromRepository(t *testing.T) {
runCodeCheckingCases(t, &codeCheckingCase{
request: &testingRequest{
url: fmt.Sprintf("%s/%s/labels/%d", resourceLabelAPIBasePath,
repo, proLibraryLabelID),
method: http.MethodDelete,
credential: projDeveloper,
},
code: http.StatusOK,
})
labels := []*models.Label{}
err := handleAndParse(&testingRequest{
url: fmt.Sprintf("%s/%s/labels", resourceLabelAPIBasePath, repo),
method: http.MethodGet,
credential: projDeveloper,
}, &labels)
require.Nil(t, err)
require.Equal(t, 0, len(labels))
dao.DeleteLabel(proLibraryLabelID)
}

452
src/core/api/repository_test.go
View File

@ -1,452 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package api
import (
"fmt"
"net/http"
"testing"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/testing/apitests/apilib"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestGetRepos(t *testing.T) {
assert := assert.New(t)
apiTest := newHarborAPI()
projectID := "1"
keyword := "library/hello-world"
fmt.Println("Testing Repos Get API")
// -------------------case 1 : response code = 200------------------------//
fmt.Println("case 1 : response code = 200")
code, repositories, err := apiTest.GetRepos(*admin, projectID, keyword)
if err != nil {
t.Errorf("failed to get repositories: %v", err)
} else {
assert.Equal(int(200), code, "response code should be 200")
if repos, ok := repositories.([]repoResp); ok {
require.Equal(t, int(1), len(repos), "the length of repositories should be 1")
assert.Equal(repos[0].Name, "library/hello-world", "unexpected repository name")
} else {
t.Error("unexpected response")
}
}
// -------------------case 2 : response code = 404------------------------//
fmt.Println("case 2 : response code = 404:project not found")
projectID = "111"
httpStatusCode, _, err := apiTest.GetRepos(*admin, projectID, keyword)
if err != nil {
t.Error("Error whihle get repos by projectID", err.Error())
t.Log(err)
} else {
assert.Equal(int(404), httpStatusCode, "httpStatusCode should be 404")
}
// -------------------case 3 : response code = 400------------------------//
fmt.Println("case 3 : response code = 400,invalid project_id")
projectID = "ccc"
httpStatusCode, _, err = apiTest.GetRepos(*admin, projectID, keyword)
if err != nil {
t.Error("Error whihle get repos by projectID", err.Error())
t.Log(err)
} else {
assert.Equal(int(400), httpStatusCode, "httpStatusCode should be 400")
}
fmt.Printf("\n")
}
func TestGetReposTags(t *testing.T) {
assert := assert.New(t)
apiTest := newHarborAPI()
// -------------------case 1 : response code = 404------------------------//
fmt.Println("case 1 : response code = 404,repo not found")
repository := "errorRepos"
code, _, err := apiTest.GetReposTags(*admin, repository)
if err != nil {
t.Errorf("failed to get tags of repository %s: %v", repository, err)
} else {
assert.Equal(int(404), code, "httpStatusCode should be 404")
}
// -------------------case 2 : response code = 200------------------------//
fmt.Println("case 2 : response code = 200")
repository = "library/hello-world"
code, tags, err := apiTest.GetReposTags(*admin, repository)
if err != nil {
t.Errorf("failed to get tags of repository %s: %v", repository, err)
} else {
assert.Equal(int(200), code, "httpStatusCode should be 200")
if tg, ok := tags.([]models.TagResp); ok {
assert.Equal(1, len(tg), fmt.Sprintf("there should be only one tag, but now %v", tg))
assert.Equal(tg[0].Name, "latest", "the tag should be latest")
} else {
t.Error("unexpected response")
}
}
// -------------------case 3 : response code = 404------------------------//
fmt.Println("case 3 : response code = 404")
repository = "library/hello-world"
tag := "not_exist_tag"
code, result, err := apiTest.GetTag(*admin, repository, tag)
assert.Nil(err)
assert.Equal(http.StatusNotFound, code)
// -------------------case 4 : response code = 200------------------------//
fmt.Println("case 4 : response code = 200")
repository = "library/hello-world"
tag = "latest"
code, result, err = apiTest.GetTag(*admin, repository, tag)
assert.Nil(err)
assert.Equal(http.StatusOK, code)
assert.Equal(tag, result.Name)
fmt.Printf("\n")
}
func TestGetReposManifests(t *testing.T) {
var httpStatusCode int
var err error
var repoName string
var tag string
assert := assert.New(t)
apiTest := newHarborAPI()
fmt.Println("Testing ReposManifests Get API")
// -------------------case 1 : response code = 200------------------------//
fmt.Println("case 1 : response code = 200")
repoName = "library/hello-world"
tag = "latest"
httpStatusCode, err = apiTest.GetReposManifests(*admin, repoName, tag)
if err != nil {
t.Error("Error whihle get reposManifests by repoName and tag", err.Error())
t.Log(err)
} else {
assert.Equal(int(200), httpStatusCode, "httpStatusCode should be 200")
}
// -------------------case 2 : response code = 404------------------------//
fmt.Println("case 2 : response code = 404:tags error,manifest unknown")
tag = "l"
httpStatusCode, err = apiTest.GetReposManifests(*admin, repoName, tag)
if err != nil {
t.Error("Error whihle get reposManifests by repoName and tag", err.Error())
t.Log(err)
} else {
assert.Equal(int(404), httpStatusCode, "httpStatusCode should be 404")
}
// -------------------case 3 : response code = 404------------------------//
fmt.Println("case 3 : response code = 404,repo not found")
repoName = "111"
httpStatusCode, err = apiTest.GetReposManifests(*admin, repoName, tag)
if err != nil {
t.Error("Error whihle get reposManifests by repoName and tag", err.Error())
t.Log(err)
} else {
assert.Equal(int(404), httpStatusCode, "httpStatusCode should be 404")
}
fmt.Printf("\n")
}
func TestGetReposTop(t *testing.T) {
assert := assert.New(t)
apiTest := newHarborAPI()
fmt.Println("Testing ReposTop Get API")
// -------------------case 1 : response code = 400------------------------//
fmt.Println("case 1 : response code = 400,invalid count")
count := "cc"
code, _, err := apiTest.GetReposTop(*admin, count)
if err != nil {
t.Errorf("failed to get the most popular repositories: %v", err)
} else {
assert.Equal(int(400), code, "response code should be 400")
}
// -------------------case 2 : response code = 200------------------------//
fmt.Println("case 2 : response code = 200")
count = "1"
code, repos, err := apiTest.GetReposTop(*admin, count)
if err != nil {
t.Errorf("failed to get the most popular repositories: %v", err)
} else {
assert.Equal(int(200), code, "response code should be 200")
if r, ok := repos.([]*repoResp); ok {
assert.Equal(int(1), len(r), "the length should be 1")
assert.Equal(r[0].Name, "library/busybox", "the name of repository should be library/busybox")
} else {
t.Error("unexpected response")
}
}
fmt.Printf("\n")
}
func TestPopulateAuthor(t *testing.T) {
author := "author"
detail := &models.TagDetail{
Author: author,
}
populateAuthor(detail)
assert.Equal(t, author, detail.Author)
detail = &models.TagDetail{}
populateAuthor(detail)
assert.Equal(t, "", detail.Author)
maintainer := "maintainer"
detail = &models.TagDetail{
Config: &models.TagCfg{
Labels: map[string]string{
"Maintainer": maintainer,
},
},
}
populateAuthor(detail)
assert.Equal(t, maintainer, detail.Author)
}
func TestPutOfRepository(t *testing.T) {
base := "/api/repositories/"
desc := struct {
Description string `json:"description"`
}{
Description: "description_for_test",
}
cases := []*codeCheckingCase{
// 404
{
request: &testingRequest{
method: http.MethodPut,
url: base + "non_exist_repository",
bodyJSON: desc,
},
code: http.StatusNotFound,
},
// 401
{
request: &testingRequest{
method: http.MethodPut,
url: base + "library/hello-world",
bodyJSON: desc,
},
code: http.StatusUnauthorized,
},
// 403 non-member
{
request: &testingRequest{
method: http.MethodPut,
url: base + "library/hello-world",
bodyJSON: desc,
credential: nonSysAdmin,
},
code: http.StatusForbidden,
},
// 403 project guest
{
request: &testingRequest{
method: http.MethodPut,
url: base + "library/hello-world",
bodyJSON: desc,
credential: projGuest,
},
code: http.StatusForbidden,
},
// 200 project developer
{
request: &testingRequest{
method: http.MethodPut,
url: base + "library/hello-world",
bodyJSON: desc,
credential: projDeveloper,
},
code: http.StatusOK,
},
// 200 project admin
{
request: &testingRequest{
method: http.MethodPut,
url: base + "library/hello-world",
bodyJSON: desc,
credential: projAdmin,
},
code: http.StatusOK,
},
// 200 system admin
{
request: &testingRequest{
method: http.MethodPut,
url: base + "library/hello-world",
bodyJSON: desc,
credential: sysAdmin,
},
code: http.StatusOK,
},
}
runCodeCheckingCases(t, cases...)
// verify that the description is changed
repositories := []*repoResp{}
err := handleAndParse(&testingRequest{
method: http.MethodGet,
url: base,
queryStruct: struct {
ProjectID int64 `url:"project_id"`
}{
ProjectID: 1,
},
}, &repositories)
require.Nil(t, err)
var repository *repoResp
for _, repo := range repositories {
if repo.Name == "library/hello-world" {
repository = repo
break
}
}
require.NotNil(t, repository)
assert.Equal(t, desc.Description, repository.Description)
}
func TestRetag(t *testing.T) {
assert := assert.New(t)
apiTest := newHarborAPI()
repo := "library/hello-world"
fmt.Println("Testing Image Retag API")
// -------------------case 1 : response code = 200------------------------//
fmt.Println("case 1 : response code = 200")
retagReq := &apilib.Retag{
Tag: "prd",
SrcImage: "library/hello-world:latest",
Override: true,
}
code, err := apiTest.RetagImage(*admin, repo, retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(200), code, "response code should be 200")
}
// -------------------case 2 : response code = 400------------------------//
fmt.Println("case 2 : response code = 400: invalid image value provided")
retagReq = &apilib.Retag{
Tag: "prd",
SrcImage: "hello-world:latest",
Override: true,
}
httpStatusCode, err := apiTest.RetagImage(*admin, repo, retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(400), httpStatusCode, "httpStatusCode should be 400")
}
// -------------------case 3 : response code = 404------------------------//
fmt.Println("case 3 : response code = 404: source image not exist")
retagReq = &apilib.Retag{
Tag: "prd",
SrcImage: "release/hello-world:notexist",
Override: true,
}
httpStatusCode, err = apiTest.RetagImage(*admin, repo, retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(404), httpStatusCode, "httpStatusCode should be 404")
}
// -------------------case 4 : response code = 404------------------------//
fmt.Println("case 4 : response code = 404: target project not exist")
retagReq = &apilib.Retag{
Tag: "prd",
SrcImage: "library/hello-world:latest",
Override: true,
}
httpStatusCode, err = apiTest.RetagImage(*admin, "nonexist/hello-world", retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(404), httpStatusCode, "httpStatusCode should be 404")
}
// -------------------case 5 : response code = 401------------------------//
fmt.Println("case 5 : response code = 401, unathorized")
retagReq = &apilib.Retag{
Tag: "prd",
SrcImage: "library/hello-world:latest",
Override: true,
}
httpStatusCode, err = apiTest.RetagImage(*unknownUsr, repo, retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(401), httpStatusCode, "httpStatusCode should be 401")
}
// -------------------case 6 : response code = 409------------------------//
fmt.Println("case 6 : response code = 409, conflict")
retagReq = &apilib.Retag{
Tag: "latest",
SrcImage: "library/hello-world:latest",
Override: false,
}
httpStatusCode, err = apiTest.RetagImage(*admin, repo, retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(409), httpStatusCode, "httpStatusCode should be 409")
}
// -------------------case 7 : response code = 400------------------------//
fmt.Println("case 7 : response code = 400")
retagReq = &apilib.Retag{
Tag: ".0.1",
SrcImage: "library/hello-world:latest",
Override: true,
}
code, err = apiTest.RetagImage(*admin, repo, retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(400), code, "response code should be 400")
}
// -------------------case 8 : response code = 400------------------------//
fmt.Println("case 8 : response code = 400")
retagReq = &apilib.Retag{
Tag: "v0.1",
SrcImage: "library/hello-world:latest",
Override: true,
}
code, err = apiTest.RetagImage(*admin, "library/Aaaa", retagReq)
if err != nil {
t.Errorf("failed to retag: %v", err)
} else {
assert.Equal(int(400), code, "response code should be 400")
}
fmt.Printf("\n")
}

217
src/core/api/utils.go
View File

@ -15,95 +15,15 @@
package api
import (
"fmt"
"net/http"
"sort"
"strings"
"github.com/goharbor/harbor/src/common/dao"
commonhttp "github.com/goharbor/harbor/src/common/http"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/common/utils/registry"
"github.com/goharbor/harbor/src/common/utils/registry/auth"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/promgr"
coreutils "github.com/goharbor/harbor/src/core/utils"
)
// SyncRegistry syncs the repositories of registry with database.
func SyncRegistry(pm promgr.ProjectManager) error {
log.Infof("Start syncing repositories from registry to DB... ")
reposInRegistry, err := Catalog()
if err != nil {
log.Error(err)
return err
}
var repoRecordsInDB []*models.RepoRecord
repoRecordsInDB, err = dao.GetRepositories()
if err != nil {
log.Errorf("error occurred while getting all registories. %v", err)
return err
}
var reposInDB []string
for _, repoRecordInDB := range repoRecordsInDB {
reposInDB = append(reposInDB, repoRecordInDB.Name)
}
var reposToAdd []string
var reposToDel []string
reposToAdd, reposToDel, err = diffRepos(reposInRegistry, reposInDB, pm)
if err != nil {
return err
}
if len(reposToAdd) > 0 {
log.Infof("Start adding repositories into DB %v ... ", len(reposToAdd))
for _, repoToAdd := range reposToAdd {
project, _ := utils.ParseRepository(repoToAdd)
pullCount, err := dao.CountPull(repoToAdd)
if err != nil {
log.Errorf("Error happens when counting pull count from access log: %v", err)
}
pro, err := pm.Get(project)
if err != nil {
log.Errorf("failed to get project %s: %v", project, err)
continue
}
repoRecord := models.RepoRecord{
Name: repoToAdd,
ProjectID: pro.ProjectID,
PullCount: pullCount,
}
if err := dao.AddRepository(repoRecord); err != nil {
log.Errorf("Error happens when adding the missing repository: %v", err)
} else {
log.Infof("Add repository: %s success.", repoToAdd)
}
}
}
if len(reposToDel) > 0 {
log.Debugf("Start deleting repositories from DB... ")
for _, repoToDel := range reposToDel {
if err := dao.DeleteRepository(repoToDel); err != nil {
log.Errorf("Error happens when deleting the repository: %v", err)
} else {
log.Debugf("Delete repository: %s success.", repoToDel)
}
}
}
log.Infof("Sync repositories from registry to DB is done.")
return nil
}
// Catalog ...
func Catalog() ([]string, error) {
repositories := []string{}
@ -121,117 +41,6 @@ func Catalog() ([]string, error) {
return repositories, nil
}
func diffRepos(reposInRegistry []string, reposInDB []string,
pm promgr.ProjectManager) ([]string, []string, error) {
var needsAdd []string
var needsDel []string
sort.Strings(reposInRegistry)
sort.Strings(reposInDB)
i, j := 0, 0
repoInR, repoInD := "", ""
for i < len(reposInRegistry) && j < len(reposInDB) {
repoInR = reposInRegistry[i]
repoInD = reposInDB[j]
d := strings.Compare(repoInR, repoInD)
if d < 0 {
i++
exist, err := projectExists(pm, repoInR)
if err != nil {
log.Errorf("failed to check the existence of project %s: %v", repoInR, err)
continue
}
if !exist {
continue
}
// TODO remove the workaround when the bug of registry is fixed
client, err := coreutils.NewRepositoryClientForUI("harbor-core", repoInR)
if err != nil {
return needsAdd, needsDel, err
}
exist, err = repositoryExist(repoInR, client)
if err != nil {
return needsAdd, needsDel, err
}
if !exist {
continue
}
needsAdd = append(needsAdd, repoInR)
} else if d > 0 {
needsDel = append(needsDel, repoInD)
j++
} else {
// TODO remove the workaround when the bug of registry is fixed
client, err := coreutils.NewRepositoryClientForUI("harbor-core", repoInR)
if err != nil {
return needsAdd, needsDel, err
}
exist, err := repositoryExist(repoInR, client)
if err != nil {
return needsAdd, needsDel, err
}
if !exist {
needsDel = append(needsDel, repoInD)
}
i++
j++
}
}
for i < len(reposInRegistry) {
repoInR = reposInRegistry[i]
i++
exist, err := projectExists(pm, repoInR)
if err != nil {
log.Errorf("failed to check whether project of %s exists: %v", repoInR, err)
continue
}
if !exist {
continue
}
client, err := coreutils.NewRepositoryClientForUI("harbor-core", repoInR)
if err != nil {
log.Errorf("failed to create repository client: %v", err)
continue
}
exist, err = repositoryExist(repoInR, client)
if err != nil {
log.Errorf("failed to check the existence of repository %s: %v", repoInR, err)
continue
}
if !exist {
continue
}
needsAdd = append(needsAdd, repoInR)
}
for j < len(reposInDB) {
needsDel = append(needsDel, reposInDB[j])
j++
}
return needsAdd, needsDel, nil
}
func projectExists(pm promgr.ProjectManager, repository string) (bool, error) {
project, _ := utils.ParseRepository(repository)
return pm.Exists(project)
}
func initRegistryClient() (r *registry.Registry, err error) {
endpoint, err := config.RegistryURL()
if err != nil {
@ -252,29 +61,3 @@ func initRegistryClient() (r *registry.Registry, err error) {
Transport: registry.NewTransport(registry.GetHTTPTransport(), authorizer),
})
}
func buildReplicationURL() string {
url := config.InternalJobServiceURL()
return fmt.Sprintf("%s/api/jobs/replication", url)
}
func buildJobLogURL(jobID string, jobType string) string {
url := config.InternalJobServiceURL()
return fmt.Sprintf("%s/api/jobs/%s/%s/log", url, jobType, jobID)
}
func buildReplicationActionURL() string {
url := config.InternalJobServiceURL()
return fmt.Sprintf("%s/api/jobs/replication/actions", url)
}
func repositoryExist(name string, client *registry.Repository) (bool, error) {
tags, err := client.ListTag()
if err != nil {
if regErr, ok := err.(*commonhttp.Error); ok && regErr.Code == http.StatusNotFound {
return false, nil
}
return false, err
}
return len(tags) != 0, nil
}

3
src/core/controllers/controllers_test.go
View File

@ -30,7 +30,6 @@ import (
utilstest "github.com/goharbor/harbor/src/common/utils/test"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/filter"
"github.com/goharbor/harbor/src/core/middlewares"
"github.com/stretchr/testify/assert"
)
@ -101,8 +100,6 @@ func TestRedirectForOIDC(t *testing.T) {
func TestAll(t *testing.T) {
config.InitWithSettings(utilstest.GetUnitTestConfig())
assert := assert.New(t)
err := middlewares.Init()
assert.Nil(err)
r, _ := http.NewRequest("POST", "/c/login", nil)
w := httptest.NewRecorder()

20
src/core/main.go
View File

@ -229,26 +229,6 @@ func main() {
server.RegisterRoutes()
syncRegistry := os.Getenv("SYNC_REGISTRY")
sync, err := strconv.ParseBool(syncRegistry)
if err != nil {
log.Errorf("Failed to parse SYNC_REGISTRY: %v", err)
// if err set it default to false
sync = false
}
if sync {
if err := api.SyncRegistry(config.GlobalProjectMgr); err != nil {
log.Error(err)
}
} else {
log.Infof("Because SYNC_REGISTRY set false , no need to sync registry \n")
}
log.Info("Init proxy")
if err := middlewares.Init(); err != nil {
log.Fatalf("init proxy error, %v", err)
}
syncQuota := os.Getenv("SYNC_QUOTA")
doSyncQuota, err := strconv.ParseBool(syncQuota)
if err != nil {

20
src/core/middlewares/chain.go
View File

@ -19,15 +19,6 @@ import (
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/chart"
"github.com/goharbor/harbor/src/core/middlewares/contenttrust"
"github.com/goharbor/harbor/src/core/middlewares/countquota"
"github.com/goharbor/harbor/src/core/middlewares/immutable"
"github.com/goharbor/harbor/src/core/middlewares/listrepo"
"github.com/goharbor/harbor/src/core/middlewares/readonly"
"github.com/goharbor/harbor/src/core/middlewares/regtoken"
"github.com/goharbor/harbor/src/core/middlewares/sizequota"
"github.com/goharbor/harbor/src/core/middlewares/url"
"github.com/goharbor/harbor/src/core/middlewares/vulnerable"
"github.com/justinas/alice"
)
@ -62,16 +53,7 @@ func (b *DefaultCreator) Create() *alice.Chain {
func (b *DefaultCreator) geMiddleware(mName string) alice.Constructor {
middlewares := map[string]alice.Constructor{
CHART: func(next http.Handler) http.Handler { return chart.New(next) },
READONLY: func(next http.Handler) http.Handler { return readonly.New(next) },
URL: func(next http.Handler) http.Handler { return url.New(next) },
LISTREPO: func(next http.Handler) http.Handler { return listrepo.New(next) },
CONTENTTRUST: func(next http.Handler) http.Handler { return contenttrust.New(next) },
VULNERABLE: func(next http.Handler) http.Handler { return vulnerable.New(next) },
SIZEQUOTA: func(next http.Handler) http.Handler { return sizequota.New(next) },
COUNTQUOTA: func(next http.Handler) http.Handler { return countquota.New(next) },
IMMUTABLE: func(next http.Handler) http.Handler { return immutable.New(next) },
REGTOKEN: func(next http.Handler) http.Handler { return regtoken.New(next) },
CHART: func(next http.Handler) http.Handler { return chart.New(next) },
}
return middlewares[mName]
}

17
src/core/middlewares/config.go
View File

@ -16,23 +16,8 @@ package middlewares
// const variables
const (
CHART = "chart"
READONLY = "readonly"
URL = "url"
LISTREPO = "listrepo"
CONTENTTRUST = "contenttrust"
VULNERABLE = "vulnerable"
SIZEQUOTA = "sizequota"
COUNTQUOTA = "countquota"
IMMUTABLE = "immutable"
REGTOKEN = "regtoken"
CHART = "chart"
)
// ChartMiddlewares middlewares for chart server
var ChartMiddlewares = []string{CHART}
// Middlewares with sequential organization
var Middlewares = []string{READONLY, URL, REGTOKEN, LISTREPO, CONTENTTRUST, VULNERABLE, SIZEQUOTA, IMMUTABLE, COUNTQUOTA}
// MiddlewaresLocal ...
var MiddlewaresLocal = []string{SIZEQUOTA, IMMUTABLE, COUNTQUOTA}

116
src/core/middlewares/contenttrust/handler.go
View File

@ -1,116 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package contenttrust
import (
"github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/goharbor/harbor/src/pkg/signature/notary"
"net/http"
"net/http/httptest"
)
// NotaryEndpoint ...
var NotaryEndpoint = ""
type contentTrustHandler struct {
next http.Handler
}
// New ...
func New(next http.Handler) http.Handler {
return &contentTrustHandler{
next: next,
}
}
// ServeHTTP ...
func (cth contentTrustHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
doContentTrustCheck, image := validate(req)
if !doContentTrustCheck {
cth.next.ServeHTTP(rw, req)
return
}
rec := httptest.NewRecorder()
cth.next.ServeHTTP(rec, req)
if rec.Result().StatusCode == http.StatusOK {
match, err := matchNotaryDigest(image)
if err != nil {
http.Error(rw, util.MarshalError("PROJECT_POLICY_VIOLATION", "Failed in communication with Notary please check the log"), http.StatusInternalServerError)
return
}
if !match {
log.Debugf("digest mismatch, failing the response.")
http.Error(rw, util.MarshalError("PROJECT_POLICY_VIOLATION", "The image is not signed in Notary."), http.StatusPreconditionFailed)
return
}
}
util.CopyResp(rec, rw)
}
func validate(req *http.Request) (bool, util.ArtifactInfo) {
var img util.ArtifactInfo
imgRaw := req.Context().Value(util.ArtifactInfoCtxKey)
if imgRaw == nil || !config.WithNotary() {
return false, img
}
img, _ = req.Context().Value(util.ArtifactInfoCtxKey).(util.ArtifactInfo)
if img.Digest == "" {
return false, img
}
if scannerPull, ok := util.ScannerPullFromContext(req.Context()); ok && scannerPull {
return false, img
}
if !util.GetPolicyChecker().ContentTrustEnabled(img.ProjectName) {
return false, img
}
return true, img
}
func matchNotaryDigest(img util.ArtifactInfo) (bool, error) {
if NotaryEndpoint == "" {
NotaryEndpoint = config.InternalNotaryEndpoint()
}
targets, err := notary.GetInternalTargets(NotaryEndpoint, util.TokenUsername, img.Repository)
if err != nil {
return false, err
}
for _, t := range targets {
if utils.IsDigest(img.Reference) {
d, err := notary.DigestFromTarget(t)
if err != nil {
return false, err
}
if img.Digest == d {
return true, nil
}
} else {
if t.Tag == img.Reference {
log.Debugf("found reference: %s in notary, try to match digest.", img.Reference)
d, err := notary.DigestFromTarget(t)
if err != nil {
return false, err
}
if img.Digest == d {
return true, nil
}
}
}
}
log.Debugf("image: %#v, not found in notary", img)
return false, nil
}

62
src/core/middlewares/contenttrust/handler_test.go
View File

@ -1,62 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package contenttrust
import (
"net/http/httptest"
"os"
"testing"
"github.com/goharbor/harbor/src/common"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/middlewares/util"
notarytest "github.com/goharbor/harbor/src/pkg/signature/notary/test"
"github.com/stretchr/testify/assert"
)
var endpoint = "10.117.4.142"
var notaryServer *httptest.Server
var token = ""
func TestMain(m *testing.M) {
notaryServer = notarytest.NewNotaryServer(endpoint)
defer notaryServer.Close()
NotaryEndpoint = notaryServer.URL
var defaultConfig = map[string]interface{}{
common.ExtEndpoint: "https://" + endpoint,
common.WithNotary: true,
common.TokenExpiration: 30,
}
config.InitWithSettings(defaultConfig)
result := m.Run()
if result != 0 {
os.Exit(result)
}
}
func TestMatchNotaryDigest(t *testing.T) {
assert := assert.New(t)
// The data from common/utils/notary/helper_test.go
img1 := util.ArtifactInfo{Repository: "notary-demo/busybox", Reference: "1.0", ProjectName: "notary-demo", Digest: "sha256:1359608115b94599e5641638bac5aef1ddfaa79bb96057ebf41ebc8d33acf8a7"}
img2 := util.ArtifactInfo{Repository: "notary-demo/busybox", Reference: "2.0", ProjectName: "notary-demo", Digest: "sha256:12345678"}
res1, err := matchNotaryDigest(img1)
assert.Nil(err, "Unexpected error: %v, image: %#v", err, img1)
assert.True(res1)
res2, err := matchNotaryDigest(img2)
assert.Nil(err, "Unexpected error: %v, image: %#v, take 2", err, img2)
assert.False(res2)
}

100
src/core/middlewares/countquota/builder.go
View File

@ -1,100 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package countquota
import (
"fmt"
"net/http"
"strconv"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
"github.com/goharbor/harbor/src/core/middlewares/interceptor/quota"
"github.com/goharbor/harbor/src/core/middlewares/util"
)
var (
defaultBuilders = []interceptor.Builder{
&manifestDeletionBuilder{},
&manifestCreationBuilder{},
}
)
type manifestDeletionBuilder struct{}
func (*manifestDeletionBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
if match, _, _ := util.MatchDeleteManifest(req); !match {
return nil, nil
}
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
var err error
info, err = util.ParseManifestInfoFromPath(req)
if err != nil {
return nil, fmt.Errorf("failed to parse manifest, error %v", err)
}
// Manifest info will be used by computeResourcesForDeleteManifest
*req = *(req.WithContext(util.NewManifestInfoContext(req.Context(), info)))
}
opts := []quota.Option{
quota.EnforceResources(config.QuotaPerProjectEnable()),
quota.WithManager("project", strconv.FormatInt(info.ProjectID, 10)),
quota.WithAction(quota.SubtractAction),
quota.StatusCode(http.StatusAccepted),
quota.MutexKeys(info.MutexKey("count")),
quota.OnResources(computeResourcesForManifestDeletion),
quota.OnFulfilled(func(http.ResponseWriter, *http.Request) error {
return dao.DeleteArtifactByDigest(info.ProjectID, info.Repository, info.Digest)
}),
}
return quota.New(opts...), nil
}
type manifestCreationBuilder struct{}
func (*manifestCreationBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
if match, _, _ := util.MatchPushManifest(req); !match {
return nil, nil
}
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
var err error
info, err = util.ParseManifestInfoFromReq(req)
if err != nil {
return nil, fmt.Errorf("failed to parse manifest, error %v", err)
}
// Manifest info will be used by computeResourcesForCreateManifest
*req = *(req.WithContext(util.NewManifestInfoContext(req.Context(), info)))
}
opts := []quota.Option{
quota.EnforceResources(config.QuotaPerProjectEnable()),
quota.WithManager("project", strconv.FormatInt(info.ProjectID, 10)),
quota.WithAction(quota.AddAction),
quota.StatusCode(http.StatusCreated),
quota.MutexKeys(info.MutexKey("count")),
quota.OnResources(computeResourcesForManifestCreation),
quota.OnFulfilled(afterManifestCreated),
}
return quota.New(opts...), nil
}

89
src/core/middlewares/countquota/handler.go
View File

@ -1,89 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package countquota
import (
"fmt"
"net/http"
"github.com/goharbor/harbor/src/common/quota"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
"github.com/goharbor/harbor/src/core/middlewares/util"
)
type countQuotaHandler struct {
builders []interceptor.Builder
next http.Handler
}
// New ...
func New(next http.Handler, builders ...interceptor.Builder) http.Handler {
if len(builders) == 0 {
builders = defaultBuilders
}
return &countQuotaHandler{
builders: builders,
next: next,
}
}
// ServeHTTP manifest ...
func (h *countQuotaHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
interceptor, err := h.getInterceptor(req)
if err != nil {
log.Warningf("Error occurred when to handle request in count quota handler: %v", err)
http.Error(rw, util.MarshalError("InternalError", fmt.Sprintf("Error occurred when to handle request in count quota handler: %v", err)),
http.StatusInternalServerError)
return
}
if interceptor == nil {
h.next.ServeHTTP(rw, req)
return
}
if err := interceptor.HandleRequest(req); err != nil {
log.Warningf("Error occurred when to handle request in count quota handler: %v", err)
if _, ok := err.(quota.Errors); ok {
util.FireQuotaEvent(req, 1, err.Error())
http.Error(rw, util.MarshalError("DENIED", fmt.Sprintf("Quota exceeded when processing the request of %v", err)), http.StatusForbidden)
return
}
http.Error(rw, util.MarshalError("InternalError", fmt.Sprintf("Error occurred when to handle request in count quota handler: %v", err)),
http.StatusInternalServerError)
return
}
h.next.ServeHTTP(rw, req)
interceptor.HandleResponse(rw, req)
}
func (h *countQuotaHandler) getInterceptor(req *http.Request) (interceptor.Interceptor, error) {
for _, builder := range h.builders {
interceptor, err := builder.Build(req)
if err != nil {
return nil, err
}
if interceptor != nil {
return interceptor, nil
}
}
return nil, nil
}

331
src/core/middlewares/countquota/handler_test.go
View File

@ -1,331 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package countquota
import (
"fmt"
"math/rand"
"net/http"
"net/http/httptest"
"os"
"testing"
"time"
"github.com/docker/distribution"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/goharbor/harbor/src/pkg/types"
"github.com/opencontainers/go-digest"
"github.com/stretchr/testify/suite"
)
func init() {
rand.Seed(time.Now().UnixNano())
}
func getProjectCountUsage(projectID int64) (int64, error) {
usage := models.QuotaUsage{Reference: "project", ReferenceID: fmt.Sprintf("%d", projectID)}
err := dao.GetOrmer().Read(&usage, "reference", "reference_id")
if err != nil {
return 0, err
}
used, err := types.NewResourceList(usage.Used)
if err != nil {
return 0, err
}
return used[types.ResourceCount], nil
}
func randomString(n int) string {
const letterBytes = "abcdefghijklmnopqrstuvwxyz"
b := make([]byte, n)
for i := range b {
b[i] = letterBytes[rand.Intn(len(letterBytes))]
}
return string(b)
}
func doDeleteManifestRequest(projectID int64, projectName, name, dgt string, next ...http.HandlerFunc) int {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/manifests/%s", repository, dgt)
req, _ := http.NewRequest("DELETE", url, nil)
ctx := util.NewManifestInfoContext(req.Context(), &util.ManifestInfo{
ProjectID: projectID,
Repository: repository,
Digest: dgt,
})
rr := httptest.NewRecorder()
var n http.HandlerFunc
if len(next) > 0 {
n = next[0]
} else {
n = func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusAccepted)
}
}
h := New(http.HandlerFunc(n))
h.ServeHTTP(util.NewCustomResponseWriter(rr), req.WithContext(ctx))
return rr.Code
}
func doPutManifestRequest(projectID int64, projectName, name, tag, dgt string, withDupBlob bool, next ...http.HandlerFunc) int {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/manifests/%s", repository, tag)
req, _ := http.NewRequest("PUT", url, nil)
mfInfo := &util.ManifestInfo{
ProjectID: projectID,
Repository: repository,
Tag: tag,
Digest: dgt,
References: []distribution.Descriptor{
{Digest: digest.FromString(randomString(15))},
{Digest: digest.FromString(randomString(15))},
},
}
ctx := util.NewManifestInfoContext(req.Context(), mfInfo)
if withDupBlob {
dupDigest := digest.FromString(randomString(15))
mfInfo.References = append(mfInfo.References, distribution.Descriptor{Digest: dupDigest})
mfInfo.References = append(mfInfo.References, distribution.Descriptor{Digest: dupDigest})
}
rr := httptest.NewRecorder()
var n http.HandlerFunc
if len(next) > 0 {
n = next[0]
} else {
n = func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusCreated)
}
}
h := New(http.HandlerFunc(n))
h.ServeHTTP(util.NewCustomResponseWriter(rr), req.WithContext(ctx))
return rr.Code
}
type HandlerSuite struct {
suite.Suite
}
func (suite *HandlerSuite) addProject(projectName string) int64 {
projectID, err := dao.AddProject(models.Project{
Name: projectName,
OwnerID: 1,
})
suite.Nil(err, fmt.Sprintf("Add project failed for %s", projectName))
return projectID
}
func (suite *HandlerSuite) checkCountUsage(expected, projectID int64) {
count, err := getProjectCountUsage(projectID)
suite.Nil(err, fmt.Sprintf("Failed to get count usage of project %d, error: %v", projectID, err))
suite.Equal(expected, count, "Failed to check count usage for project %d", projectID)
}
func (suite *HandlerSuite) TearDownTest() {
for _, table := range []string{
"artifact", "blob",
"artifact_blob", "project_blob",
"quota", "quota_usage",
} {
dao.ClearTable(table)
}
}
func (suite *HandlerSuite) TestPutManifestCreated() {
projectName := randomString(5)
projectID := suite.addProject(projectName)
defer func() {
dao.DeleteProject(projectID)
}()
dgt := digest.FromString(randomString(15)).String()
code := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt, false)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(1, projectID)
total, err := dao.GetTotalOfArtifacts(&models.ArtifactQuery{Digest: dgt})
suite.Nil(err)
suite.Equal(int64(1), total, "Artifact should be created")
// Push the photon:latest with photon:dev
code = doPutManifestRequest(projectID, projectName, "photon", "dev", dgt, false)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(2, projectID)
total, err = dao.GetTotalOfArtifacts(&models.ArtifactQuery{Digest: dgt})
suite.Nil(err)
suite.Equal(int64(2), total, "Artifact should be created")
// Push the photon:latest with new image
newDgt := digest.FromString(randomString(15)).String()
code = doPutManifestRequest(projectID, projectName, "photon", "latest", newDgt, false)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(2, projectID)
total, err = dao.GetTotalOfArtifacts(&models.ArtifactQuery{Digest: newDgt})
suite.Nil(err)
suite.Equal(int64(1), total, "Artifact should be updated")
}
func (suite *HandlerSuite) TestPutManifestCreatedDupBlobs() {
projectName := randomString(5)
projectID := suite.addProject(projectName)
defer func() {
dao.DeleteProject(projectID)
}()
dgt := digest.FromString(randomString(15)).String()
code := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt, true)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(1, projectID)
var count int64
err := dao.GetOrmer().Raw("select count(*) from artifact_blob where digest_af = ?", dgt).QueryRow(&count)
suite.Nil(err)
// 4 = self + 3 distinct blobs
suite.Equal(int64(4), count)
}
func (suite *HandlerSuite) TestPutManifestFailed() {
projectName := randomString(5)
projectID := suite.addProject(projectName)
defer func() {
dao.DeleteProject(projectID)
}()
next := func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusForbidden)
}
dgt := digest.FromString(randomString(15)).String()
code := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt, false, next)
suite.Equal(http.StatusForbidden, code)
suite.checkCountUsage(0, projectID)
total, err := dao.GetTotalOfArtifacts(&models.ArtifactQuery{Digest: dgt})
suite.Nil(err)
suite.Equal(int64(0), total, "Artifact should not be created")
}
func (suite *HandlerSuite) TestDeleteManifestAccepted() {
projectName := randomString(5)
projectID := suite.addProject(projectName)
defer func() {
dao.DeleteProject(projectID)
}()
dgt := digest.FromString(randomString(15)).String()
code := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt, false)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(1, projectID)
code = doDeleteManifestRequest(projectID, projectName, "photon", dgt)
suite.Equal(http.StatusAccepted, code)
suite.checkCountUsage(0, projectID)
}
func (suite *HandlerSuite) TestDeleteManifestFailed() {
projectName := randomString(5)
projectID := suite.addProject(projectName)
defer func() {
dao.DeleteProject(projectID)
}()
dgt := digest.FromString(randomString(15)).String()
code := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt, false)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(1, projectID)
next := func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}
code = doDeleteManifestRequest(projectID, projectName, "photon", dgt, next)
suite.Equal(http.StatusInternalServerError, code)
suite.checkCountUsage(1, projectID)
}
func (suite *HandlerSuite) TestDeleteManifestInMultiProjects() {
projectName := randomString(5)
projectID := suite.addProject(projectName)
defer func() {
dao.DeleteProject(projectID)
}()
dgt := digest.FromString(randomString(15)).String()
code := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt, false)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(1, projectID)
{
projectName := randomString(5)
projectID := suite.addProject(projectName)
defer func() {
dao.DeleteProject(projectID)
}()
code := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt, false)
suite.Equal(http.StatusCreated, code)
suite.checkCountUsage(1, projectID)
code = doDeleteManifestRequest(projectID, projectName, "photon", dgt)
suite.Equal(http.StatusAccepted, code)
suite.checkCountUsage(0, projectID)
}
code = doDeleteManifestRequest(projectID, projectName, "photon", dgt)
suite.Equal(http.StatusAccepted, code)
suite.checkCountUsage(0, projectID)
}
func TestMain(m *testing.M) {
config.Init()
dao.PrepareTestForPostgresSQL()
if result := m.Run(); result != 0 {
os.Exit(result)
}
}
func TestRunHandlerSuite(t *testing.T) {
suite.Run(t, new(HandlerSuite))
}

124
src/core/middlewares/countquota/util.go
View File

@ -1,124 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package countquota
import (
"errors"
"fmt"
"net/http"
"strings"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/quota"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/goharbor/harbor/src/pkg/types"
)
// computeResourcesForManifestCreation returns count resource required for manifest
// no count required if the tag of the repository exists in the project
func computeResourcesForManifestCreation(req *http.Request) (types.ResourceList, error) {
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
return nil, errors.New("manifest info missing")
}
// only count quota required when push new tag
if info.IsNewTag() {
return quota.ResourceList{quota.ResourceCount: 1}, nil
}
return nil, nil
}
// computeResourcesForManifestDeletion returns count resource will be released when manifest deleted
// then result will be the sum of manifest count of the same repository in the project
func computeResourcesForManifestDeletion(req *http.Request) (types.ResourceList, error) {
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
return nil, errors.New("manifest info missing")
}
total, err := dao.GetTotalOfArtifacts(&models.ArtifactQuery{
PID: info.ProjectID,
Repo: info.Repository,
Digest: info.Digest,
})
if err != nil {
return nil, fmt.Errorf("error occurred when get artifacts %v ", err)
}
return types.ResourceList{types.ResourceCount: total}, nil
}
// afterManifestCreated the handler after manifest created success
// it will create or update the artifact info in db, and then attach blobs to artifact
func afterManifestCreated(w http.ResponseWriter, req *http.Request) error {
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
return errors.New("manifest info missing")
}
artifact := info.Artifact()
if artifact.ID == 0 {
if _, err := dao.AddArtifact(artifact); err != nil {
return fmt.Errorf("error to add artifact, %v", err)
}
} else {
if err := dao.UpdateArtifact(artifact); err != nil {
return fmt.Errorf("error to update artifact, %v", err)
}
}
return attachBlobsToArtifact(info)
}
// attachBlobsToArtifact attach the blobs which from manifest to artifact
func attachBlobsToArtifact(info *util.ManifestInfo) error {
temp := make(map[string]interface{})
artifactBlobs := []*models.ArtifactAndBlob{}
temp[info.Digest] = nil
// self
artifactBlobs = append(artifactBlobs, &models.ArtifactAndBlob{
DigestAF: info.Digest,
DigestBlob: info.Digest,
})
// avoid the duplicate layers.
for _, reference := range info.References {
_, exist := temp[reference.Digest.String()]
if !exist {
temp[reference.Digest.String()] = nil
artifactBlobs = append(artifactBlobs, &models.ArtifactAndBlob{
DigestAF: info.Digest,
DigestBlob: reference.Digest.String(),
})
}
}
if err := dao.AddArtifactNBlobs(artifactBlobs); err != nil {
if strings.Contains(err.Error(), dao.ErrDupRows.Error()) {
log.Warning("the artifact and blobs have already in the DB, it maybe an existing image with different tag")
return nil
}
return fmt.Errorf("error to add artifact and blobs in proxy response handler, %v", err)
}
return nil
}

54
src/core/middlewares/immutable/builder.go
View File

@ -1,54 +0,0 @@
package immutable
import (
"fmt"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
"github.com/goharbor/harbor/src/core/middlewares/interceptor/immutable"
"github.com/goharbor/harbor/src/core/middlewares/util"
"net/http"
)
var (
defaultBuilders = []interceptor.Builder{
&manifestDeletionBuilder{},
&manifestCreationBuilder{},
}
)
type manifestDeletionBuilder struct{}
func (*manifestDeletionBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
if match, _, _ := util.MatchDeleteManifest(req); !match {
return nil, nil
}
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
var err error
info, err = util.ParseManifestInfoFromPath(req)
if err != nil {
return nil, fmt.Errorf("failed to parse manifest, error %v", err)
}
}
return immutable.NewDeleteMFInteceptor(info), nil
}
type manifestCreationBuilder struct{}
func (*manifestCreationBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
if match, _, _ := util.MatchPushManifest(req); !match {
return nil, nil
}
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
var err error
info, err = util.ParseManifestInfoFromReq(req)
if err != nil {
return nil, fmt.Errorf("failed to parse manifest, error %v", err)
}
}
return immutable.NewPushMFInteceptor(info), nil
}

95
src/core/middlewares/immutable/handler.go
View File

@ -1,95 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package immutable
import (
"errors"
"fmt"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
middlerware_err "github.com/goharbor/harbor/src/core/middlewares/util/error"
internal_errors "github.com/goharbor/harbor/src/internal/error"
"net/http"
)
type immutableHandler struct {
builders []interceptor.Builder
next http.Handler
}
// New ...
func New(next http.Handler, builders ...interceptor.Builder) http.Handler {
if len(builders) == 0 {
builders = defaultBuilders
}
return &immutableHandler{
builders: builders,
next: next,
}
}
// ServeHTTP ...
func (rh *immutableHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
interceptor, err := rh.getInterceptor(req)
if err != nil {
log.Warningf("Error occurred when to handle request in immutable handler: %v", err)
pkgE := internal_errors.New(fmt.Errorf("error occurred when to handle request in immutable handler: %v", err)).WithCode(internal_errors.GeneralCode)
msg := internal_errors.NewErrs(pkgE).Error()
http.Error(rw, msg, http.StatusInternalServerError)
return
}
if interceptor == nil {
rh.next.ServeHTTP(rw, req)
return
}
if err := interceptor.HandleRequest(req); err != nil {
log.Warningf("Error occurred when to handle request in immutable handler: %v", err)
var e *middlerware_err.ErrImmutable
if errors.As(err, &e) {
pkgE := internal_errors.New(e).WithCode(internal_errors.PreconditionCode)
msg := internal_errors.NewErrs(pkgE).Error()
http.Error(rw, msg, http.StatusPreconditionFailed)
return
}
pkgE := internal_errors.New(fmt.Errorf("error occurred when to handle request in immutable handler: %v", err)).WithCode(internal_errors.GeneralCode)
msg := internal_errors.NewErrs(pkgE).Error()
http.Error(rw, msg, http.StatusInternalServerError)
return
}
rh.next.ServeHTTP(rw, req)
interceptor.HandleResponse(rw, req)
}
func (rh *immutableHandler) getInterceptor(req *http.Request) (interceptor.Interceptor, error) {
for _, builder := range rh.builders {
interceptor, err := builder.Build(req)
if err != nil {
return nil, err
}
if interceptor != nil {
return interceptor, nil
}
}
return nil, nil
}

151
src/core/middlewares/immutable/handler_test.go
View File

@ -1,151 +0,0 @@
package immutable
import (
"github.com/docker/distribution"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/opencontainers/go-digest"
"fmt"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/pkg/immutabletag"
immu_model "github.com/goharbor/harbor/src/pkg/immutabletag/model"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
"math/rand"
"net/http"
"net/http/httptest"
"os"
"testing"
)
type HandlerSuite struct {
suite.Suite
}
func doPutManifestRequest(projectID int64, projectName, name, tag, dgt string, next ...http.HandlerFunc) int {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/manifests/%s", repository, tag)
req, _ := http.NewRequest("PUT", url, nil)
mfInfo := &util.ManifestInfo{
ProjectID: projectID,
Repository: repository,
Tag: tag,
Digest: dgt,
References: []distribution.Descriptor{
{Digest: digest.FromString(randomString(15))},
{Digest: digest.FromString(randomString(15))},
},
}
ctx := util.NewManifestInfoContext(req.Context(), mfInfo)
rr := httptest.NewRecorder()
var n http.HandlerFunc
if len(next) > 0 {
n = next[0]
} else {
n = func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusCreated)
}
}
h := New(http.HandlerFunc(n))
h.ServeHTTP(util.NewCustomResponseWriter(rr), req.WithContext(ctx))
return rr.Code
}
func randomString(n int) string {
const letterBytes = "abcdefghijklmnopqrstuvwxyz"
b := make([]byte, n)
for i := range b {
b[i] = letterBytes[rand.Intn(len(letterBytes))]
}
return string(b)
}
func (suite *HandlerSuite) addProject(projectName string) int64 {
projectID, err := dao.AddProject(models.Project{
Name: projectName,
OwnerID: 1,
})
suite.Nil(err, fmt.Sprintf("Add project failed for %s", projectName))
return projectID
}
func (suite *HandlerSuite) addArt(pid int64, repo string, tag string) int64 {
afid, err := dao.AddArtifact(&models.Artifact{
PID: pid,
Repo: repo,
Tag: tag,
Digest: digest.FromString(randomString(15)).String(),
Kind: "Docker-Image",
})
suite.Nil(err, fmt.Sprintf("Add artifact failed for %s", repo))
return afid
}
func (suite *HandlerSuite) addImmutableRule(pid int64) int64 {
metadata := &immu_model.Metadata{
ProjectID: pid,
Priority: 1,
Action: "immutable",
Template: "immutable_template",
TagSelectors: []*immu_model.Selector{
{
Kind: "doublestar",
Decoration: "matches",
Pattern: "release-**",
},
},
ScopeSelectors: map[string][]*immu_model.Selector{
"repository": {
{
Kind: "doublestar",
Decoration: "repoMatches",
Pattern: "**",
},
},
},
}
id, err := immutabletag.ImmuCtr.CreateImmutableRule(metadata)
require.NoError(suite.T(), err, "nil error expected but got %s", err)
return id
}
func (suite *HandlerSuite) TestPutManifestCreated() {
projectName := randomString(5)
projectID := suite.addProject(projectName)
immuRuleID := suite.addImmutableRule(projectID)
afID := suite.addArt(projectID, projectName+"/photon", "release-1.10")
defer func() {
dao.DeleteProject(projectID)
dao.DeleteArtifact(afID)
immutabletag.ImmuCtr.DeleteImmutableRule(immuRuleID)
}()
dgt := digest.FromString(randomString(15)).String()
code1 := doPutManifestRequest(projectID, projectName, "photon", "release-1.10", dgt)
suite.Equal(http.StatusPreconditionFailed, code1)
code2 := doPutManifestRequest(projectID, projectName, "photon", "latest", dgt)
suite.Equal(http.StatusCreated, code2)
}
func TestMain(m *testing.M) {
dao.PrepareTestForPostgresSQL()
if result := m.Run(); result != 0 {
os.Exit(result)
}
}
func TestRunHandlerSuite(t *testing.T) {
suite.Run(t, new(HandlerSuite))
}

58
src/core/middlewares/inlet.go
View File

@ -1,58 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package middlewares
import (
"errors"
"net/http"
"github.com/goharbor/harbor/src/common/security"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/registryproxy"
"github.com/goharbor/harbor/src/core/middlewares/util"
)
var head http.Handler
var proxy http.Handler
// Init initialize the Proxy instance and handler chain.
func Init() error {
proxy = registryproxy.New()
if proxy == nil {
return errors.New("get nil when to create proxy")
}
return nil
}
// Handle handles the request.
func Handle(rw http.ResponseWriter, req *http.Request) {
securityCtx, ok := security.FromContext(req.Context())
if !ok {
log.Errorf("failed to get security context in middlerware")
// error to get security context, use the default chain.
head = New(Middlewares).Create().Then(proxy)
} else {
// true: the request is from 127.0.0.1, only quota middlewares are applied to request
// false: the request is from outside, all of middlewares are applied to the request.
if securityCtx.IsSolutionUser() {
head = New(MiddlewaresLocal).Create().Then(proxy)
} else {
head = New(Middlewares).Create().Then(proxy)
}
}
customResW := util.NewCustomResponseWriter(rw)
head.ServeHTTP(customResW, req)
}

67
src/core/middlewares/interceptor/immutable/deletemf.go
View File

@ -1,67 +0,0 @@
package immutable
import (
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
common_util "github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
"github.com/goharbor/harbor/src/core/middlewares/util"
middlerware_err "github.com/goharbor/harbor/src/core/middlewares/util/error"
"github.com/goharbor/harbor/src/pkg/art"
"github.com/goharbor/harbor/src/pkg/immutabletag/match/rule"
"net/http"
)
// NewDeleteMFInteceptor ....
func NewDeleteMFInteceptor(mf *util.ManifestInfo) interceptor.Interceptor {
return &delmfInterceptor{
mf: mf,
}
}
type delmfInterceptor struct {
mf *util.ManifestInfo
}
// HandleRequest ...
func (dmf *delmfInterceptor) HandleRequest(req *http.Request) (err error) {
artifactQuery := &models.ArtifactQuery{
Digest: dmf.mf.Digest,
Repo: dmf.mf.Repository,
PID: dmf.mf.ProjectID,
}
var afs []*models.Artifact
afs, err = dao.ListArtifacts(artifactQuery)
if err != nil {
log.Error(err)
return
}
if len(afs) == 0 {
return
}
for _, af := range afs {
_, repoName := common_util.ParseRepository(dmf.mf.Repository)
var matched bool
matched, err = rule.NewRuleMatcher().Match(dmf.mf.ProjectID, art.Candidate{
Repository: repoName,
Tags: []string{af.Tag},
NamespaceID: dmf.mf.ProjectID,
})
if err != nil {
log.Error(err)
return
}
if matched {
return middlerware_err.NewErrImmutable(repoName, af.Tag)
}
}
return
}
// HandleRequest ...
func (dmf *delmfInterceptor) HandleResponse(w http.ResponseWriter, r *http.Request) {
}

65
src/core/middlewares/interceptor/immutable/pushmf.go
View File

@ -1,65 +0,0 @@
package immutable
import (
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
common_util "github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
"github.com/goharbor/harbor/src/core/middlewares/util"
middlerware_err "github.com/goharbor/harbor/src/core/middlewares/util/error"
"github.com/goharbor/harbor/src/pkg/art"
"github.com/goharbor/harbor/src/pkg/immutabletag/match/rule"
"net/http"
)
// NewPushMFInteceptor ....
func NewPushMFInteceptor(mf *util.ManifestInfo) interceptor.Interceptor {
return &pushmfInterceptor{
mf: mf,
}
}
type pushmfInterceptor struct {
mf *util.ManifestInfo
}
// HandleRequest ...
func (pmf *pushmfInterceptor) HandleRequest(req *http.Request) (err error) {
_, repoName := common_util.ParseRepository(pmf.mf.Repository)
var matched bool
matched, err = rule.NewRuleMatcher().Match(pmf.mf.ProjectID, art.Candidate{
Repository: repoName,
Tags: []string{pmf.mf.Tag},
NamespaceID: pmf.mf.ProjectID,
})
if err != nil {
log.Error(err)
return
}
if !matched {
return
}
artifactQuery := &models.ArtifactQuery{
PID: pmf.mf.ProjectID,
Repo: pmf.mf.Repository,
Tag: pmf.mf.Tag,
}
var afs []*models.Artifact
afs, err = dao.ListArtifacts(artifactQuery)
if err != nil {
log.Error(err)
return
}
if len(afs) == 0 {
return
}
return middlerware_err.NewErrImmutable(repoName, pmf.mf.Tag)
}
// HandleRequest ...
func (pmf *pushmfInterceptor) HandleResponse(w http.ResponseWriter, r *http.Request) {
}

104
src/core/middlewares/listrepo/handler.go
View File

@ -1,104 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package listrepo
import (
"encoding/json"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/util"
"net/http"
"net/http/httptest"
"regexp"
"strconv"
)
const (
catalogURLPattern = `/v2/_catalog`
)
type listReposHandler struct {
next http.Handler
}
// New ...
func New(next http.Handler) http.Handler {
return &listReposHandler{
next: next,
}
}
// ServeHTTP ...
func (lrh listReposHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
var rec *httptest.ResponseRecorder
listReposFlag := matchListRepos(req)
if listReposFlag {
rec = httptest.NewRecorder()
lrh.next.ServeHTTP(rec, req)
if rec.Result().StatusCode != http.StatusOK {
util.CopyResp(rec, rw)
return
}
var ctlg struct {
Repositories []string `json:"repositories"`
}
decoder := json.NewDecoder(rec.Body)
if err := decoder.Decode(&ctlg); err != nil {
log.Errorf("Decode repositories error: %v", err)
util.CopyResp(rec, rw)
return
}
var entries []string
for repo := range ctlg.Repositories {
log.Debugf("the repo in the response %s", ctlg.Repositories[repo])
exist := dao.RepositoryExists(ctlg.Repositories[repo])
if exist {
entries = append(entries, ctlg.Repositories[repo])
}
}
type Repos struct {
Repositories []string `json:"repositories"`
}
resp := &Repos{Repositories: entries}
respJSON, err := json.Marshal(resp)
if err != nil {
log.Errorf("Encode repositories error: %v", err)
util.CopyResp(rec, rw)
return
}
for k, v := range rec.Header() {
rw.Header()[k] = v
}
clen := len(respJSON)
rw.Header().Set(http.CanonicalHeaderKey("Content-Length"), strconv.Itoa(clen))
rw.Write(respJSON)
return
}
lrh.next.ServeHTTP(rw, req)
}
// matchListRepos checks if the request looks like a request to list repositories.
func matchListRepos(req *http.Request) bool {
if req.Method != http.MethodGet {
return false
}
re := regexp.MustCompile(catalogURLPattern)
s := re.FindStringSubmatch(req.URL.Path)
if len(s) == 1 {
return true
}
return false
}

37
src/core/middlewares/listrepo/handler_test.go
View File

@ -1,37 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package listrepo
import (
"github.com/stretchr/testify/assert"
"net/http"
"testing"
)
func TestMatchListRepos(t *testing.T) {
assert := assert.New(t)
req1, _ := http.NewRequest("POST", "http://127.0.0.1:5000/v2/_catalog", nil)
res1 := matchListRepos(req1)
assert.False(res1, "%s %v is not a request to list repos", req1.Method, req1.URL)
req2, _ := http.NewRequest("GET", "http://127.0.0.1:5000/v2/_catalog", nil)
res2 := matchListRepos(req2)
assert.True(res2, "%s %v is a request to list repos", req2.Method, req2.URL)
req3, _ := http.NewRequest("GET", "https://192.168.0.5:443/v1/_catalog", nil)
res3 := matchListRepos(req3)
assert.False(res3, "%s %v is not a request to pull manifest", req3.Method, req3.URL)
}

45
src/core/middlewares/readonly/hanlder.go
View File

@ -1,45 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package readonly
import (
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/middlewares/util"
"net/http"
)
type readonlyHandler struct {
next http.Handler
}
// New ...
func New(next http.Handler) http.Handler {
return &readonlyHandler{
next: next,
}
}
// ServeHTTP ...
func (rh readonlyHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
if config.ReadOnly() {
if req.Method == http.MethodDelete || req.Method == http.MethodPost || req.Method == http.MethodPatch || req.Method == http.MethodPut {
log.Warningf("The request is prohibited in readonly mode, url is: %s", req.URL.Path)
http.Error(rw, util.MarshalError("DENIED", "The system is in read only mode. Any modification is prohibited."), http.StatusForbidden)
return
}
}
rh.next.ServeHTTP(rw, req)
}

61
src/core/middlewares/registryproxy/handler.go
View File

@ -1,61 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package registryproxy
import (
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
"net/http"
"net/http/httputil"
"net/url"
)
type proxyHandler struct {
handler http.Handler
}
// New ...
func New(urls ...string) http.Handler {
var registryURL string
var err error
if len(urls) > 1 {
log.Errorf("the parm, urls should have only 0 or 1 elements")
return nil
}
if len(urls) == 0 {
registryURL, err = config.RegistryURL()
if err != nil {
log.Error(err)
return nil
}
} else {
registryURL = urls[0]
}
targetURL, err := url.Parse(registryURL)
if err != nil {
log.Error(err)
return nil
}
return &proxyHandler{
handler: httputil.NewSingleHostReverseProxy(targetURL),
}
}
// ServeHTTP ...
func (ph proxyHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
ph.handler.ServeHTTP(rw, req)
}

72
src/core/middlewares/regtoken/handler.go
View File

@ -1,72 +0,0 @@
package regtoken
import (
"github.com/docker/distribution/registry/auth"
"github.com/goharbor/harbor/src/common/rbac"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/util"
pkg_token "github.com/goharbor/harbor/src/pkg/token"
"github.com/goharbor/harbor/src/pkg/token/claims/registry"
"net/http"
"strings"
)
// regTokenHandler is responsible for decoding the registry token in the docker pull request header,
// as harbor adds customized claims action into registry auth token, the middlerware is for decode it and write it into
// request context, then for other middlerwares in chain to use it to bypass request validation.
type regTokenHandler struct {
next http.Handler
}
// New ...
func New(next http.Handler) http.Handler {
return &regTokenHandler{
next: next,
}
}
// ServeHTTP ...
func (r *regTokenHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
imgRaw := req.Context().Value(util.ArtifactInfoCtxKey)
if imgRaw == nil {
r.next.ServeHTTP(rw, req)
return
}
img, _ := req.Context().Value(util.ArtifactInfoCtxKey).(util.ArtifactInfo)
if img.Digest == "" {
r.next.ServeHTTP(rw, req)
return
}
parts := strings.Split(req.Header.Get("Authorization"), " ")
if len(parts) != 2 || strings.ToLower(parts[0]) != "bearer" {
r.next.ServeHTTP(rw, req)
return
}
rawToken := parts[1]
opt := pkg_token.DefaultTokenOptions()
regTK, err := pkg_token.Parse(opt, rawToken, &registry.Claim{})
if err != nil {
log.Errorf("failed to decode reg token: %v, the error is skipped and round the request to native registry.", err)
r.next.ServeHTTP(rw, req)
return
}
accessItems := []auth.Access{}
accessItems = append(accessItems, auth.Access{
Resource: auth.Resource{
Type: rbac.ResourceRepository.String(),
Name: img.Repository,
},
Action: rbac.ActionScannerPull.String(),
})
accessSet := regTK.Claims.(*registry.Claim).GetAccess()
for _, access := range accessItems {
if accessSet.Contains(access) {
*req = *(req.WithContext(util.NewScannerPullContext(req.Context(), true)))
}
}
r.next.ServeHTTP(rw, req)
}

55
src/core/middlewares/regtoken/handler_test.go
View File

@ -1,55 +0,0 @@
package regtoken
import (
"fmt"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/stretchr/testify/suite"
"net/http"
"net/http/httptest"
"os"
"testing"
)
type HandlerSuite struct {
suite.Suite
}
func doPullManifestRequest(projectName, name, tag string, next ...http.HandlerFunc) int {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/manifests/%s", repository, tag)
req, _ := http.NewRequest("GET", url, nil)
token := "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkNWUTc6REM3NTpHVEROOkxTTUs6VUFJTjpIUUVWOlZVSDQ6Q0lRRDpRV01COlM0Qzc6U0c0STpGRUhYIn0.eyJpc3MiOiJoYXJib3ItdG9rZW4taXNzdWVyIiwic3ViIjoicm9ib3QkZGVtbzExIiwiYXVkIjoiaGFyYm9yLXJlZ2lzdHJ5IiwiZXhwIjoxNTcxNzYzOTI2LCJuYmYiOjE1NzE3NjM4NjYsImlhdCI6MTU3MTc2Mzg2NiwianRpIjoiTnRaZWx4Z01KTUU1MXlEMCIsImFjY2VzcyI6W3sidHlwZSI6InJlcG9zaXRvcnkiLCJuYW1lIjoibGlicmFyeS9oZWxsby13b3JsZCIsImFjdGlvbnMiOlsicHVzaCIsIioiLCJwdWxsIiwic2Nhbm5lcnB1bGwiXX1dfQ.GlWuvtoxmChnpvbWaG5901Z9-g63DrzyNUREWlDbR5gnNeuOKjLNyE4QpogAQKx2yYtcGxbqNL3VfJkExJ_gMS0Qw8e10utGOawwqD4oqf_J06eKq4HzpZJengZfcjMA4g2RoeOlqdVdwimB_PdX9vkBO1od0wX0Cc2v0p2w5TkibcThKRoeLeVs2oRewkKLuVHNSM8wwRIlAvpWJuNnvRCFlHRkLcZM_KpGXqT7H-PZETTisWCi1pMxeYEwIsDFLlTKdV8LaiDeDmH-RaLOsuyAySYEW9Ynk5K3P_dUl2c_SYQXloPyi0MvXxSn6EWE4eHF2oQDM_SvIzR9sOVB8TtjMjKKMQ4yr_mqgMcfEpnInJATExBR56wmxNdLESncHl8rUYCe2jCjQFuR9NGQA1tGdjI4NoBN-OVD0dBs9rm_mkb2tgD-3gEhyzAw6hg0uzDsF7bj5Aq8scoi42UurhX2bZM89s4-TWBp4DWuBG0HDiwpOiBvB3RMm6MpQxsqrl0hQm_WH18L6QCknAW2e3d_6DJWJ0eBzISrhDr7LkqJKl1J8pv4zqoh_EUVeLyzTmjEULm-VbnpVF4wW5yTLF3S6F7Ox4vwWtVfi1XQNVOcJDB3VPUsRgiTTuCW-ZGcBLw-OdIcwaJ3T_QZkEjUw1f6i1JcGa0Mpgl83aLiSdQ 0xc0003c77c0 map[alg:RS256 kid:CVQ7:DC75:GTDN:LSMK:UAIN:HQEV:VUH4:CIQD:QWMB:S4C7:SG4I:FEHX typ:JWT] 0xc000496000 GlWuvtoxmChnpvbWaG5901Z9-g63DrzyNUREWlDbR5gnNeuOKjLNyE4QpogAQKx2yYtcGxbqNL3VfJkExJ_gMS0Qw8e10utGOawwqD4oqf_J06eKq4HzpZJengZfcjMA4g2RoeOlqdVdwimB_PdX9vkBO1od0wX0Cc2v0p2w5TkibcThKRoeLeVs2oRewkKLuVHNSM8wwRIlAvpWJuNnvRCFlHRkLcZM_KpGXqT7H-PZETTisWCi1pMxeYEwIsDFLlTKdV8LaiDeDmH-RaLOsuyAySYEW9Ynk5K3P_dUl2c_SYQXloPyi0MvXxSn6EWE4eHF2oQDM_SvIzR9sOVB8TtjMjKKMQ4yr_mqgMcfEpnInJATExBR56wmxNdLESncHl8rUYCe2jCjQFuR9NGQA1tGdjI4NoBN-OVD0dBs9rm_mkb2tgD-3gEhyzAw6hg0uzDsF7bj5Aq8scoi42UurhX2bZM89s4-TWBp4DWuBG0HDiwpOiBvB3RMm6MpQxsqrl0hQm_WH18L6QCknAW2e3d_6DJWJ0eBzISrhDr7LkqJKl1J8pv4zqoh_EUVeLyzTmjEULm-VbnpVF4wW5yTLF3S6F7Ox4vwWtVfi1XQNVOcJDB3VPUsRgiTTuCW-ZGcBLw-OdIcwaJ3T_QZkEjUw1f6i1JcGa0Mpgl83aLiSdQ"
req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", token))
rr := httptest.NewRecorder()
var n http.HandlerFunc
if len(next) > 0 {
n = next[0]
} else {
n = func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusNotFound)
}
}
h := New(http.HandlerFunc(n))
h.ServeHTTP(util.NewCustomResponseWriter(rr), req)
return rr.Code
}
func (suite *HandlerSuite) TestPullManifest() {
code1 := doPullManifestRequest("library", "photon", "release-1.10")
suite.Equal(http.StatusNotFound, code1)
}
func TestMain(m *testing.M) {
if result := m.Run(); result != 0 {
os.Exit(result)
}
}
func TestRunHandlerSuite(t *testing.T) {
suite.Run(t, new(HandlerSuite))
}

208
src/core/middlewares/sizequota/builder.go
View File

@ -1,208 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sizequota
import (
"fmt"
"net/http"
"strconv"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
"github.com/goharbor/harbor/src/core/middlewares/interceptor/quota"
"github.com/goharbor/harbor/src/core/middlewares/util"
)
var (
defaultBuilders = []interceptor.Builder{
&blobStreamUploadBuilder{},
&blobStorageQuotaBuilder{},
&manifestCreationBuilder{},
&manifestDeletionBuilder{},
}
)
// blobStreamUploadBuilder interceptor builder for PATCH /v2/<name>/blobs/uploads/<uuid>
type blobStreamUploadBuilder struct{}
func (*blobStreamUploadBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
if !match(req, http.MethodPatch, blobUploadURLRe) {
return nil, nil
}
s := blobUploadURLRe.FindStringSubmatch(req.URL.Path)
uuid := s[2]
onResponse := func(w http.ResponseWriter, req *http.Request) {
if !config.QuotaPerProjectEnable() {
return
}
size, err := parseUploadedBlobSize(w)
if err != nil {
log.Errorf("failed to parse uploaded blob size for upload %s, error: %v", uuid, err)
return
}
ok, err := setUploadedBlobSize(uuid, size)
if err != nil {
log.Errorf("failed to update blob update size for upload %s, error: %v", uuid, err)
return
}
if !ok {
// ToDo discuss what to do here.
log.Errorf("fail to set bunk: %s size: %d in redis, it causes unable to set correct quota for the artifact", uuid, size)
}
}
return interceptor.ResponseInterceptorFunc(onResponse), nil
}
// blobStorageQuotaBuilder interceptor builder for these requests
// PUT /v2/<name>/blobs/uploads/<uuid>?digest=<digest>
// POST /v2/<name>/blobs/uploads/?mount=<digest>&from=<repository name>
type blobStorageQuotaBuilder struct{}
func (*blobStorageQuotaBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
parseBlobInfo := getBlobInfoParser(req)
if parseBlobInfo == nil {
return nil, nil
}
info, err := parseBlobInfo(req)
if err != nil {
return nil, err
}
// replace req with blob info context
*req = *(req.WithContext(util.NewBlobInfoContext(req.Context(), info)))
opts := []quota.Option{
quota.EnforceResources(config.QuotaPerProjectEnable()),
quota.WithManager("project", strconv.FormatInt(info.ProjectID, 10)),
quota.WithAction(quota.AddAction),
quota.StatusCode(http.StatusCreated), // NOTICE: mount blob and blob upload complete both return 201 when success
quota.OnResources(computeResourcesForBlob),
quota.MutexKeys(info.MutexKey()),
quota.OnFulfilled(func(http.ResponseWriter, *http.Request) error {
return syncBlobInfoToProject(info)
}),
}
return quota.New(opts...), nil
}
// manifestCreationBuilder interceptor builder for the request PUT /v2/<name>/manifests/<reference>
type manifestCreationBuilder struct{}
func (*manifestCreationBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
if match, _, _ := util.MatchPushManifest(req); !match {
return nil, nil
}
info, err := util.ParseManifestInfoFromReq(req)
if err != nil {
return nil, err
}
// Replace request with manifests info context
*req = *req.WithContext(util.NewManifestInfoContext(req.Context(), info))
// Sync manifest layers to blobs for foreign layers not pushed and they are not in blob table
if err := info.SyncBlobs(); err != nil {
log.Warningf("Failed to sync blobs, error: %v", err)
}
opts := []quota.Option{
quota.EnforceResources(config.QuotaPerProjectEnable()),
quota.WithManager("project", strconv.FormatInt(info.ProjectID, 10)),
quota.WithAction(quota.AddAction),
quota.StatusCode(http.StatusCreated),
quota.OnResources(computeResourcesForManifestCreation),
quota.MutexKeys(info.MutexKey("size")),
quota.OnFulfilled(func(http.ResponseWriter, *http.Request) error {
// manifest created, sync manifest itself as blob to blob and project_blob table
blobInfo, err := parseBlobInfoFromManifest(req)
if err != nil {
return err
}
if err := syncBlobInfoToProject(blobInfo); err != nil {
return err
}
// sync blobs from manifest which are not in project to project_blob table
blobs, err := info.GetBlobsNotInProject()
if err != nil {
return err
}
_, err = dao.AddBlobsToProject(info.ProjectID, blobs...)
return err
}),
}
return quota.New(opts...), nil
}
// deleteManifestBuilder interceptor builder for the request DELETE /v2/<name>/manifests/<reference>
type manifestDeletionBuilder struct{}
func (*manifestDeletionBuilder) Build(req *http.Request) (interceptor.Interceptor, error) {
if match, _, _ := util.MatchDeleteManifest(req); !match {
return nil, nil
}
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
var err error
info, err = util.ParseManifestInfoFromPath(req)
if err != nil {
return nil, fmt.Errorf("failed to parse manifest, error %v", err)
}
// Manifest info will be used by computeResourcesForDeleteManifest
*req = *(req.WithContext(util.NewManifestInfoContext(req.Context(), info)))
}
blobs, err := dao.GetBlobsByArtifact(info.Digest)
if err != nil {
return nil, fmt.Errorf("failed to query blobs of %s, error: %v", info.Digest, err)
}
mutexKeys := []string{info.MutexKey("size")}
for _, blob := range blobs {
mutexKeys = append(mutexKeys, info.BlobMutexKey(blob))
}
opts := []quota.Option{
quota.EnforceResources(config.QuotaPerProjectEnable()),
quota.WithManager("project", strconv.FormatInt(info.ProjectID, 10)),
quota.WithAction(quota.SubtractAction),
quota.StatusCode(http.StatusAccepted),
quota.OnResources(computeResourcesForManifestDeletion),
quota.MutexKeys(mutexKeys...),
quota.OnFulfilled(func(http.ResponseWriter, *http.Request) error {
blobs := info.ExclusiveBlobs
return dao.RemoveBlobsFromProject(info.ProjectID, blobs...)
}),
}
return quota.New(opts...), nil
}

89
src/core/middlewares/sizequota/handler.go
View File

@ -1,89 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sizequota
import (
"fmt"
"net/http"
"github.com/goharbor/harbor/src/common/quota"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/interceptor"
"github.com/goharbor/harbor/src/core/middlewares/util"
)
type sizeQuotaHandler struct {
builders []interceptor.Builder
next http.Handler
}
// New ...
func New(next http.Handler, builders ...interceptor.Builder) http.Handler {
if len(builders) == 0 {
builders = defaultBuilders
}
return &sizeQuotaHandler{
builders: builders,
next: next,
}
}
// ServeHTTP ...
func (h *sizeQuotaHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
interceptor, err := h.getInterceptor(req)
if err != nil {
log.Warningf("Error occurred when to handle request in size quota handler: %v", err)
http.Error(rw, util.MarshalError("InternalError", fmt.Sprintf("Error occurred when to handle request in size quota handler: %v", err)),
http.StatusInternalServerError)
return
}
if interceptor == nil {
h.next.ServeHTTP(rw, req)
return
}
if err := interceptor.HandleRequest(req); err != nil {
log.Warningf("Error occurred when to handle request in size quota handler: %v", err)
if _, ok := err.(quota.Errors); ok {
util.FireQuotaEvent(req, 1, err.Error())
http.Error(rw, util.MarshalError("DENIED", fmt.Sprintf("Quota exceeded when processing the request of %v", err)), http.StatusForbidden)
return
}
http.Error(rw, util.MarshalError("InternalError", fmt.Sprintf("Error occurred when to handle request in size quota handler: %v", err)),
http.StatusInternalServerError)
return
}
h.next.ServeHTTP(rw, req)
interceptor.HandleResponse(rw, req)
}
func (h *sizeQuotaHandler) getInterceptor(req *http.Request) (interceptor.Interceptor, error) {
for _, builder := range h.builders {
interceptor, err := builder.Build(req)
if err != nil {
return nil, err
}
if interceptor != nil {
return interceptor, nil
}
}
return nil, nil
}

751
src/core/middlewares/sizequota/handler_test.go
View File

@ -1,751 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sizequota
import (
"bytes"
"encoding/json"
"fmt"
"math/rand"
"net/http"
"net/http/httptest"
"os"
"strconv"
"sync"
"testing"
"time"
"github.com/docker/distribution"
"github.com/docker/distribution/manifest"
"github.com/docker/distribution/manifest/schema2"
"github.com/goharbor/harbor/src/common"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/core/config"
"github.com/goharbor/harbor/src/core/middlewares/countquota"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/goharbor/harbor/src/pkg/types"
"github.com/opencontainers/go-digest"
"github.com/stretchr/testify/suite"
)
func init() {
rand.Seed(time.Now().UnixNano())
}
func genUUID() string {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return ""
}
return fmt.Sprintf("%X-%X-%X-%X-%X", b[0:4], b[4:6], b[6:8], b[8:10], b[10:])
}
func getProjectCountUsage(projectID int64) (int64, error) {
usage := models.QuotaUsage{Reference: "project", ReferenceID: fmt.Sprintf("%d", projectID)}
err := dao.GetOrmer().Read(&usage, "reference", "reference_id")
if err != nil {
return 0, err
}
used, err := types.NewResourceList(usage.Used)
if err != nil {
return 0, err
}
return used[types.ResourceCount], nil
}
func getProjectStorageUsage(projectID int64) (int64, error) {
usage := models.QuotaUsage{Reference: "project", ReferenceID: fmt.Sprintf("%d", projectID)}
err := dao.GetOrmer().Read(&usage, "reference", "reference_id")
if err != nil {
return 0, err
}
used, err := types.NewResourceList(usage.Used)
if err != nil {
return 0, err
}
return used[types.ResourceStorage], nil
}
func randomString(n int) string {
const letterBytes = "abcdefghijklmnopqrstuvwxyz"
b := make([]byte, n)
for i := range b {
b[i] = letterBytes[rand.Intn(len(letterBytes))]
}
return string(b)
}
func makeManifest(configSize int64, layerSizes []int64) schema2.Manifest {
manifest := schema2.Manifest{
Versioned: manifest.Versioned{SchemaVersion: 2, MediaType: schema2.MediaTypeManifest},
Config: distribution.Descriptor{
MediaType: schema2.MediaTypeImageConfig,
Size: configSize,
Digest: digest.FromString(randomString(15)),
},
}
for _, size := range layerSizes {
manifest.Layers = append(manifest.Layers, distribution.Descriptor{
MediaType: schema2.MediaTypeLayer,
Size: size,
Digest: digest.FromString(randomString(15)),
})
}
return manifest
}
func manifestWithAdditionalLayers(raw schema2.Manifest, layerSizes []int64) schema2.Manifest {
var manifest schema2.Manifest
manifest.Versioned = raw.Versioned
manifest.Config = raw.Config
manifest.Layers = append(manifest.Layers, raw.Layers...)
for _, size := range layerSizes {
manifest.Layers = append(manifest.Layers, distribution.Descriptor{
MediaType: schema2.MediaTypeLayer,
Size: size,
Digest: digest.FromString(randomString(15)),
})
}
return manifest
}
func manifestWithAdditionalForeignLayers(raw schema2.Manifest, layerSizes []int64) schema2.Manifest {
var manifest schema2.Manifest
manifest.Versioned = raw.Versioned
manifest.Config = raw.Config
manifest.Layers = append(manifest.Layers, raw.Layers...)
for _, size := range layerSizes {
manifest.Layers = append(manifest.Layers, distribution.Descriptor{
MediaType: schema2.MediaTypeForeignLayer,
Size: size,
Digest: digest.FromString(randomString(15)),
})
}
return manifest
}
func digestOfManifest(manifest schema2.Manifest) string {
bytes, _ := json.Marshal(manifest)
return digest.FromBytes(bytes).String()
}
func sizeOfManifest(manifest schema2.Manifest) int64 {
bytes, _ := json.Marshal(manifest)
return int64(len(bytes))
}
func sizeOfImage(manifest schema2.Manifest) int64 {
totalSizeOfLayers := manifest.Config.Size
for _, layer := range manifest.Layers {
if layer.MediaType != schema2.MediaTypeForeignLayer {
totalSizeOfLayers += layer.Size
}
}
return sizeOfManifest(manifest) + totalSizeOfLayers
}
func doHandle(req *http.Request, next ...http.HandlerFunc) int {
rr := httptest.NewRecorder()
var n http.HandlerFunc
if len(next) > 0 {
n = next[0]
} else {
n = func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusCreated)
}
}
h := New(http.HandlerFunc(n))
h.ServeHTTP(util.NewCustomResponseWriter(rr), req)
return rr.Code
}
func patchBlobUpload(projectName, name, uuid, blobDigest string, chunkSize int64) {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/blobs/uploads/%s?digest=%s", repository, uuid, blobDigest)
req, _ := http.NewRequest(http.MethodPatch, url, nil)
doHandle(req, func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusAccepted)
w.Header().Add("Range", fmt.Sprintf("0-%d", chunkSize-1))
})
}
func putBlobUpload(projectName, name, uuid, blobDigest string, blobSize ...int64) {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/blobs/uploads/%s?digest=%s", repository, uuid, blobDigest)
req, _ := http.NewRequest(http.MethodPut, url, nil)
if len(blobSize) > 0 {
req.Header.Add("Content-Length", strconv.FormatInt(blobSize[0], 10))
}
doHandle(req, func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusCreated)
})
}
func mountBlob(projectName, name, blobDigest, fromRepository string) {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/blobs/uploads/?mount=%s&from=%s", repository, blobDigest, fromRepository)
req, _ := http.NewRequest(http.MethodPost, url, nil)
doHandle(req, func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusCreated)
})
}
func deleteManifest(projectName, name, digest string, accepted ...func() bool) {
repository := fmt.Sprintf("%s/%s", projectName, name)
url := fmt.Sprintf("/v2/%s/manifests/%s", repository, digest)
req, _ := http.NewRequest(http.MethodDelete, url, nil)
next := countquota.New(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
if len(accepted) > 0 {
if accepted[0]() {
w.WriteHeader(http.StatusAccepted)
} else {
w.WriteHeader(http.StatusNotFound)
}
return
}
w.WriteHeader(http.StatusAccepted)
}))
rr := httptest.NewRecorder()
h := New(next)
h.ServeHTTP(util.NewCustomResponseWriter(rr), req)
}
func putManifest(projectName, name, tag string, manifest schema2.Manifest) {
repository := fmt.Sprintf("%s/%s", projectName, name)
buf, _ := json.Marshal(manifest)
url := fmt.Sprintf("/v2/%s/manifests/%s", repository, tag)
req, _ := http.NewRequest(http.MethodPut, url, bytes.NewReader(buf))
req.Header.Add("Content-Type", manifest.MediaType)
next := countquota.New(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusCreated)
}))
rr := httptest.NewRecorder()
h := New(next)
h.ServeHTTP(util.NewCustomResponseWriter(rr), req)
}
func pushImage(projectName, name, tag string, manifest schema2.Manifest) {
putBlobUpload(projectName, name, genUUID(), manifest.Config.Digest.String(), manifest.Config.Size)
for _, layer := range manifest.Layers {
if layer.MediaType != schema2.MediaTypeForeignLayer {
putBlobUpload(projectName, name, genUUID(), layer.Digest.String(), layer.Size)
}
}
putManifest(projectName, name, tag, manifest)
}
func withProject(f func(int64, string)) {
projectName := randomString(5)
projectID, err := dao.AddProject(models.Project{
Name: projectName,
OwnerID: 1,
})
if err != nil {
panic(err)
}
defer func() {
dao.DeleteProject(projectID)
}()
f(projectID, projectName)
}
type HandlerSuite struct {
suite.Suite
}
func (suite *HandlerSuite) checkCountUsage(expected, projectID int64) {
count, err := getProjectCountUsage(projectID)
suite.Nil(err, fmt.Sprintf("Failed to get count usage of project %d, error: %v", projectID, err))
suite.Equal(expected, count, "Failed to check count usage for project %d", projectID)
}
func (suite *HandlerSuite) checkStorageUsage(expected, projectID int64) {
value, err := getProjectStorageUsage(projectID)
suite.Nil(err, fmt.Sprintf("Failed to get storage usage of project %d, error: %v", projectID, err))
suite.Equal(expected, value, "Failed to check storage usage for project %d", projectID)
}
func (suite *HandlerSuite) TearDownTest() {
for _, table := range []string{
"artifact", "blob",
"artifact_blob", "project_blob",
"quota", "quota_usage",
} {
dao.ClearTable(table)
}
}
func (suite *HandlerSuite) TestPatchBlobUpload() {
withProject(func(projectID int64, projectName string) {
uuid := genUUID()
blobDigest := digest.FromString(randomString(15)).String()
patchBlobUpload(projectName, "photon", uuid, blobDigest, 1024)
size, err := getUploadedBlobSize(uuid)
suite.Nil(err)
suite.Equal(int64(1024), size)
})
}
func (suite *HandlerSuite) TestPutBlobUpload() {
withProject(func(projectID int64, projectName string) {
uuid := genUUID()
blobDigest := digest.FromString(randomString(15)).String()
putBlobUpload(projectName, "photon", uuid, blobDigest, 1024)
suite.checkStorageUsage(1024, projectID)
blob, err := dao.GetBlob(blobDigest)
suite.Nil(err)
suite.Equal(int64(1024), blob.Size)
})
}
func (suite *HandlerSuite) TestPutBlobUploadWithPatch() {
withProject(func(projectID int64, projectName string) {
uuid := genUUID()
blobDigest := digest.FromString(randomString(15)).String()
patchBlobUpload(projectName, "photon", uuid, blobDigest, 1024)
putBlobUpload(projectName, "photon", uuid, blobDigest)
suite.checkStorageUsage(1024, projectID)
blob, err := dao.GetBlob(blobDigest)
suite.Nil(err)
suite.Equal(int64(1024), blob.Size)
})
}
func (suite *HandlerSuite) TestMountBlob() {
withProject(func(projectID int64, projectName string) {
blobDigest := digest.FromString(randomString(15)).String()
putBlobUpload(projectName, "photon", genUUID(), blobDigest, 1024)
suite.checkStorageUsage(1024, projectID)
repository := fmt.Sprintf("%s/%s", projectName, "photon")
withProject(func(projectID int64, projectName string) {
mountBlob(projectName, "harbor", blobDigest, repository)
suite.checkStorageUsage(1024, projectID)
})
})
}
func (suite *HandlerSuite) TestPutManifestCreated() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(100, []int64{100, 100})
putBlobUpload(projectName, "photon", genUUID(), manifest.Config.Digest.String(), manifest.Config.Size)
for _, layer := range manifest.Layers {
putBlobUpload(projectName, "photon", genUUID(), layer.Digest.String(), layer.Size)
}
putManifest(projectName, "photon", "latest", manifest)
suite.checkStorageUsage(int64(300+sizeOfManifest(manifest)), projectID)
})
}
func (suite *HandlerSuite) TestDeleteManifest() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "photon", "latest", manifest)
suite.checkStorageUsage(size, projectID)
deleteManifest(projectName, "photon", digestOfManifest(manifest))
suite.checkStorageUsage(0, projectID)
})
}
func (suite *HandlerSuite) TestImageWithForeignLayers() {
withProject(func(projectID int64, projectName string) {
manifest := manifestWithAdditionalForeignLayers(makeManifest(1, []int64{2, 3, 4, 5}), []int64{6, 7})
size := sizeOfImage(manifest)
pushImage(projectName, "photon", "latest", manifest)
suite.checkStorageUsage(size, projectID)
suite.checkStorageUsage(sizeOfManifest(manifest)+1+2+3+4+5, projectID)
blobs, err := dao.GetBlobsByArtifact(digestOfManifest(manifest))
if suite.Nil(err) {
suite.Len(blobs, 8)
}
deleteManifest(projectName, "photon", digestOfManifest(manifest))
suite.checkStorageUsage(0, projectID)
})
}
func (suite *HandlerSuite) TestImageOverwrite() {
withProject(func(projectID int64, projectName string) {
manifest1 := makeManifest(1, []int64{2, 3, 4, 5})
size1 := sizeOfImage(manifest1)
pushImage(projectName, "photon", "latest", manifest1)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size1, projectID)
manifest2 := makeManifest(1, []int64{2, 3, 4, 5})
size2 := sizeOfImage(manifest2)
pushImage(projectName, "photon", "latest", manifest2)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size1+size2, projectID)
manifest3 := makeManifest(1, []int64{2, 3, 4, 5})
size3 := sizeOfImage(manifest2)
pushImage(projectName, "photon", "latest", manifest3)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size1+size2+size3, projectID)
})
}
func (suite *HandlerSuite) TestPushImageMultiTimes() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "photon", "latest", manifest)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "photon", "latest", manifest)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "photon", "latest", manifest)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size, projectID)
})
}
func (suite *HandlerSuite) TestPushImageToSameRepository() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "photon", "latest", manifest)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "photon", "dev", manifest)
suite.checkCountUsage(2, projectID)
suite.checkStorageUsage(size, projectID)
})
}
func (suite *HandlerSuite) TestPushImageToDifferentRepositories() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "mysql", "latest", manifest)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "redis", "latest", manifest)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "postgres", "latest", manifest)
suite.checkStorageUsage(size, projectID)
})
}
func (suite *HandlerSuite) TestPushImageToDifferentProjects() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "mysql", "latest", manifest)
suite.checkStorageUsage(size, projectID)
withProject(func(id int64, name string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(name, "mysql", "latest", manifest)
suite.checkStorageUsage(size, id)
suite.checkStorageUsage(size, projectID)
})
})
}
func (suite *HandlerSuite) TestDeleteManifestShareLayersInSameRepository() {
withProject(func(projectID int64, projectName string) {
manifest1 := makeManifest(1, []int64{2, 3, 4, 5})
size1 := sizeOfImage(manifest1)
pushImage(projectName, "mysql", "latest", manifest1)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size1, projectID)
manifest2 := manifestWithAdditionalLayers(manifest1, []int64{6, 7})
pushImage(projectName, "mysql", "dev", manifest2)
suite.checkCountUsage(2, projectID)
totalSize := size1 + sizeOfManifest(manifest2) + 6 + 7
suite.checkStorageUsage(totalSize, projectID)
deleteManifest(projectName, "mysql", digestOfManifest(manifest1))
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(totalSize-sizeOfManifest(manifest1), projectID)
})
}
func (suite *HandlerSuite) TestDeleteManifestShareLayersInDifferentRepositories() {
withProject(func(projectID int64, projectName string) {
manifest1 := makeManifest(1, []int64{2, 3, 4, 5})
size1 := sizeOfImage(manifest1)
pushImage(projectName, "mysql", "latest", manifest1)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size1, projectID)
pushImage(projectName, "mysql", "dev", manifest1)
suite.checkCountUsage(2, projectID)
suite.checkStorageUsage(size1, projectID)
manifest2 := manifestWithAdditionalLayers(manifest1, []int64{6, 7})
pushImage(projectName, "mariadb", "latest", manifest2)
suite.checkCountUsage(3, projectID)
totalSize := size1 + sizeOfManifest(manifest2) + 6 + 7
suite.checkStorageUsage(totalSize, projectID)
deleteManifest(projectName, "mysql", digestOfManifest(manifest1))
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(totalSize-sizeOfManifest(manifest1), projectID)
})
}
func (suite *HandlerSuite) TestDeleteManifestInSameRepository() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "photon", "latest", manifest)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "photon", "dev", manifest)
suite.checkCountUsage(2, projectID)
suite.checkStorageUsage(size, projectID)
deleteManifest(projectName, "photon", digestOfManifest(manifest))
suite.checkCountUsage(0, projectID)
suite.checkStorageUsage(0, projectID)
})
}
func (suite *HandlerSuite) TestDeleteManifestInDifferentRepositories() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "mysql", "latest", manifest)
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "mysql", "5.6", manifest)
suite.checkCountUsage(2, projectID)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "redis", "latest", manifest)
suite.checkCountUsage(3, projectID)
suite.checkStorageUsage(size, projectID)
deleteManifest(projectName, "redis", digestOfManifest(manifest))
suite.checkCountUsage(2, projectID)
suite.checkStorageUsage(size, projectID)
pushImage(projectName, "redis", "latest", manifest)
suite.checkCountUsage(3, projectID)
suite.checkStorageUsage(size, projectID)
})
}
func (suite *HandlerSuite) TestDeleteManifestInDifferentProjects() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "mysql", "latest", manifest)
suite.checkStorageUsage(size, projectID)
withProject(func(id int64, name string) {
pushImage(name, "mysql", "latest", manifest)
suite.checkStorageUsage(size, id)
suite.checkStorageUsage(size, projectID)
deleteManifest(projectName, "mysql", digestOfManifest(manifest))
suite.checkCountUsage(0, projectID)
suite.checkStorageUsage(0, projectID)
})
})
}
func (suite *HandlerSuite) TestPushDeletePush() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
pushImage(projectName, "photon", "latest", manifest)
suite.checkStorageUsage(size, projectID)
deleteManifest(projectName, "photon", digestOfManifest(manifest))
suite.checkStorageUsage(0, projectID)
pushImage(projectName, "photon", "latest", manifest)
suite.checkStorageUsage(size, projectID)
})
}
func (suite *HandlerSuite) TestPushImageRace() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
size := sizeOfImage(manifest)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
defer wg.Done()
pushImage(projectName, "photon", "latest", manifest)
}()
}
wg.Wait()
suite.checkCountUsage(1, projectID)
suite.checkStorageUsage(size, projectID)
})
}
func (suite *HandlerSuite) TestDeleteImageRace() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
pushImage(projectName, "photon", "latest", manifest)
count := 100
size := sizeOfImage(manifest)
for i := 0; i < count; i++ {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
pushImage(projectName, "mysql", fmt.Sprintf("tag%d", i), manifest)
size += sizeOfImage(manifest)
}
suite.checkCountUsage(int64(count+1), projectID)
suite.checkStorageUsage(size, projectID)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
deleteManifest(projectName, "photon", digestOfManifest(manifest), func() bool {
return i == 0
})
}(i)
}
wg.Wait()
suite.checkCountUsage(int64(count), projectID)
suite.checkStorageUsage(size-sizeOfImage(manifest), projectID)
})
}
func (suite *HandlerSuite) TestDisableProjectQuota() {
withProject(func(projectID int64, projectName string) {
manifest := makeManifest(1, []int64{2, 3, 4, 5})
pushImage(projectName, "photon", "latest", manifest)
quotas, err := dao.ListQuotas(&models.QuotaQuery{
Reference: "project",
ReferenceID: strconv.FormatInt(projectID, 10),
})
suite.Nil(err)
suite.Len(quotas, 1)
})
withProject(func(projectID int64, projectName string) {
cfg := config.GetCfgManager()
cfg.Set(common.QuotaPerProjectEnable, false)
defer cfg.Set(common.QuotaPerProjectEnable, true)
manifest := makeManifest(1, []int64{2, 3, 4, 5})
pushImage(projectName, "photon", "latest", manifest)
quotas, err := dao.ListQuotas(&models.QuotaQuery{
Reference: "project",
ReferenceID: strconv.FormatInt(projectID, 10),
})
suite.Nil(err)
suite.Len(quotas, 0)
})
}
func TestMain(m *testing.M) {
config.Init()
dao.PrepareTestForPostgresSQL()
if result := m.Run(); result != 0 {
os.Exit(result)
}
}
func TestRunHandlerSuite(t *testing.T) {
suite.Run(t, new(HandlerSuite))
}

334
src/core/middlewares/sizequota/util.go
View File

@ -1,334 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sizequota
import (
"errors"
"fmt"
"net/http"
"regexp"
"strconv"
"strings"
"time"
"github.com/garyburd/redigo/redis"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/goharbor/harbor/src/pkg/types"
"github.com/opencontainers/go-digest"
)
var (
blobUploadURLRe = regexp.MustCompile(`^/v2/((?:[a-z0-9]+(?:[._-][a-z0-9]+)*/)+)blobs/uploads/([a-zA-Z0-9-_.=]+)/?$`)
initiateBlobUploadURLRe = regexp.MustCompile(`^/v2/((?:[a-z0-9]+(?:[._-][a-z0-9]+)*/)+)blobs/uploads/?$`)
)
// parseUploadedBlobSize parse the blob stream upload response and return the size blob uploaded
func parseUploadedBlobSize(w http.ResponseWriter) (int64, error) {
// Range: Range indicating the current progress of the upload.
// https://github.com/opencontainers/distribution-spec/blob/master/spec.md#get-blob-upload
r := w.Header().Get("Range")
if r == "" {
return 0, errors.New("range header not found")
}
parts := strings.SplitN(r, "-", 2)
if len(parts) != 2 {
return 0, fmt.Errorf("range header bad value: %s", r)
}
size, err := strconv.ParseInt(parts[1], 10, 64)
if err != nil {
return 0, err
}
// docker registry did '-1' in the response
if size > 0 {
size = size + 1
}
return size, nil
}
// setUploadedBlobSize update the size of stream upload blob
func setUploadedBlobSize(uuid string, size int64) (bool, error) {
conn, err := util.GetRegRedisCon()
if err != nil {
return false, err
}
defer conn.Close()
key := fmt.Sprintf("upload:%s:size", uuid)
reply, err := redis.String(conn.Do("SET", key, size))
if err != nil {
return false, err
}
return reply == "OK", nil
}
// getUploadedBlobSize returns the size of stream upload blob
func getUploadedBlobSize(uuid string) (int64, error) {
conn, err := util.GetRegRedisCon()
if err != nil {
return 0, err
}
defer conn.Close()
key := fmt.Sprintf("upload:%s:size", uuid)
size, err := redis.Int64(conn.Do("GET", key))
if err != nil {
return 0, err
}
return size, nil
}
// parseBlobSize returns blob size from blob upload complete request
func parseBlobSize(req *http.Request, uuid string) (int64, error) {
size, err := strconv.ParseInt(req.Header.Get("Content-Length"), 10, 64)
if err == nil && size != 0 {
return size, nil
}
return getUploadedBlobSize(uuid)
}
// match returns true if request method equal method and path match re
func match(req *http.Request, method string, re *regexp.Regexp) bool {
return req.Method == method && re.MatchString(req.URL.Path)
}
// parseBlobInfoFromComplete returns blob info from blob upload complete request
func parseBlobInfoFromComplete(req *http.Request) (*util.BlobInfo, error) {
if !match(req, http.MethodPut, blobUploadURLRe) {
return nil, fmt.Errorf("not match url %s for blob upload complete", req.URL.Path)
}
s := blobUploadURLRe.FindStringSubmatch(req.URL.Path)
repository, uuid := s[1][:len(s[1])-1], s[2]
projectName, _ := utils.ParseRepository(repository)
project, err := dao.GetProjectByName(projectName)
if err != nil {
return nil, fmt.Errorf("failed to get project %s, error: %v", projectName, err)
}
if project == nil {
return nil, fmt.Errorf("project %s not found", projectName)
}
dgt, err := digest.Parse(req.FormValue("digest"))
if err != nil {
return nil, fmt.Errorf("blob digest invalid for upload %s", uuid)
}
size, err := parseBlobSize(req, uuid)
if err != nil {
return nil, fmt.Errorf("failed to get content length of blob upload %s, error: %v", uuid, err)
}
return &util.BlobInfo{
ProjectID: project.ProjectID,
Repository: repository,
Digest: dgt.String(),
Size: size,
}, nil
}
// parseBlobInfoFromManifest returns blob info from put the manifest request
func parseBlobInfoFromManifest(req *http.Request) (*util.BlobInfo, error) {
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
manifest, err := util.ParseManifestInfoFromReq(req)
if err != nil {
return nil, err
}
info = manifest
// replace the request with manifest info
*req = *(req.WithContext(util.NewManifestInfoContext(req.Context(), info)))
}
return &util.BlobInfo{
ProjectID: info.ProjectID,
Repository: info.Repository,
Digest: info.Descriptor.Digest.String(),
Size: info.Descriptor.Size,
ContentType: info.Descriptor.MediaType,
}, nil
}
// parseBlobInfoFromMount returns blob info from blob mount request
func parseBlobInfoFromMount(req *http.Request) (*util.BlobInfo, error) {
if !match(req, http.MethodPost, initiateBlobUploadURLRe) {
return nil, fmt.Errorf("not match url %s for mount blob", req.URL.Path)
}
if req.FormValue("mount") == "" || req.FormValue("from") == "" {
return nil, fmt.Errorf("not match url %s for mount blob", req.URL.Path)
}
dgt, err := digest.Parse(req.FormValue("mount"))
if err != nil {
return nil, errors.New("mount must be digest")
}
s := initiateBlobUploadURLRe.FindStringSubmatch(req.URL.Path)
repository := strings.TrimSuffix(s[1], "/")
projectName, _ := utils.ParseRepository(repository)
project, err := dao.GetProjectByName(projectName)
if err != nil {
return nil, fmt.Errorf("failed to get project %s, error: %v", projectName, err)
}
if project == nil {
return nil, fmt.Errorf("project %s not found", projectName)
}
blob, err := dao.GetBlob(dgt.String())
if err != nil {
return nil, fmt.Errorf("failed to get blob %s, error: %v", dgt.String(), err)
}
if blob == nil {
return nil, fmt.Errorf("the blob in the mount request with digest: %s doesn't exist", dgt.String())
}
return &util.BlobInfo{
ProjectID: project.ProjectID,
Repository: repository,
Digest: dgt.String(),
Size: blob.Size,
}, nil
}
// getBlobInfoParser return parse blob info function for request
// returns parseBlobInfoFromComplete when request match PUT /v2/<name>/blobs/uploads/<uuid>?digest=<digest>
// returns parseBlobInfoFromMount when request match POST /v2/<name>/blobs/uploads/?mount=<digest>&from=<repository name>
func getBlobInfoParser(req *http.Request) func(*http.Request) (*util.BlobInfo, error) {
if match(req, http.MethodPut, blobUploadURLRe) {
if req.FormValue("digest") != "" {
return parseBlobInfoFromComplete
}
}
if match(req, http.MethodPost, initiateBlobUploadURLRe) {
if req.FormValue("mount") != "" && req.FormValue("from") != "" {
return parseBlobInfoFromMount
}
}
return nil
}
// computeResourcesForBlob returns storage required for blob, no storage required if blob exists in project
func computeResourcesForBlob(req *http.Request) (types.ResourceList, error) {
info, ok := util.BlobInfoFromContext(req.Context())
if !ok {
return nil, errors.New("blob info missing")
}
exist, err := info.BlobExists()
if err != nil {
return nil, err
}
if exist {
return nil, nil
}
return types.ResourceList{types.ResourceStorage: info.Size}, nil
}
// computeResourcesForManifestCreation returns storage resource required for manifest
// no storage required if manifest exists in project
// the sum size of manifest itself and blobs not in project will return if manifest not exists in project
func computeResourcesForManifestCreation(req *http.Request) (types.ResourceList, error) {
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
return nil, errors.New("manifest info missing")
}
exist, err := info.ManifestExists()
if err != nil {
return nil, err
}
// manifest exist in project, so no storage quota required
if exist {
return nil, nil
}
blobs, err := info.GetBlobsNotInProject()
if err != nil {
return nil, err
}
size := info.Descriptor.Size
for _, blob := range blobs {
if !blob.IsForeignLayer() {
size += blob.Size
}
}
return types.ResourceList{types.ResourceStorage: size}, nil
}
// computeResourcesForManifestDeletion returns storage resource will be released when manifest deleted
// then result will be the sum of manifest itself and blobs which will not be used by other manifests of project
func computeResourcesForManifestDeletion(req *http.Request) (types.ResourceList, error) {
info, ok := util.ManifestInfoFromContext(req.Context())
if !ok {
return nil, errors.New("manifest info missing")
}
blobs, err := dao.GetExclusiveBlobs(info.ProjectID, info.Repository, info.Digest)
if err != nil {
return nil, err
}
info.ExclusiveBlobs = blobs
var size int64
for _, blob := range blobs {
if !blob.IsForeignLayer() {
size = size + blob.Size
}
}
return types.ResourceList{types.ResourceStorage: size}, nil
}
// syncBlobInfoToProject create the blob and add it to project
func syncBlobInfoToProject(info *util.BlobInfo) error {
_, blob, err := dao.GetOrCreateBlob(&models.Blob{
Digest: info.Digest,
ContentType: info.ContentType,
Size: info.Size,
CreationTime: time.Now(),
})
if err != nil {
return err
}
if _, err := dao.AddBlobToProject(blob.ID, info.ProjectID); err != nil {
return err
}
return nil
}

59
src/core/middlewares/sizequota/util_test.go
View File

@ -1,59 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sizequota
import (
"net/http"
"net/http/httptest"
"testing"
)
func Test_parseUploadedBlobSize(t *testing.T) {
writer := func(header string) http.ResponseWriter {
rr := httptest.NewRecorder()
if header != "" {
rr.Header().Add("Range", header)
}
return rr
}
type args struct {
w http.ResponseWriter
}
tests := []struct {
name string
args args
want int64
wantErr bool
}{
{"success", args{writer("0-99")}, 100, false},
{"ranage header not found", args{writer("")}, 0, true},
{"ranage header bad value", args{writer("0")}, 0, true},
{"ranage header bad value", args{writer("0-")}, 0, true},
{"ranage header bad value", args{writer("0-a")}, 0, true},
{"ranage header bad value", args{writer("0-1-2")}, 0, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := parseUploadedBlobSize(tt.args.w)
if (err != nil) != tt.wantErr {
t.Errorf("parseUploadedBlobSize() error = %v, wantErr %v", err, tt.wantErr)
return
}
if got != tt.want {
t.Errorf("parseUploadedBlobSize() = %v, want %v", got, tt.want)
}
})
}
}

110
src/core/middlewares/url/handler.go
View File

@ -1,110 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package url
import (
"context"
"fmt"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/util"
coreutils "github.com/goharbor/harbor/src/core/utils"
"github.com/opencontainers/go-digest"
"net/http"
"regexp"
"strings"
)
var (
urlPatterns = []*regexp.Regexp{
util.ManifestURLRe, util.TagListURLRe, util.BlobURLRe, util.BlobUploadURLRe,
}
)
// urlHandler extracts the artifact info from the url of request to V2 handler and propagates it to context
type urlHandler struct {
next http.Handler
}
// New ...
func New(next http.Handler) http.Handler {
return &urlHandler{
next: next,
}
}
// ServeHTTP ...
func (uh urlHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
path := req.URL.Path
log.Debugf("in url handler, path: %s", path)
m, ok := parse(path)
if !ok {
uh.next.ServeHTTP(rw, req)
}
repo := m[util.RepositorySubexp]
components := strings.SplitN(repo, "/", 2)
if len(components) < 2 {
http.Error(rw, util.MarshalError("PROJECT_POLICY_VIOLATION", fmt.Sprintf("Bad repository name: %s", repo)), http.StatusBadRequest)
return
}
art := util.ArtifactInfo{
Repository: repo,
ProjectName: components[0],
}
if digest, ok := m[util.DigestSubexp]; ok {
art.Digest = digest
}
if ref, ok := m[util.ReferenceSubexp]; ok {
art.Reference = ref
}
if util.ManifestURLRe.MatchString(path) && req.Method == http.MethodGet { // Request for pulling manifest
client, err := coreutils.NewRepositoryClientForUI(util.TokenUsername, art.Repository)
if err != nil {
log.Errorf("Error creating repository Client: %v", err)
http.Error(rw, util.MarshalError("PROJECT_POLICY_VIOLATION", fmt.Sprintf("Failed due to internal Error: %v", err)), http.StatusInternalServerError)
return
}
digest, _, err := client.ManifestExist(art.Reference)
if err != nil {
log.Errorf("Failed to get digest for reference: %s, error: %v", art.Reference, err)
http.Error(rw, util.MarshalError("PROJECT_POLICY_VIOLATION", fmt.Sprintf("Failed due to internal Error: %v", err)), http.StatusInternalServerError)
return
}
art.Digest = digest
log.Debugf("artifact info of the request: %#v", art)
ctx := context.WithValue(req.Context(), util.ArtifactInfoCtxKey, art)
req = req.WithContext(ctx)
}
uh.next.ServeHTTP(rw, req)
}
func parse(urlPath string) (map[string]string, bool) {
m := make(map[string]string)
match := false
for _, re := range urlPatterns {
l := re.FindStringSubmatch(urlPath)
if len(l) > 0 {
match = true
for i := 1; i < len(l); i++ {
m[re.SubexpNames()[i]] = l[i]
}
}
}
if digest.DigestRegexp.MatchString(m[util.ReferenceSubexp]) {
m[util.DigestSubexp] = m[util.ReferenceSubexp]
}
return m, match
}

101
src/core/middlewares/url/handler_test.go
View File

@ -1,101 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License
package url
import (
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/stretchr/testify/assert"
"os"
"testing"
)
func TestMain(m *testing.M) {
if result := m.Run(); result != 0 {
os.Exit(result)
}
}
func TestParseURL(t *testing.T) {
cases := []struct {
input string
expect map[string]string
match bool
}{
{
input: "/api/projects",
expect: map[string]string{},
match: false,
},
{
input: "/v2/_catalog",
expect: map[string]string{},
match: false,
},
{
input: "/v2/no-project-repo/tags/list",
expect: map[string]string{
util.RepositorySubexp: "no-project-repo",
},
match: true,
},
{
input: "/v2/development/golang/manifests/sha256:08e4a417ff4e3913d8723a05cc34055db01c2fd165b588e049c5bad16ce6094f",
expect: map[string]string{
util.RepositorySubexp: "development/golang",
util.ReferenceSubexp: "sha256:08e4a417ff4e3913d8723a05cc34055db01c2fd165b588e049c5bad16ce6094f",
util.DigestSubexp: "sha256:08e4a417ff4e3913d8723a05cc34055db01c2fd165b588e049c5bad16ce6094f",
},
match: true,
},
{
input: "/v2/development/golang/manifests/shaxxx:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
expect: map[string]string{},
match: false,
},
{
input: "/v2/multi/sector/repository/blobs/sha256:08e4a417ff4e3913d8723a05cc34055db01c2fd165b588e049c5bad16ce6094f",
expect: map[string]string{
util.RepositorySubexp: "multi/sector/repository",
util.DigestSubexp: "sha256:08e4a417ff4e3913d8723a05cc34055db01c2fd165b588e049c5bad16ce6094f",
},
match: true,
},
{
input: "/v2/blobs/uploads",
expect: map[string]string{},
match: false,
},
{
input: "/v2/library/ubuntu/blobs/uploads",
expect: map[string]string{
util.RepositorySubexp: "library/ubuntu",
},
match: true,
},
{
input: "/v2/library/centos/blobs/uploads/u-12345",
expect: map[string]string{
util.RepositorySubexp: "library/centos",
},
match: true,
},
}
for _, c := range cases {
e, m := parse(c.input)
assert.Equal(t, c.match, m)
assert.Equal(t, c.expect, e)
}
}

29
src/core/middlewares/util/error/immutable.go
View File

@ -1,29 +0,0 @@
package error
import (
"fmt"
)
// ErrImmutable ...
type ErrImmutable struct {
repo string
tag string
}
// Error ...
func (ei *ErrImmutable) Error() string {
return fmt.Sprintf("Failed to process request due to '%s:%s' configured as immutable.", ei.repo, ei.tag)
}
// Unwrap ...
func (ei *ErrImmutable) Unwrap() error {
return nil
}
// NewErrImmutable ...
func NewErrImmutable(msg, tag string) error {
return &ErrImmutable{
repo: msg,
tag: tag,
}
}

141
src/core/middlewares/vulnerable/handler.go
View File

@ -1,141 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vulnerable
import (
"net/http"
"net/http/httptest"
sc "github.com/goharbor/harbor/src/api/scan"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/middlewares/util"
"github.com/goharbor/harbor/src/pkg/scan/report"
v1 "github.com/goharbor/harbor/src/pkg/scan/rest/v1"
"github.com/goharbor/harbor/src/pkg/scan/vuln"
"github.com/pkg/errors"
)
type vulnerableHandler struct {
next http.Handler
}
// New ...
func New(next http.Handler) http.Handler {
return &vulnerableHandler{
next: next,
}
}
// ServeHTTP ...
func (vh vulnerableHandler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
doVulCheck, img, projectVulnerableSeverity, wl := validate(req)
if !doVulCheck {
vh.next.ServeHTTP(rw, req)
return
}
rec := httptest.NewRecorder()
vh.next.ServeHTTP(rec, req)
// only enable vul policy check the response 200
if rec.Result().StatusCode == http.StatusOK {
// Invalid project ID
if wl.ProjectID == 0 {
err := errors.Errorf("project verification error: project %s", img.ProjectName)
vh.sendError(err, rw)
return
}
// Get the vulnerability summary
artifact := &v1.Artifact{
NamespaceID: wl.ProjectID,
Repository: img.Repository,
Tag: img.Reference,
Digest: img.Digest,
MimeType: v1.MimeTypeDockerArtifact,
}
cve := report.CVESet(wl.CVESet())
summaries, err := sc.DefaultController.GetSummary(
artifact,
[]string{v1.MimeTypeNativeReport},
report.WithCVEWhitelist(&cve),
)
if err != nil {
err = errors.Wrap(err, "middleware: vulnerable handler")
vh.sendError(err, rw)
return
}
rawSummary, ok := summaries[v1.MimeTypeNativeReport]
// No report yet?
if !ok {
err = errors.Errorf("no scan report existing for the artifact: %s:%s@%s", img.Repository, img.Reference, img.Digest)
vh.sendError(err, rw)
return
}
summary := rawSummary.(*vuln.NativeReportSummary)
// Do judgement
if summary.Severity.Code() >= projectVulnerableSeverity.Code() {
err = errors.Errorf("current image with '%q vulnerable' cannot be pulled due to configured policy in 'Prevent images with vulnerability severity of %q from running.' "+
"Please contact your project administrator for help'", summary.Severity, projectVulnerableSeverity)
vh.sendError(err, rw)
return
}
// Print scannerPull CVE list
if len(summary.CVEBypassed) > 0 {
for _, cve := range summary.CVEBypassed {
log.Infof("Vulnerable policy check: scannerPull CVE %s", cve)
}
}
}
util.CopyResp(rec, rw)
}
func validate(req *http.Request) (bool, util.ArtifactInfo, vuln.Severity, models.CVEWhitelist) {
var vs vuln.Severity
var wl models.CVEWhitelist
var img util.ArtifactInfo
imgRaw := req.Context().Value(util.ArtifactInfoCtxKey)
if imgRaw == nil {
return false, img, vs, wl
}
// Expected artifact specified?
img, ok := imgRaw.(util.ArtifactInfo)
if !ok || len(img.Digest) == 0 {
return false, img, vs, wl
}
if scannerPull, ok := util.ScannerPullFromContext(req.Context()); ok && scannerPull {
return false, img, vs, wl
}
// Is vulnerable policy set?
projectVulnerableEnabled, projectVulnerableSeverity, wl := util.GetPolicyChecker().VulnerablePolicy(img.ProjectName)
if !projectVulnerableEnabled {
return false, img, vs, wl
}
return true, img, projectVulnerableSeverity, wl
}
func (vh vulnerableHandler) sendError(err error, rw http.ResponseWriter) {
log.Error(err)
http.Error(rw, util.MarshalError("PROJECT_POLICY_VIOLATION", err.Error()), http.StatusPreconditionFailed)
}

308
src/core/service/notifications/registry/handler.go
View File

@ -1,308 +0,0 @@
// Copyright 2018 Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package registry
import (
"encoding/json"
"regexp"
"strconv"
"strings"
"time"
"github.com/goharbor/harbor/src/core/service/notifications"
"github.com/goharbor/harbor/src/api/scan"
"github.com/goharbor/harbor/src/api/scanner"
"github.com/goharbor/harbor/src/common/dao"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/core/config"
coreutils "github.com/goharbor/harbor/src/core/utils"
notifierEvt "github.com/goharbor/harbor/src/pkg/notifier/event"
v1 "github.com/goharbor/harbor/src/pkg/scan/rest/v1"
"github.com/goharbor/harbor/src/replication"
"github.com/goharbor/harbor/src/replication/adapter"
repevent "github.com/goharbor/harbor/src/replication/event"
"github.com/goharbor/harbor/src/replication/model"
"github.com/pkg/errors"
)
// NotificationHandler handles request on /service/notifications/, which listens to registry's events.
type NotificationHandler struct {
notifications.BaseHandler
}
const manifestPattern = `^application/vnd.docker.distribution.manifest.v\d\+(json|prettyjws)`
// Post handles POST request, and records audit log or refreshes cache based on event.
func (n *NotificationHandler) Post() {
var notification models.Notification
err := json.Unmarshal(n.Ctx.Input.CopyBody(1<<32), &notification)
if err != nil {
log.Errorf("failed to decode notification: %v", err)
return
}
events, err := filterEvents(&notification)
if err != nil {
log.Errorf("failed to filter events: %v", err)
return
}
for _, event := range events {
repository := event.Target.Repository
project, _ := utils.ParseRepository(repository)
tag := event.Target.Tag
action := event.Action
user := event.Actor.Name
if len(user) == 0 {
user = "anonymous"
}
pro, err := config.GlobalProjectMgr.Get(project)
if err != nil {
log.Errorf("failed to get project by name %s: %v", project, err)
return
}
if pro == nil {
log.Warningf("project %s not found", project)
continue
}
go func() {
if err := dao.AddAccessLog(models.AccessLog{
Username: user,
ProjectID: pro.ProjectID,
RepoName: repository,
RepoTag: tag,
Operation: action,
OpTime: time.Now(),
}); err != nil {
log.Errorf("failed to add access log: %v", err)
}
}()
if action == "push" {
// discard the notification without tag.
if tag != "" {
go func() {
exist := dao.RepositoryExists(repository)
if exist {
return
}
log.Debugf("Add repository %s into DB.", repository)
repoRecord := models.RepoRecord{
Name: repository,
ProjectID: pro.ProjectID,
}
if err := dao.AddRepository(repoRecord); err != nil {
log.Errorf("Error happens when adding repository: %v", err)
}
}()
}
if !coreutils.WaitForManifestReady(repository, tag, 6) {
log.Errorf("Manifest for image %s:%s is not ready, skip the follow up actions.", repository, tag)
return
}
// build and publish image push event
evt := &notifierEvt.Event{}
imgPushMetadata := &notifierEvt.ImagePushMetaData{
Project: pro,
Tag: tag,
Digest: event.Target.Digest,
RepoName: event.Target.Repository,
OccurAt: time.Now(),
Operator: event.Actor.Name,
}
if err := evt.Build(imgPushMetadata); err == nil {
if err := evt.Publish(); err != nil {
// do not return when publishing event failed
log.Errorf("failed to publish image push event: %v", err)
}
} else {
// do not return when building event metadata failed
log.Errorf("failed to build image push event metadata: %v", err)
}
go func() {
e := &repevent.Event{
Type: repevent.EventTypeImagePush,
Resource: &model.Resource{
Type: model.ResourceTypeImage,
Metadata: &model.ResourceMetadata{
Repository: &model.Repository{
Name: repository,
Metadata: map[string]interface{}{
"public": strconv.FormatBool(pro.IsPublic()),
},
},
Vtags: []string{tag},
},
},
}
if err := replication.EventHandler.Handle(e); err != nil {
log.Errorf("failed to handle event: %v", err)
}
}()
if autoScanEnabled(pro) {
artifact := &v1.Artifact{
NamespaceID: pro.ProjectID,
Repository: repository,
Tag: tag,
MimeType: v1.MimeTypeDockerArtifact,
Digest: event.Target.Digest,
}
if err := scan.DefaultController.Scan(artifact); err != nil {
log.Error(errors.Wrap(err, "registry notification: trigger scan when pushing automatically"))
}
}
}
if action == "pull" {
// build and publish image pull event
evt := &notifierEvt.Event{}
imgPullMetadata := &notifierEvt.ImagePullMetaData{
Project: pro,
Tag: tag,
Digest: event.Target.Digest,
RepoName: event.Target.Repository,
OccurAt: time.Now(),
Operator: event.Actor.Name,
}
if err := evt.Build(imgPullMetadata); err == nil {
if err := evt.Publish(); err != nil {
// do not return when publishing event failed
log.Errorf("failed to publish image pull event: %v", err)
}
} else {
// do not return when building event metadata failed
log.Errorf("failed to build image push event metadata: %v", err)
}
go func() {
log.Debugf("Increase the repository %s pull count.", repository)
if err := dao.IncreasePullCount(repository); err != nil {
log.Errorf("Error happens when increasing pull count: %v", repository)
}
}()
// update the artifact pull time, and ignore the events without tag.
if tag != "" {
go func() {
artifactQuery := &models.ArtifactQuery{
PID: pro.ProjectID,
Repo: repository,
}
// handle pull by tag or digest
pullByDigest := utils.IsDigest(tag)
if pullByDigest {
artifactQuery.Digest = tag
} else {
artifactQuery.Tag = tag
}
afs, err := dao.ListArtifacts(artifactQuery)
if err != nil {
log.Errorf("Error occurred when to get artifact %v", err)
return
}
if len(afs) > 0 {
log.Warningf("get multiple artifact records when to update pull time with query :%d-%s-%s, "+
"all of them will be updated.", artifactQuery.PID, artifactQuery.Repo, artifactQuery.Tag)
}
// ToDo: figure out how to do batch update in Pg as beego orm doesn't support update multiple like insert does.
for _, af := range afs {
log.Debugf("Update the artifact: %s pull time.", af.Repo)
af.PullTime = time.Now()
if err := dao.UpdateArtifactPullTime(af); err != nil {
log.Errorf("Error happens when updating the pull time of artifact: %d-%s, with err: %v",
artifactQuery.PID, artifactQuery.Repo, err)
}
}
}()
}
}
}
}
func filterEvents(notification *models.Notification) ([]*models.Event, error) {
events := make([]*models.Event, 0)
for _, event := range notification.Events {
log.Debugf("receive an event: \n----ID: %s \n----target: %s:%s \n----digest: %s \n----action: %s \n----mediatype: %s \n----user-agent: %s", event.ID, event.Target.Repository,
event.Target.Tag, event.Target.Digest, event.Action, event.Target.MediaType, event.Request.UserAgent)
isManifest, err := regexp.MatchString(manifestPattern, event.Target.MediaType)
if err != nil {
log.Errorf("failed to match the media type against pattern: %v", err)
continue
}
if !isManifest {
continue
}
if checkEvent(&event) {
events = append(events, &event)
log.Debugf("add event to collection: %s", event.ID)
continue
}
}
return events, nil
}
func checkEvent(event *models.Event) bool {
// push action
if event.Action == "push" {
return true
}
// if it is pull action, check the user-agent
userAgent := strings.ToLower(strings.TrimSpace(event.Request.UserAgent))
if userAgent == "harbor-registry-client" || userAgent == strings.ToLower(adapter.UserAgentReplication) {
return false
}
return true
}
func autoScanEnabled(project *models.Project) bool {
r, err := scanner.DefaultController.GetRegistrationByProject(project.ProjectID)
if err != nil {
log.Error(errors.Wrap(err, "check auto scan enable"))
return false
}
// In case
if r == nil {
log.Errorf("no scanner is available for project: %s", project.Name)
return false
}
return !r.Disabled && project.AutoScan()
}
// Render returns nil as it won't render any template.
func (n *NotificationHandler) Render() error {
return nil
}

95
src/core/utils/retag.go
View File

@ -1,95 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package utils
import (
"fmt"
"github.com/goharbor/harbor/src/common/models"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/common/utils/registry"
"github.com/docker/distribution/manifest/schema1"
"github.com/docker/distribution/manifest/schema2"
)
// Retag tags an image to another
func Retag(srcImage, destImage *models.Image) error {
isSameRepo := getRepoName(srcImage) == getRepoName(destImage)
srcClient, err := NewRepositoryClientForLocal("harbor-ui", getRepoName(srcImage))
if err != nil {
return err
}
destClient := srcClient
if !isSameRepo {
destClient, err = NewRepositoryClientForLocal("harbor-ui", getRepoName(destImage))
if err != nil {
return err
}
}
_, exist, err := srcClient.ManifestExist(srcImage.Tag)
if err != nil {
log.Errorf("check existence of manifest '%s:%s' error: %v", srcClient.Name, srcImage.Tag, err)
return err
}
if !exist {
log.Errorf("source image %s:%s not found", srcClient.Name, srcImage.Tag)
return fmt.Errorf("image %s:%s not found", srcClient.Name, srcImage.Tag)
}
accepted := []string{schema1.MediaTypeManifest, schema2.MediaTypeManifest}
digest, mediaType, payload, err := srcClient.PullManifest(srcImage.Tag, accepted)
if err != nil {
return err
}
manifest, _, err := registry.UnMarshal(mediaType, payload)
if err != nil {
return err
}
destDigest, exist, err := destClient.ManifestExist(destImage.Tag)
if err != nil {
log.Errorf("check existence of manifest '%s:%s' error: %v", destClient.Name, destImage.Tag, err)
return err
}
if exist && destDigest == digest {
log.Infof("manifest of '%s:%s' already exist", destClient.Name, destImage.Tag)
return nil
}
if !isSameRepo {
for _, descriptor := range manifest.References() {
err := destClient.MountBlob(descriptor.Digest.String(), srcClient.Name)
if err != nil {
log.Errorf("mount blob '%s' error: %v", descriptor.Digest.String(), err)
return err
}
}
}
if _, err = destClient.PushManifest(destImage.Tag, mediaType, payload); err != nil {
log.Errorf("push manifest '%s:%s' error: %v", destClient.Name, destImage.Tag, err)
return err
}
return nil
}
func getRepoName(image *models.Image) string {
return fmt.Sprintf("%s/%s", image.Project, image.Repo)
}

44
src/core/utils/utils.go
View File

@ -16,14 +16,10 @@
package utils
import (
"net/http"
"os"
"time"
"github.com/goharbor/harbor/src/common/utils/log"
"github.com/goharbor/harbor/src/common/utils/registry"
"github.com/goharbor/harbor/src/common/utils/registry/auth"
"github.com/goharbor/harbor/src/core/config"
"net/http"
)
// NewRepositoryClientForUI creates a repository client that can only be used to
@ -36,16 +32,6 @@ func NewRepositoryClientForUI(username, repository string) (*registry.Repository
return newRepositoryClient(endpoint, username, repository)
}
// NewRepositoryClientForLocal creates a repository client that can only be used to
// access the internal registry with 127.0.0.1
func NewRepositoryClientForLocal(username, repository string) (*registry.Repository, error) {
// The 127.0.0.1:8080 is not reachable as we do not enable core in UT env.
if os.Getenv("UTTEST") == "true" {
return NewRepositoryClientForUI(username, repository)
}
return newRepositoryClient(config.LocalCoreURL(), username, repository)
}
func newRepositoryClient(endpoint, username, repository string) (*registry.Repository, error) {
uam := &auth.UserAgentModifier{
UserAgent: "harbor-registry-client",
@ -57,31 +43,3 @@ func newRepositoryClient(endpoint, username, repository string) (*registry.Repos
}
return registry.NewRepository(repository, endpoint, client)
}
// WaitForManifestReady implements exponential sleep to wait until manifest is ready in registry.
// This is a workaround for https://github.com/docker/distribution/issues/2625
func WaitForManifestReady(repository string, tag string, maxRetry int) bool {
// The initial wait interval, hard-coded to 80ms, interval will be 80ms,200ms,500ms,1.25s,3.124999936s
interval := 80 * time.Millisecond
repoClient, err := NewRepositoryClientForUI("harbor-core", repository)
if err != nil {
log.Errorf("Failed to create repo client.")
return false
}
for i := 0; i < maxRetry; i++ {
if i != 0 {
log.Warningf("manifest for image %s:%s is not ready, retry after %v", repository, tag, interval)
time.Sleep(interval)
interval = time.Duration(int64(float32(interval) * 2.5))
}
_, exist, err := repoClient.ManifestExist(tag)
if err != nil {
log.Errorf("Unexpected error when checking manifest existence, image: %s:%s, error: %v", repository, tag, err)
continue
}
if exist {
return true
}
}
return false
}

28
src/core/utils/utils_test.go
View File

@ -1,28 +0,0 @@
// Copyright Project Harbor Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package utils
import (
"os"
"testing"
"github.com/goharbor/harbor/src/core/config"
)
func TestMain(m *testing.M) {
config.Init()
rc := m.Run()
os.Exit(rc)
}

3
src/server/route.go
View File

@ -22,7 +22,6 @@ import (
"github.com/goharbor/harbor/src/core/controllers"
"github.com/goharbor/harbor/src/core/service/notifications/admin"
"github.com/goharbor/harbor/src/core/service/notifications/jobs"
"github.com/goharbor/harbor/src/core/service/notifications/registry"
"github.com/goharbor/harbor/src/core/service/notifications/scheduler"
"github.com/goharbor/harbor/src/core/service/token"
)
@ -39,12 +38,10 @@ func registerRoutes() {
beego.Router(common.OIDCCallbackPath, &controllers.OIDCController{}, "get:Callback")
beego.Router("/api/internal/configurations", &api.ConfigAPI{}, "get:GetInternalConfig;put:Put")
beego.Router("/api/internal/syncregistry", &api.InternalAPI{}, "post:SyncRegistry")
beego.Router("/api/internal/renameadmin", &api.InternalAPI{}, "post:RenameAdmin")
beego.Router("/api/internal/switchquota", &api.InternalAPI{}, "put:SwitchQuota")
beego.Router("/api/internal/syncquota", &api.InternalAPI{}, "post:SyncQuota")
beego.Router("/service/notifications", &registry.NotificationHandler{})
beego.Router("/service/notifications/jobs/adminjob/:id([0-9]+)", &admin.Handler{}, "post:HandleAdminJob")
beego.Router("/service/notifications/jobs/replication/:id([0-9]+)", &jobs.Handler{}, "post:HandleReplicationScheduleJob")
beego.Router("/service/notifications/jobs/replication/task/:id([0-9]+)", &jobs.Handler{}, "post:HandleReplicationTask")

17
src/server/v2.0/route/legacy.go
View File

@ -54,18 +54,6 @@ func registerLegacyRoutes() {
beego.Router("/api/"+version+"/quotas", &api.QuotaAPI{}, "get:List")
beego.Router("/api/"+version+"/quotas/:id([0-9]+)", &api.QuotaAPI{}, "get:Get;put:Put")
beego.Router("/api/"+version+"/repositories", &api.RepositoryAPI{}, "get:Get")
beego.Router("/api/"+version+"/repositories/*", &api.RepositoryAPI{}, "delete:Delete;put:Put")
beego.Router("/api/"+version+"/repositories/*/labels", &api.RepositoryLabelAPI{}, "get:GetOfRepository;post:AddToRepository")
beego.Router("/api/"+version+"/repositories/*/labels/:id([0-9]+)", &api.RepositoryLabelAPI{}, "delete:RemoveFromRepository")
beego.Router("/api/"+version+"/repositories/*/tags/:tag", &api.RepositoryAPI{}, "delete:Delete;get:GetTag")
beego.Router("/api/"+version+"/repositories/*/tags/:tag/labels", &api.RepositoryLabelAPI{}, "get:GetOfImage;post:AddToImage")
beego.Router("/api/"+version+"/repositories/*/tags/:tag/labels/:id([0-9]+)", &api.RepositoryLabelAPI{}, "delete:RemoveFromImage")
beego.Router("/api/"+version+"/repositories/*/tags", &api.RepositoryAPI{}, "get:GetTags;post:Retag")
beego.Router("/api/"+version+"/repositories/*/tags/:tag/manifest", &api.RepositoryAPI{}, "get:GetManifests")
beego.Router("/api/"+version+"/repositories/*/signatures", &api.RepositoryAPI{}, "get:GetSignatures")
beego.Router("/api/"+version+"/repositories/top", &api.RepositoryAPI{}, "get:GetTopRepos")
beego.Router("/api/"+version+"/system/gc", &api.GCAPI{}, "get:List")
beego.Router("/api/"+version+"/system/gc/:id", &api.GCAPI{}, "get:GetGC")
beego.Router("/api/"+version+"/system/gc/:id([0-9]+)/log", &api.GCAPI{}, "get:GetLog")
@ -155,11 +143,6 @@ func registerLegacyRoutes() {
beego.Router("/api/"+version+"/projects/:pid([0-9]+)/scanner", proScannerAPI, "get:GetProjectScanner;put:SetProjectScanner")
beego.Router("/api/"+version+"/projects/:pid([0-9]+)/scanner/candidates", proScannerAPI, "get:GetProScannerCandidates")
// Add routes for scan
scanAPI := &api.ScanAPI{}
beego.Router("/api/"+version+"/repositories/*/tags/:tag/scan", scanAPI, "post:Scan;get:Report")
beego.Router("/api/"+version+"/repositories/*/tags/:tag/scan/:uuid/log", scanAPI, "get:Log")
// Add routes for scan all metrics
scanAllAPI := &api.ScanAllAPI{}
beego.Router("/api/"+version+"/scans/all/metrics", scanAllAPI, "get:GetScanAllMetrics")

48
tests/robot-cases/Group0-BAT/API_DB.robot
View File

@ -16,16 +16,18 @@ ${SERVER_URL} https://${SERVER}
${SERVER_API_ENDPOINT} ${SERVER_URL}/api
&{SERVER_CONFIG} endpoint=${SERVER_API_ENDPOINT} verify_ssl=False
# TODO the cases commented by "###" can be uncommented after implementing the repository python library based on new API
*** Test Cases ***
# TODO uncomment this after re-implement the case
# Test Case - Garbage Collection
# Harbor API Test ./tests/apitests/python/test_garbage_collection.py
Test Case - Add Private Project Member and Check User Can See It
Harbor API Test ./tests/apitests/python/test_add_member_to_private_project.py
Test Case - Delete a Repository of a Certain Project Created by Normal User
Harbor API Test ./tests/apitests/python/test_del_repo.py
Test Case - Add a System Global Label to a Certain Tag
Harbor API Test ./tests/apitests/python/test_add_sys_label_to_tag.py
# Test Case - Delete a Repository of a Certain Project Created by Normal User
# Harbor API Test ./tests/apitests/python/test_del_repo.py
#Test Case - Add a System Global Label to a Certain Tag
# Harbor API Test ./tests/apitests/python/test_add_sys_label_to_tag.py
Test Case - Add Replication Rule
Harbor API Test ./tests/apitests/python/test_add_replication_rule.py
Test Case - Edit Project Creation
@ -33,10 +35,10 @@ Test Case - Edit Project Creation
# TODO uncomment this after image scan work with basic auth - #10277
#Test Case - Scan Image
# Harbor API Test ./tests/apitests/python/test_scan_image.py
Test Case - Manage Project Member
Harbor API Test ./tests/apitests/python/test_manage_project_member.py
Test Case - Project Level Policy Content Trust
Harbor API Test ./tests/apitests/python/test_project_level_policy_content_trust.py
### Test Case - Manage Project Member
### Harbor API Test ./tests/apitests/python/test_manage_project_member.py
### Test Case - Project Level Policy Content Trust
### Harbor API Test ./tests/apitests/python/test_project_level_policy_content_trust.py
# TODO uncomment this after we move the accesslog away from registry notificaiton
# TODO potentially #10602 may also fix this.
# Test Case - User View Logs
@ -47,23 +49,21 @@ Test Case - Project Level Policy Content Trust
# TODO uncomment this after bump up chart API version to v2.0
# Test Case - List Helm Charts
# Harbor API Test ./tests/apitests/python/test_list_helm_charts.py
Test Case - Assign Sys Admin
Harbor API Test ./tests/apitests/python/test_assign_sys_admin.py
Test Case - Retag Image
Harbor API Test ./tests/apitests/python/test_retag.py
Test Case - Robot Account
Harbor API Test ./tests/apitests/python/test_robot_account.py
Test Case - Sign A Image
Harbor API Test ./tests/apitests/python/test_sign_image.py
### Test Case - Assign Sys Admin
### Harbor API Test ./tests/apitests/python/test_assign_sys_admin.py
### Test Case - Robot Account
### Harbor API Test ./tests/apitests/python/test_robot_account.py
### Test Case - Sign A Image
### Harbor API Test ./tests/apitests/python/test_sign_image.py
# TODO uncomment this after making quota work with OCI registry
# Test Case - Project Quota
# Harbor API Test ./tests/apitests/python/test_project_quota.py
Test Case - System Level CVE Whitelist
Harbor API Test ./tests/apitests/python/test_sys_cve_whitelists.py
Test Case - Project Level CVE Whitelist
Harbor API Test ./tests/apitests/python/test_project_level_cve_whitelist.py
Test Case - Tag Retention
Harbor API Test ./tests/apitests/python/test_retention.py
Test Case - Health Check
Harbor API Test ./tests/apitests/python/test_health_check.py
### Test Case - System Level CVE Whitelist
### Harbor API Test ./tests/apitests/python/test_sys_cve_whitelists.py
### Test Case - Project Level CVE Whitelist
### Harbor API Test ./tests/apitests/python/test_project_level_cve_whitelist.py
### Test Case - Tag Retention
### Harbor API Test ./tests/apitests/python/test_retention.py
### Test Case - Health Check
### Harbor API Test ./tests/apitests/python/test_health_check.py

7
tests/robot-cases/Group0-BAT/API_LDAP.robot
View File

@ -9,11 +9,14 @@ Library Process
Default Tags API
*** Test Cases ***
# TODO the cases commented by "###" can be uncommented after implementing the repository python library based on new API
Test Case - LDAP Group Admin Role
Harbor API Test ./tests/apitests/python/test_ldap_admin_role.py
Test Case - LDAP Group User Group
Harbor API Test ./tests/apitests/python/test_user_group.py
Test Case - Run LDAP Group Related API Test
Harbor API Test ./tests/apitests/python/test_assign_role_to_ldap_group.py
### Test Case - Run LDAP Group Related API Test
### Harbor API Test ./tests/apitests/python/test_assign_role_to_ldap_group.py