1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler
Signed-off-by: chlins <chenyuzh@vmware.com>
It takes about 1 hour to perform data for 40000 repositories per performance testing.
Roll back the runner count to 100 could speed the data preparation time.
It's safe since it only takes 100 DB connection counts at most per execution per core.
Signed-off-by: Wang Yan <wangyan@vmware.com>
fixes#15736
For the current imple, the GetWorker() may hang when there is no worker available, and will not release the DB connection.
In this case, the DB connection could reach the up limit that leads to harbor core for service unavailable.
1, move GetWorker() in the goroutine, release the DB connection for API.
2, reduce the worker count per harbor-core from 1024 to 10.
3, reduce the runner count per worker to 30.
After above, the max connection per harbor-core should be 300.
Worker: To control how many replicaiton exectuions can have at most at the same time.
Runner: To control the speed to generate an jobservice replicaiton job.
Signed-off-by: Wang Yan <wangyan@vmware.com>
Improve the performance of replication by introducing a new API to check whether the blob can be mounted directly
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* Refactor registry API
Refactor registry API
Signed-off-by: Wenkai Yin <yinw@vmware.com>
* Fix bugs of replications
1. Fix the scheduled replication doesn't work issue
2. Fix the destination name lost issue when updating replication policy
Signed-off-by: Wenkai Yin <yinw@vmware.com>
