Upstream/harbor
1
0
Fork 0
mirror of https://github.com/goharbor/harbor.git synced 2024-12-22 16:48:30 +01:00
Code Issues Projects Releases Wiki Activity

Vulnerability data import (#2981)

Browse Source 
* update offline data import for vul scan

* update vul scan data import guide

* update vul data guide
This commit is contained in:
Haining Henry Zhang 2017-08-07 18:34:13 +08:00 committed by Wenkai Yin
parent cf7326cf18
commit c53c5f65a9
2 changed files with 58 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
docs/clair_offlinedata_guide.md
View File

@ -1,55 +0,0 @@
## Guild for update Clair offline data
### Summary
In some case when user install harbor in an environment without internet access. Then Clair will not be able to fetch the latest vulnerability database. In this circumstance user need manfully update the Clair database.
This document is a step by step instruction on update Clair vulnerability database in Harbor v1.2.
### Preparation
A. User need to install Clair 2.0.1 ( if you have a harbor1.2 instance with internet access will also works.)
B. Check the Clair already update the vulnerability to the latest.
a. 'docker ps' to list the Clair container. Get the Clair container id .
b. Check the log of the Clair container.
c. If you are using harbor you can find the latest Clair log under /var/log/harbor/2017--xx-xx/clair.log
d. You will find some logs like follow:
```
Jul 3 20:40:45 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:45.890364","updater name":"rhel"}
Jul 3 20:40:46 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:46.768924","updater name":"alpine"}
Jul 3 20:40:47 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:47.190982","updater name":"oracle"}
Jul 3 20:41:07 172.18.0.1 clair[3516]: {"Event":"Debian buster is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:128","Time":"2017-07-04 03:41:07.833720"}
Jul 3 20:41:07 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:41:07.833975","updater name":"debian"}
Jul 4 00:26:17 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 07:26:17.596986","updater name":"ubuntu"}
Jul 4 00:26:18 172.18.0.1 clair[3516]: {"Event":"adding metadata to vulnerabilities","Level":"info","Location":"updater.go:253","Time":"2017-07-04 07:26:18.060810"}
Jul 4 00:38:05 172.18.0.1 clair[3516]: {"Event":"update finished","Level":"info","Location":"updater.go:198","Time":"2017-07-04 07:38:05.251580"}
```
e. The update finished indicate that Clair has finished an vulnerability update round. You need to check that logs above it to make sure all the endpoints are update correctly.
### Data dump
A. Login into the host where Clair db is running.
B. Dump the Clair vulnerability database by the follow command
* $>docker exec clair-db /bin/bash -c "pg_dump -U postgres -a -t feature -t keyvalue -t namespace -t schema_migrations -t vulnerability -t vulnerability_fixedin_feature" > vulnerability.sql
* $>docker exec clair-db /bin/bash -c "pg_dump -U postgres -c -s" > clear.sql
### Back Up Clair DB
A. Before update the offline data, user are strongly suggested to backup their Clair db.
* docker exec clair-db /bin/bash -c "pg_dump -U postgres -c" > all.sql
### Update Clair DB
A. Copy the vulnerability.sql and clear.sql to the host where clair-db container which you want to update is running on.
C. $>docker exec -i clair-db psql -U postgres < clear.sql
D. $>docker exec -i clair-db psql –U postgres < vulnerability.sql
### Rescan
After update the offline data, user need to trigger the "rescan all" functionality to scan all the images and Harbor reflect the new changes automatically after the scan finished.(Otherwise the vulnerability detail will not show up)

58
docs/import_vulnerability_data.md Normal file
View File

@ -0,0 +1,58 @@
## Update an offline Harbor instance with new vulnerability data
Harbor has integrated with Clair to scan vulnerabilities in images. When Harbor is installed in an environment without internet connection, Clair cannot fetch data from the public vulnerability database. Under this circumstance, Harbor administrator needs to manually update the Clair database.
This document provides step-by-step instructions on updating Clair vulnerability database in Harbor v1.2.
**NOTE:** Harbor does not ship with any vulnerability data. For this reason, if Harbor cannot connect to Internet, the administrator must manually import vulnerability data to Harbor by using instructions given in this document.
### Preparation
A. You need to install an instance of Clair 2.0.1 with internet connection. If you have another instance of Harbor v1.2 with internet access, it also works.
B. Check whether your Clair instance has already updated the vulnerability database to the latest version. If it has not, wait for Clair to get the data from public endpoints.
- Use command `docker ps` to find out the container id of Clair.
- Run command `docker logs container_id` to check the log of the Clair container. If you are using Harbor you can find the latest Clair log under /var/log/harbor/2017-xx-xx/clair.log
- Look for logs that look like the below:
```
Jul 3 20:40:45 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:45.890364","updater name":"rhel"}
Jul 3 20:40:46 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:46.768924","updater name":"alpine"}
Jul 3 20:40:47 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:47.190982","updater name":"oracle"}
Jul 3 20:41:07 172.18.0.1 clair[3516]: {"Event":"Debian buster is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:128","Time":"2017-07-04 03:41:07.833720"}
Jul 3 20:41:07 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:41:07.833975","updater name":"debian"}
Jul 4 00:26:17 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 07:26:17.596986","updater name":"ubuntu"}
Jul 4 00:26:18 172.18.0.1 clair[3516]: {"Event":"adding metadata to vulnerabilities","Level":"info","Location":"updater.go:253","Time":"2017-07-04 07:26:18.060810"}
Jul 4 00:38:05 172.18.0.1 clair[3516]: {"Event":"update finished","Level":"info","Location":"updater.go:198","Time":"2017-07-04 07:38:05.251580"}
```
- The phrase "finished fetching" indicates that Clair has finished a round of vulnerability update from an endpoint. Please make sure all five endpoints (rhel, alpine, oracle, debian, ubuntu) are updated correctly.
### Dumping vulnerability data
- Log in to the host (that is connected to Internet) where Clair database (Postgres) is running.
- Dump Clair's vulnerability database by the following commands, two files (`vulnerability.sql` and `clear.sql`) are generated:
```
$ docker exec clair-db /bin/bash -c "pg_dump -U postgres -a -t feature -t keyvalue -t namespace -t schema_migrations -t vulnerability -t vulnerability_fixedin_feature" > vulnerability.sql
$ docker exec clair-db /bin/bash -c "pg_dump -U postgres -c -s" > clear.sql
```
### Back up Harbor's Clair database
Before importing the data, it is strongly recommended to back up the Clair database in Harbor.
```
$ docker exec clair-db /bin/bash -c "pg_dump -U postgres -c" > all.sql
```
### Update Harbor's Clair database
Copy the `vulnerability.sql` and `clear.sql` to the host where Harbor is running on. Run the below commands to import the data to Harbor's Clair database:
```
$ docker exec -i clair-db psql -U postgres < clear.sql
$ docker exec -i clair-db psql –U postgres < vulnerability.sql
```
### Rescanning images
After importing the data, trigger the scanning process in the administrator's web UI: **Administration**->**Configuration**->**Vulnerability**->**SCAN NOW**. Harbor reflects the new changes after the scanning is completed. (Otherwise the summary of the image vulnerabilities will not be displayed correctly.)