harbor/docs/import_vulnerability_data.md

Update an offline Harbor instance with new vulnerability data

Harbor has integrated with Clair to scan vulnerabilities in images. When Harbor is installed in an environment without internet connection, Clair cannot fetch data from the public vulnerability database. Under this circumstance, Harbor administrator needs to manually update the Clair database.

This document provides step-by-step instructions on updating Clair vulnerability database in Harbor v1.2.

NOTE: Harbor does not ship with any vulnerability data. For this reason, if Harbor cannot connect to Internet, the administrator must manually import vulnerability data to Harbor by using instructions given in this document.

Preparation

A. You need to install an instance of Clair 2.0.1 with internet connection. If you have another instance of Harbor v1.2 with internet access, it also works.

B. Check whether your Clair instance has already updated the vulnerability database to the latest version. If it has not, wait for Clair to get the data from public endpoints.

• Use command docker ps to find out the container id of Clair.

• Run command docker logs container_id to check the log of the Clair container. If you are using Harbor you can find the latest Clair log under /var/log/harbor/2017-xx-xx/clair.log

• Look for logs that look like the below:

  Jul 3 20:40:45 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:45.890364","updater name":"rhel"}
  Jul 3 20:40:46 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:46.768924","updater name":"alpine"}
  Jul 3 20:40:47 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:40:47.190982","updater name":"oracle"}
  Jul 3 20:41:07 172.18.0.1 clair[3516]: {"Event":"Debian buster is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:128","Time":"2017-07-04 03:41:07.833720"}
  Jul 3 20:41:07 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 03:41:07.833975","updater name":"debian"}
  Jul 4 00:26:17 172.18.0.1 clair[3516]: {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-04 07:26:17.596986","updater name":"ubuntu"}
  Jul 4 00:26:18 172.18.0.1 clair[3516]: {"Event":"adding metadata to vulnerabilities","Level":"info","Location":"updater.go:253","Time":"2017-07-04 07:26:18.060810"}
  Jul 4 00:38:05 172.18.0.1 clair[3516]: {"Event":"update finished","Level":"info","Location":"updater.go:198","Time":"2017-07-04 07:38:05.251580"}
  

• The phrase "finished fetching" indicates that Clair has finished a round of vulnerability update from an endpoint. Please make sure all five endpoints (rhel, alpine, oracle, debian, ubuntu) are updated correctly.

Dumping vulnerability data

• Log in to the host (that is connected to Internet) where Clair database (Postgres) is running.
• Dump Clair's vulnerability database by the following commands, two files (vulnerability.sql and clear.sql) are generated:

  $ docker exec clair-db /bin/bash -c  "pg_dump -U postgres -a -t feature -t keyvalue -t namespace -t schema_migrations -t vulnerability -t vulnerability_fixedin_feature" > vulnerability.sql
  $ docker exec clair-db /bin/bash -c "pg_dump -U postgres -c -s" > clear.sql

Back up Harbor's Clair database

Before importing the data, it is strongly recommended to back up the Clair database in Harbor.

 $ docker exec clair-db /bin/bash -c  "pg_dump -U postgres -c" > all.sql

Update Harbor's Clair database

Copy the vulnerability.sql and clear.sql to the host where Harbor is running on. Run the below commands to import the data to Harbor's Clair database:

 $ docker exec -i clair-db psql -U postgres < clear.sql
 $ docker exec -i clair-db psql -U postgres < vulnerability.sql

Rescanning images

After importing the data, trigger the scanning process in the administrator's web UI: Administration->Configuration->Vulnerability->SCAN NOW. Harbor reflects the new changes after the scanning is completed. (Otherwise the summary of the image vulnerabilities will not be displayed correctly.)