Version 2.3.0
This software was developed at the National Institute of Standards and Technology by employees of the Federal Government in the course of their official duties. Pursuant to title 17 Section 105 of the United States Code this software is not subject to copyright protection and is in the public domain. This software is an experimental system. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guarantees, expressed or implied, about its quality, reliability, or any other characteristic. We would appreciate acknowledgement if the software is used.
YOU SHOULD NOT DEPLOY THIS SYSTEM ON A PUBLIC SERVER SINCE THE SOFTWARE DOES NOT INCLUDE ANY ACCOUNT AND UPLOAD ACCESS MANAGEMENT. THE CURRENT IMAGE UPLOAD IS COMPLETELY UNRESTRICTED AND COULD BE USED TO UPLOAD MALWARE, VIRUSES OR INNAPROPRIATE CONTENT.
The Web Image Processing Pipelines system (WIPP) version 2.3.0 does not include
any web security management. WIPP 2.3.0 allows unrestricted uploading of files via the web
browser interface and the uploaded files interact with the file system as well as with
an instance of MongoDB database.
WIPP 2.3.0 is intended for deployment on private networks behind a firewall. Future releases
will include account and upload access management.
We provide a CPU-only version of WIPP, as well as a CPU+GPU-support version, so the deployment steps will depend on the version you choose.
WIPP with GPU support is recommended if you are planning on doing training of the Convolutional Neural Network model we provide for Deep Learning-based segmentation.
Deploying WIPP on a single host consists of five (CPU-only) to six (GPU support) steps:Download WIPP Zip file containing the installation script and a README file here.
To install Docker, please, follow the instructions from the
Docker web site (menu "Get Docker")
according to the operating system the host is running.
Official Docker instructions available here.
sudo apt-get updatesudo apt-get upgrade
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
- Add Docker’s official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- Set up the Docker stable repository (for architecture amd64):
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
- Install the latest version of Docker:
sudo apt-get install docker-ce
- Verify that Docker CE is properly installed (will print an informational message):
sudo docker run hello-world
After installing Docker, you will have to use sudo for all Docker commands.
To avoid that, follow the instructions from the
Docker Linux postinstall
web page:
sudo groupadd docker
- Add your user to the docker group:
sudo usermod -aG docker $USER
- Log out and log back in so that your group membership is re-evaluated,
then verify that you can run Docker commands without sudo:
docker run hello-world
In order to have the Docker service automatically started when the system boots, follow the instructions from the Docker Linux postinstall web page: - For Ubuntu 16.04 and higher, run:
sudo systemctl enable docker
Docker Compose is a useful tool for running multi-container applications such as WIPP. For convenience, we use Docker Compose to deploy WIPP in single-host deployments.
Follow the instructions from the Docker Compose installation page to install it (choosing the appropriate Operating System) and verify that the installation is working properly:
docker-compose --version
WIPP is available with GPU support for running CNN training jobs using TensorFlow. We only support Nvidia Cuda 8, due to the version of TensorFlow installed in WIPP.
Install Nvidia Cuda 8 following the instructions available on the Nvidia website.
Install the Nvidia Docker runtime following the instructions available on the Nvidia Docker Github repository.
In order to run the Nvidia Docker runtime automatically when running Docker containers, create (or edit if existing) a configuration file for the Docker daemon at /etc/docker/daemon.json (see the Docker daemon documentation for more information).
Add the following configuration to the /etc/docker/daemon.json file:
{
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
},
"default-runtime": "nvidia"
}
Relad the Docker daemon and restart Docker to apply the changes:
sudo systemctl daemon-reload
sudo systemctl restart docker
Make sure you can run nvidia-smi with the latest official CUDA image
docker run --rm nvidia/cuda nvidia-smi
The WIPP Docker containers are using Docker volumes to store data on the host (see Docker volumes web page for more information about Docker volumes).
- Create a Docker volume for the WIPP database:
docker volume create --name wippdbvolume
- Create a Docker volume for the WIPP data storage:
docker volume create --name wippdatavolume
- Create a Docker volume for the Pegasus database:
docker volume create --name pegasusdbvolume
- Create a Docker volume for the Pegasus data storage:
docker volume create --name pegasusdatavolume
We provide compose files for both the CPU-only and CPU+GPU WIPP versions.
- To install WIPP 2.3.0 with CPU-only support (regular version):
docker-compose -f wipp-cpu.yml up -d
- To install WIPP 2.3.0 with CPU and GPU support (version recommended for CNN training capabilities):
docker-compose -f wipp-gpu.yml up -d
- (Optional) Install WIPP with custom RAM allocation
export WIPP_MAX_RAM=4G
Replace "4G" by the maximum amount of RAM you want to allow for the image processing algorithms.
This amount should be scaled according to the available RAM on the host and the size of the data you want to process,
knowing that algorithms may require up to 10 times the size of a single image in RAM size, i.e., a 1GB image may need
up to 10GB of RAM. Format: size[g|G|m|M|k|K]This script will pull the WIPP Docker images from the WIPP Docker Hub repository, as well as the public MongoDB (database) image, and deploy them as containers.
After running the script, the containers will start their initialization. To get the status of the WIPP containers, run:
docker ps
The output should be similar to the following:
ID NAME MODE REPLICAS IMAGE CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES aee9f5faf03f wipp/wipp_executor:2.3.0 "/sbin/tini -- /usr/…" 3 days ago Up 3 days 22/tcp, 5000/tcp wipp_exec 1342a6bdeb01 wipp/wipp:2.3.0 "/sbin/tini -- /usr/…" 3 days ago Up 3 days (healthy) 22/tcp, 5000/tcp, 0.0.0.0:5005->5005/tcp, 0.0.0.0:80->8080/tcp wipp 10c3d2c399e9 wipp/wipp_master:2.3.0 "/sbin/tini -- /usr/…" 3 days ago Up 3 days 22/tcp, 5000/tcp wipp_master 0c2fa94472f3 mongo:3.4 "docker-entrypoint.s…" 3 days ago Up 3 days 27017/tcp wipp_mongoWIPP system containers will start in this order: mongodb, master, wipp, exec, and the deployment will take a few minutes to download the Docker images and start the system.
WIPP is running through the ports 80 (web interface) and 5005 (Pegasus dashboard). In order to access the WIPP system, these ports have to be open for communication.
- On Ubuntu 16.04 and up, one option is to use the uncomplicated firewall (ufw):- Check that the ports are open:sudo ufw allow 80sudo ufw allow 5005
sudo ufw status verbose
Once deployed, the WIPP web interface will be accessible from http://host-ip
and the Pegasus dashboard (for troubleshooting any failing jobs) will be accessible from
https://host-ip:5005 (replace host-ip by the IP address or server name of the host.)
The Pegasus dashboard is accessible via HTTPS, but does not ship with a SSL certificate. Some web browsers will complain about the lack of valid certificate and ask the user to add a security exception, or confirm "Proceed to the website (unsafe)" before being able to access the dashboard. The dashboard is set up to be accessible with the credentials "wipp/zaq123"".
To stop WIPP (stopping the containers, but keeping the data):
docker-compose -f wipp-cpu.yml down # CPU versiondocker-compose -f wipp-gpu.yml down # CPU + GPU version
This set of instructions is intended for advanced users and system administrators. We only support Unix operating systems for the multi-hosts deployment, and have been testing on three Ubuntu 16.04 servers.
Deploying WIPP on a multiple hosts consists of six steps:Download the WIPP Zip file containing the installation script and a README file here.
To install Docker, please, follow the instructions from the
Docker web site (menu "Get Docker")
according to the operating system the host is running.
The WIPP Docker deployment is using Docker Swarm to create a cluster of Docker nodes on a single or multiple hosts configuration.
- Choose one of the hosts to be the Swarm manager and initialize the swarm on this manager host:The output should be similar to:IP_MGR=129.1.2.3(replace 129.1.2.3 by the IP address of the host)docker swarm init --advertise-addr ${IP_MGR} --listen-addr ${IP_MGR}
Swarm initialized: current node (xxxxxxxxx) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token xxx-yyyy-zzzz \
129.1.2.3:2377
Save this docker swarm command to use it on the worker hosts later to make them join the swarm.
- Open port 2377 for Docker Swarm communications (using ufw for example):
sudo ufw allow 2377
- Open ports 4789 and 7946 for Swarm network overlay communications (using ufw for example):
- Join the Swarm from the other hosts using the saved docker swarm command similar to the following:sudo ufw allow 4789sudo ufw allow 7946
docker swarm join \
--token xxx-yyyy-zzzz \
129.1.2.3:2377
On a multi-hosts deployment, the WIPP Docker containers are binding folders that are shared across all the hosts (see Docker bind mounts web page for more information about the usage of bind mounts).
Four shared folders need to be available from all hosts:
- Shared folder for the WIPP database (wippdbvolume)
- Shared folder for the WIPP data storage (wippdatavolume)
- Shared folder for the Pegasus database (pegasusdbvolume)
- Shared folder for the Pegasus data storage (pegasusdatavolume)
Notes: One solution for sharing folders across multiple hosts is to use
GlusterFS to mount shared volumes on all hosts.
Please refer to the GlusterFS documentation for configuring shared gluster volumes.
The wipp user inside of the Docker containers has the UID 1000, so may need to change the ownership
of the shared folders to the user 1000, and set read and execution permissions for group and others as well.
Please note that the wippdbvolume, pegasusdbvolume and pegasusdatavolume will contain database
files, which may cause issues when using a NAS (Network Attached Storage) for hosting the shared folders,
for example using a CIFS mount.
A deployment script setup.sh is provided in the WIPP Zip file.
This script must be executed from the Swarm manager host.
chmod a+x setup.sh
- Modify the deployment configuration (first lines of the script) to set the paths to the shared folders and maximum RAM allocated to WIPP
computations
wippdbvolume=/path/to/wippdbvolumewippdatavolume=/path/to/wippdatavolumepegasusdbvolume=/path/to/pegasusdbvolumepegasusdatavolume=/path/to/pegasusdatavolumewippmaxram=4G
Replace "4G" by the maximum amount of RAM you want to allow for the image processing algorithms. This amount should be scaled according to the available RAM on the host and the size of the data you want to process, knowing that algorithms may require up to 10 times the size of a single image in RAM size, i.e., a 1GB image may need up to 10GB of RAM. Format: size[g|G|m|M|k|K]
- Run the script from the Swarm manager host to deploy WIPP on the Docker Swarm:
./setup.sh
This script will pull the WIPP Docker images from the
WIPP Docker Hub repository,
as well as the public MongoDB (database) image,
and deploy them as services in the Docker Swarm.
docker service ls
The output should be similar to the following:
ID NAME MODE REPLICAS IMAGE 1x4y8ee96kn2 master replicated 0/1 wipp/wipp_master:2.3.0 3wx2zpmbj6om wipp replicated 0/1 wipp/wipp:2.3.0 iyyrjoln0t0t mongodb replicated 0/1 mongo:3.4 n3bgifaet7vf exec replicated 0/1 wipp/wipp_executor:2.3.0
REPLICAS 0/1 means services are not yet started (either loading or failing to start).
For detailed information about a service:
docker service ps $NAME
For example, to check the state of the mongodb (database) service: docker service ps mongodb
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
02zpjbxq42a7 mongodb.1 mongo:latest vm-itl-ssd-063 Running Running 44 seconds ago
docker service ls ID NAME MODE REPLICAS IMAGE 1x4y8ee96kn2 master replicated 1/1 wipp/wipp_master:2.3.0 3wx2zpmbj6om wipp replicated 1/1 wipp/wipp:2.3.0 iyyrjoln0t0t mongodb replicated 1/1 mongo:3.4 n3bgifaet7vf exec replicated 1/1 wipp/wipp_executor:2.3.0
WIPP is running through the ports 80 (web interface) and 5005 (Pegasus dashboard). In order to access the WIPP system, these ports have to be open for communication.
- For example, using ufw:sudo ufw allow 80sudo ufw allow 5005
Once deployed, the WIPP web interface will be accessible from http://host-ip
and the Pegasus dashboard (for troubleshooting any failing jobs) will be accessible from
https://host-ip:5005 (replace host-ip by the IP address or server name of the host.)
The Pegasus dashboard is accessible via HTTPS, but does not ship with a SSL certificate. Some web browsers will complain about the lack of valid certificate and ask the user to add a security exception, or confirm "Proceed to the website (unsafe)" before being able to access the dashboard. The dashboard is set up to be accessible with the credentials "wipp/zaq123"".
To stop and remove the WIPP Docker services that are currently running on the machine, use the following command:
docker service rm exec wipp master mongodb
The WIPP system will be deleted, but the WIPP database and file system will remain intact in the Docker volumes.
docker exec -u root wipp /bin/sh -c "/opt/WIPP/scripts/migration.sh 1000 1000 5100 5100"
(In case of a multi-host deployment, you will need to run this command from the host of the wipp container, and
replace "wipp" by the actual name of the container (should be similar to wipp_wipp.1.c6m7u8ttixr2l5frds2prt7hk, you can
use tab on Linux systems for auto-complexion of the name).
Notes: Docker, docker-compose and VirtualBox will be automatically installed
after installing Docker Toolbox.
A step-by-step guide on how to install Docker Toolbox on Windows 7 is provided in the documentation
PDF "WIPP-docker-machine-deployment.pdf" under the "doc" folder of the downloaded WIPP Zip file.
Docker for Windows (for Windows 10) is not supported for this WIPP installation, due to the use of
VirtualBox, Docker Toolbox must be used instead.
#!/bin/bash
export cpus=2 # 2 CPUs - number of CPUs allocated to VM
export ram=8192 # 8 GB - amount of RAM allocated to VM in MB
export disk=80000 # ~80GB - disk space allocated to VM (hosting the WIPP system + data)
export backend_memory="4G" # 4 GB - maximum amount of RAM in GB allocated to computations, should be lower than amount of RAM allocated to the VM
sh wipp.sh setup
docker-machine ip wipp