Version 1.0.0
This software was developed at the National Institute of Standards and Technology by employees of the Federal Government in the course of their official duties. Pursuant to title 17 Section 105 of the United States Code this software is not subject to copyright protection and is in the public domain. This software is an experimental system. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guarantees, expressed or implied, about its quality, reliability, or any other characteristic. We would appreciate acknowledgement if the software is used.
YOU SHOULD NOT DEPLOY THIS SYSTEM ON A PUBLIC SERVER SINCE THE SOFTWARE DOES NOT INCLUDE ANY ACCOUNT AND UPLOAD ACCESS MANAGEMENT. THE CURRENT IMAGE UPLOAD IS COMPLETELY UNRESTRICTED AND COULD BE USED TO UPLOAD MALWARE, VIRUSES OR INNAPROPRIATE CONTENT.
The Web Image Processing Pipelines system (WIPP) version 1.0.0 does not include
any web security management. WIPP 1.0.0 allows unrestricted uploading of files via the web
browser interface and the uploaded files interact with the file system as well as with
an instance of MongoDB database.
WIPP 1.0.0 is intended for deployment on private networks behind a firewall. Future releases
will include account and upload access management.
Download WIPP Zip file containing the installation script and a README file here.
To install Docker, please, follow the instructions from the
Docker web site (menu "Get Docker")
according to the operating system the host is running.
Official Docker instructions available here.
sudo apt-get updatesudo apt-get upgrade
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
- Add Docker’s official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- Set up the Docker stable repository (for architecture amd64):
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
- Install the latest version of Docker:
sudo apt-get install docker-ce
- Verify that Docker CE is properly installed (will print an informational message):
sudo docker run hello-world
After installing Docker, you will have to use sudo for all Docker commands.
To avoid that, follow the instructions from the
Docker Linux postinstall
web page:
sudo groupadd docker
- Add your user to the docker group:
sudo usermod -aG docker $USER
- Log out and log back in so that your group membership is re-evaluated,
then verify that you can run Docker commands without sudo:
docker run hello-world
In order to have the Docker service automatically started when the system boots, follow the instructions from the Docker Linux postinstall web page: - For Ubuntu 16.04 and higher, run:
sudo systemctl enable docker
The WIPP Docker deployment is using Docker Swarm to create a cluster of Docker nodes on a single or multiple hosts configuration.
- Initialize the swarm:- Configure the swarm network overlay:IP_MGR=129.1.2.3(replace 129.1.2.3 by the IP address of the host)docker swarm init --advertise-addr ${IP_MGR} --listen-addr ${IP_MGR}
docker network create --driver overlay wippnet
The WIPP Docker containers are using Docker volumes to store data on the host (see Docker volumes web page for more information about Docker volumes).
- Create a Docker volume for the WIPP database:
docker volume create --name wippdbvolume
- Create a Docker volume for the WIPP data storage:
docker volume create --name wippdatavolume
A deployment script setup.sh is provided in the WIPP Zip file.
chmod a+x setup.sh
- Run the script to deploy WIPP on the Docker Swarm:
./setup.sh wippnet wippdbvolume wippdatavolume 4G
Replace "4G" by the maximum amount of RAM you want to allow for the image processing algorithms.
This amount should be scaled according to the available RAM on the host and the size of the data you want to process,
knowing that algorithms may require up to 10 times the size of a single image in RAM size, i.e., a 1GB image may need
up to 10GB of RAM. Format: size[g|G|m|M|k|K]
docker service ls
The output should be similar to the following:
ID NAME MODE REPLICAS IMAGE 1x4y8ee96kn2 master replicated 0/1 dscnaf/htcondor-debian:release-0.2.0 3wx2zpmbj6om wipp replicated 0/1 wipp/wipp:latest iyyrjoln0t0t mongodb replicated 0/1 mongo:3.4 n3bgifaet7vf exec replicated 0/1 wipp/wipp_executor:latest
REPLICAS 0/1 mean services are not yet started (either loading or failing to start).
For detailed information about a service:
docker service ps $NAME
For example, to check the state of the mongodb (database) service: docker service ps mongodb
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
02zpjbxq42a7 mongodb.1 mongo:latest vm-itl-ssd-063 Running Running 44 seconds ago
docker service ls ID NAME MODE REPLICAS IMAGE 1x4y8ee96kn2 master replicated 1/1 dscnaf/htcondor-debian:release-0.2.0 3wx2zpmbj6om wipp replicated 1/1 wipp/wipp:latest iyyrjoln0t0t mongodb replicated 1/1 mongo:3.4 n3bgifaet7vf exec replicated 1/1 wipp/wipp_executor:latest
WIPP is running through the ports 18080 (web interface) and 15005 (Pegasus dashboard). In order to access the WIPP system, these ports have to be open for communication.
- On Ubuntu 16.04 and up, one option is to use the uncomplicated firewall (ufw):- Check that the ports are open:sudo ufw allow 18080sudo ufw allow 15005
sudo ufw status verbose
Once deployed, the WIPP web interface will be accessible from http://host-ip:18080
and the Pegasus dashboard (for troubleshooting any failing jobs) will be accessible from
https://host-ip:15005 (replace host-ip by the IP address of the host.)
The Pegasus dashboard is accessible via HTTPS, but does not ship with a SSL certificate. Some web browsers will complain about the lack of valid certificate and ask the user to add a security exception, or confirm "Proceed to the website (unsafe)" before being able to access the dashboard. The dashboard is set up to be accessible with the credentials "wipp/zaq123".