Singularity on XStream

Singularity containers

Singularity containers can be used to package entire scientific workflows, software and libraries, and even data. Singularity is useful to users that have different environmental needs than what is provided by the service provider.

The Singularity software can import your Docker images without having Docker installed or being a superuser:

$ cd $WORK
$ singularity pull docker://godlovedc/lolcow
$ ./lolcow.img

Alternatively, you can also package and build your containers using Singularity Hub and then use URLs using the shub:// protocol:

$ SINGULARITY_CACHEDIR=$PWD singularity pull shub://vsoch/hello-world
$ ./vsoch-hello-world-master.img

Notice that we set the SINGULARITY_CACHEDIR to the present working directory to not save the image to our cache.

If not specified, SINGULARITY_CACHEDIR is set to $WORK/.singularity/.

GPU support

Singularity 2.3 adds support for Nvidia GPUs. Adding the --nv option will ensure that your container is using the same Nvidia drivers than the host.

Example with TensorFlow

This section shows an example of use with the latest TensorFlow with GPU support.

Singularity itself is available as a module on XStream:

$ module spider singularity
$ module load singularity/2.3

Please be sure to store your Singularity images in $WORK or $GROUP_WORK. $HOME is NOT suited to store container images. See Storage Resources for more information about storage spaces on XStream.

$ cd $WORK

Get the latest (stable) TensorFlow docker image with GPU support:

$ singularity pull docker://tensorflow/tensorflow:latest-gpu

Alternatively, you can get a more recent build:

$ singularity pull docker://tensorflow/tensorflow:nightly-gpu-py3

Test your TensorFlow container by launching a shell:

$ singularity shell --home $WORK:/home --nv tensorflow-latest-gpu.img

Importing the tensorflow module should not return any error:

Singularity tensorflow.img:~> python
>>> import tensorflow as tf

You’re good to go and to run the latest TensorFlow within a job on XStream.

Below is an example of SLURM batch script to execute TensorFlow within Singularity to train the CIFAR-10 model on a single GPU. The example also makes use of the local SSD ($LSTOR) available on the compute nodes to speed up processing of all the small images.

#!/bin/bash
#SBATCH --job-name=cifar10_1gpu
#SBATCH --output=slurm_cifar10_1gpu_%j.out
#SBATCH --cpus-per-task=1
#SBATCH --gres gpu:1
#SBATCH --time=1:00:00

TF_IMG=tensorflow-latest-gpu.img
CIFAR10_DIR=$WORK/tensorflow/cifar10

mkdir $LSTOR/cifar10_data
cp -v cifar-10-binary.tar.gz $LSTOR/cifar10_data/

module load singularity

srun singularity exec --home $WORK:/home --bind $LSTOR:/tmp --nv $TF_IMG \
    python $CIFAR10_DIR/cifar10_train.py --batch_size=128 \
                                         --max_steps=100000

Getting help or information

You can either get support from the Singularity team or contact Research Computing Support regarding the use of Singularity on XStream.