Singularity

IST Cluster uses Singularity containers to control environment. Singularity containers can be used to package entire scientific workflows, softwares, libraries, and even data. Containers can be pull from Docker.

Singularity Usage Workflow

You have to build a container or environment from your workstation.

1. Develop and test containers using --sandbox(build into a writable directory)

2. Build your production containers with a squashfs filesystem.

Once you have the container with the necessary libraries and data inside, it can be executed by HPC cluster without requireing root access.

####A production container should be an immutable object, so if you need to make changes you should go back to your build system with root privilleges, rebuild container with the changes, and re-upload the container to the HPC cluster.#####

Singularity Commands

build: Build a container or environment
exec: Execute a command to container
inspect: See labels, run and test scripts, and environment variables
pull: Pull an image from Docker or Singularity Hub
run: Run image as an executable
shell: Shell into image

Example: Build a container from a Singularity definition file

Create a file name my-def-file as a definition file.

touch my-def-file

Edit my-def-file

Bootstrap: docker
From: ubuntu:bionic

%post
    apt-get -y update
    apt-get -y install cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:${PATH}

%runscript
    cowsay "Hello World" | lolcat

Build a container from my-def-file (root privillege)

singularity build --sandbox --fakeroot hello-world my-def-file

singularity build needs root privillege, since you are user of IST cluster and not granted sudo access. --fakeroot helps you to build singularity containers and images without real root privillege.
Build a sif(Singularity Image File) from the hello-world container.

singularity build --fakeroot hello-world.sif hello-world
You can upload hello-world.sif into IST cluster to execute it.

Since this is tutorial and hello-world.sif is small enough to execute from local computer.

./hello-world.sif

Example: Copying data from host into the container

If you have necessary data in your local computer and you want to execute them with your scientific jobs. Assume you have a file hello.txt in your local computer (If not, echo "Hello World" > hello.txt)

Create a file name copy-into-container as a definition file.

touch copy-into-container

Edit copy-into-container

Bootstrap: docker
From: ubuntu:bionic

%post
    apt-get -y update
    apt-get -y install

%environment
    export LC_ALL=C
    export PATH=/usr/games:${PATH}

%runscript
    cat /data/hello.txt | cowsay | lolcat

Build a container from copy-into-container singularity build --sandbox --fakeroot example-container copy-into-container
Since --sandbox mode, we can read and write data into the container (With root privillege). Try to make a directory and copy a file into the container with these commands.
```
   sudo mkdir example-container/data
   sudo cp hello.txt example-container/data
```
Build a sif from copy-into-container container.

singularity build --fakeroot copy-into-container.sif copy-into-container

Then run the image.

./copy-into-container.sif

Example: Execute the image and get the result

If you want output from your work. Singularity can execute works and simply save output file in your workstation (Outside container)

Write some python script that read and write a file.
```
from sys import argv
from random import randint

inp = []
with open(argv[1], 'r') as reader:
    for line in reader:
    inp.append(str(line)*randint(1, 6))

with open(argv[2], 'w') as writer:
    for line in inp:
        writer.write(line)
```
saved as py-script.py.

In this python script, it takes 2 arguments. 1st argument is input.txt and 2nd argument is output.txt which this script will generate when execution end.

Create input.txt with this following command.

printf "some string\nfish\nthis is python\nHello world\n" > input.txt

The runscript should be python py-script.py input.txt output.txt

Create a definition file py-container-def

Bootstrap: docker
From: ubuntu:bionic

%post
    apt-get -y update
    apt-get -y install python3
    alias python=python3

%runscript
    python py-script.py input.txt output.txt

Build a container from py-container-def

singularity build --sandbox --fakeroot my-python-container py-container-def
Build a sif from my-python-container

singularity build --fakeroot my-python.sif my-python-container
Singularity image can access to the host filesystem automatically. Make sure you have all necessary files(Or folders) in your workstation.

Following this section of example, you should have
- py-script.py (necessary)
- input.txt (necessary)
- py-container-def (unnecessary)
- my-python-container (folder, unnecessary)
- my-python.sif (necessary)
Run the command ./my-python.sif

You should have the output.txt in your current directory.