Using containers with Docker
Rockefeller University, Bioinformatics Resource Centre
https://rockefelleruniversity.github.io/RU_reproducibleR/
Containers
Set Up
All prerequisites, links to material and slides for this course can be found on github.
Or can be downloaded as a zip archive from here.
Course materials
Once the zip file in unarchived. All presentations as HTML slides and pages, their R code and HTML practical sheets will be available in the directories underneath.
- presentations/slides/ Presentations as an HTML slide show.
- presentations/singlepage/ Presentations as an HTML single page.
- presentations/r_code/ R code in presentations.
- exercises/ Practicals as HTML pages.
- answers/ Practicals with answers as HTML pages and R code solutions.
- data/ Data used in this presentation.
What are containers? Why should we use them?
The problem
Something works on your computer (e.g. bioinformatics analysis or software deployment), and you want to make sure that it will work on another computer.
https://jhudatascience.org/Adv_Reproducibility_in_Cancer_Informatics/launching-a-docker-image.html - CC-BY 4.0
The solution - Docker!
Docker allows for the creation of an isolated environment that can be shipped across different users, machines, or operating systems, and to virtual machines or the cloud.
https://jhudatascience.org/Adv_Reproducibility_in_Cancer_Informatics/launching-a-docker-image.html - CC-BY 4.0
Docker client and host
Once installed on your computer, Docker runs a process called the Docker daemon. A daemon is a program that runs as a background process and is not under direct control of the computer user, and the Docker daemon is the engine that manages Docker services and objects by communicating with the client.
The Docker client, typically the command line interface, communicates with the Docker daemon based on user commands.
Creating Docker images
The ‘docker image build’ command uses a Dockerfile to create an image.
A Docker image is a read-only, isolated file system that contains all software, dependencies, scripts, and metadata required to run a container.
Launching Docker containers
Once an image is built, an instance of this image can be launched as a stand-alone application, also known as a container.
Pulling Docker images
There are public repositories of Docker images (e.g. Docker Hub), and typically you start with an existing image and build on top of this.
Installing Docker
Use this link to install Docker.
- Click on the Docker desktop icon and make an account with Docker.
- Docker must be open and running to use the command line interface (CLI), which is how we will primarily use Docker.
- See here for Docker CLI commands.
Check Docker version to make sure Docker is installed and running.
Code (terminal):
Output:
Installing Docker
If previous command isn’t found check the Docker Desktop advanced settings and make sure CLI tools are available system-wide.
Running Docker containers from Docker Hub images
Pulling Docker images - Rocker
Rocker is a very useful source of images on Docker Hub for R and RStudio. We can pull these images immediately after installing Docker. Here we pull an image containing RStudio and a specific version of R.
Code (terminal):
Viewing local Docker images
After pulling, the image is now available on our system to run.
Images have names, tags, and image IDs as shown in the output. The ID is a hash of the metadata and filesystem of the Docker image.
Code (terminal):
Output:
Viewing local Docker images
After pulling, the image is now available on our system to run.
Images have names, tags, and image IDs as shown in the output. The ID is a hash of the metadata and filesystem of the Docker image.
We can also view the image in Docker desktop:
Running Docker containers
Once the image is on our system, we can launch a container with the ‘docker container run’ command.
Components of the run command: * –rm: this will automatically remove a container when you exit, otherwise can take up room on computer with old, unused containers * -p: before the colon is the port on your computer to be exposed and after the colon is the port inside the container * -e: an environmental variable is set when the container is run, and this will be the password to login * the last argument is the image name and tag (both seen with ‘docker images’)
Code (terminal):
Running Docker containers
While the container is running, we can go to ‘http://localhost:8787’ in a browser and log in with the the user ‘rstudio’ and the password from ‘docker container run’.
Listing active Docker containers
To see all containers running in the local environment, use the ‘docker container ls’ command
Code (terminal):
Output:
Stopping Docker containers
To stop the container currently running, if you are in the terminal tab where it was launched press Ctrl+C.
Or open up another tab and the ‘docker stop’ command can be used with the ID listed from ‘docker container ls’
Code (terminal):
Output:
Adding volumes to containers
The docker container has it’s own file system, and we can mount a local directory onto that file system with the ‘-v’ flag for the ‘docker container run’ command.
- Navigate to the ‘r_course’ directory within the downloaded course using the ‘cd’ command in the terminal
- Use the ‘docker container run’ command with the ‘-v’ flag
- the left side of the colon is the path on your computer to mount
- the right side is the location within the docker container file system where that data will be accessible
- ‘/home/rstudio’ is set by the container to be the working directory of Rstudio
Code (terminal):
Adding volumes to containers
The RStudio interface now shows the files in the ‘data’ directory
Adding volumes to containers
These files can be read into R, and also files can be written to the local environment
Code (R in docker image):
Adding volumes to containers
These files can be read into R, and also files can be written to the local environment
Output:
Adding volumes to containers
In addition to the files deliberately written to the local directory, the R environment files from this RStudio session are written to the working directory in the container, and therefore are copied to the local directory as hidden folders (.config and .local).
This R environment will then be loaded the next time you launch an RStudio container with this volume mounted. While this is normally okay, if desired a fresh RStudio session can be launched with the same mounted volume by removing these hidden directories.
Code (terminal):
Output:
Building custom images from a Dockerfile
Dockerfile basics and commands
The image we pull from Rocker contains base R and its associated packages. To customize the image, we will need to make a Dockerfile that adds to the Rocker image.
A Dockerfile provides the recipe to make the image. Using specialized commands, this file provides instructions to install the R packages and its dependencies.
Some examples: * FROM: sets the base image and further instructions build off of this * RUN: executes a command as if in terminal * LABEL: add metadata to the image * COPY: copies files from the the host system to the image file system * CMD: when the container is launched, this is the command that will be run
Dockerfile components
Dockerfile components
Here we start with the same RStudio base image we used previously, and then add some key R packages.
Dockerfile components
The first RUN command installs system dependencies that are common to R packages. This command looks for updates, installs, and cleans up unnecessary files.
Dockerfile components
This is a list of libraries is not comprehensive and adding more R packages could result in missing dependencies.
These can be identified in the log for the build command and manually added to the apt-get command, or alternatively, dependencies for a specific R or Python package can be found using the Posit package manager. This resource can also be used to install only those libraries that are necessary, which can help to limit the size of the image.
Dockerfile components
Then the R packages are installed using ‘install.packages’ or ‘BiocManager::install’ for Bioconductor packages.
Note: The ‘options(warn=2)’ at the beginning of the R command will stop the installation when there is a warning, making it easier to debug.
Dockerfile components
The port 8787 is exposed and the ‘init’ script that is included with the base RStudio image. These commands are specific to this image and will vary depending on what the Docker container is meant to do when launched.
Building an image with a Dockerfile
- A tag is added to distinguish this image
- The directory that contains the Dockerfile is the last argument
- If no file name is given, it will look for a file called
‘Dockerfile’
- ‘Dockerfile’ is in the data directory of course materials
Code (terminal):
cd ~/Downloads/RU_reproducibleR-master/r_course
docker image build -t rocker/rstudio:4.4.0_v2 ./dataOutput:
Building an image with a Dockerfile
Use the docker images command to see image
Code (terminal):
Output:
Running the custom container
As done previously, use the docker container run command to launch a container with our customized RStudio session
Code (terminal):
Running the custom container
As done previously, use the docker container run command to launch a container with our customized RStudio session
Output:
Install conda packages in a Docker image
Use Herper for conda packages
Use Herper for conda packages
The directory that contains the Dockerfile is the last argument
This Dockerfile is not named ‘Dockerfile’, so we specify the exact path with ‘-f’ argument
Code (terminal):
Output:
Use Herper for conda packages
Code (terminal):
Output:
Code (terminal):
Run container from Docker Desktop
Run container from Docker Desktop
We can also run containers from Docker Desktop
Run container from Docker Desktop
We can also run containers from Docker Desktop
Run Docker containers interactively in the terminal
Run R interactively in container
Oftentimes Docker is used to run a specific set of software interactively within the terminal. This allows for running custom scripts contained within the Docker container as well.
So far we have only been using the R Studio Docker image which provides it’s own user interface. Here we will use Docker to run a specific version of R and R packages in the terminal interactively. The Rocker suite has images that just contain R, without R studio.
Run R interactively in container
All we have to do is change the base Docker image that we pull, and change the command at the end of the Dockerfile to ‘R’ so it will run R once we launch a container.
Run R interactively in container
After building this image as we have done previously, it should be run with the ‘-it’ flag, which will run this container within an interactive terminal. Using these flags, when the ‘R’ command at the end of the Dockerfile is run upon launch, it will open the version of R in the image.
First we build this image from the Dockerfile like we’ve done before.
Code (terminal):
Then we run a container with the data directory mounted and the ‘-it’ flag.
Code (terminal):
cd ~/Downloads/RU_reproducibleR-master/r_course
docker container run -it -v ./data:/data rocker/r-ver:4.4.0_cust *output on next slide
Run R interactively in container
Output (terminal):
Run Python interactively in container
Python also has images available on Docker Hub and we can make a Dockerfile that has a specific version of Python and any other packages we want available in that environment.
Here we pull the python image, use pip to install a Python package (scanpy), then end with the ‘python’ command to open a Python session in the terminal.
Dockerfile:
Share images with Docker Hub
Pushing images to Docker Hub
If we then want to share our images with someone else, or simply store them elsewhere for future use, we can push to Docker Hub.
Make sure you have an account on Docker Hub.
Then create a repository with the same name as the image we want to push. Let’s share the R Studio image with samtools, so it would be ‘rstudio_4.4.0_samtools’. A repository must exist on Docker Hub before pushing to it.
Pushing images to Docker Hub
Once the repository is created on Docker Hub, we do the following steps to push the image:
- login to Docker with your credentials
- tag the image you want to push with the version
- push to Docker Hub
Code (terminal):
# log in and provide credentials used to sign into Docker Hub
# include the username you used for dockerhub and this will prompt you to enter the password
docker login -u (enter your username)
# tag the image you want to push with your Docker Hub username and a tag name after the colon
# the ID is from the 'docker images' command
docker image tag 292c85d1812f rubrc/rstudio_4.4.0_samtools:topush
# push to Docker Hub
docker image push rubrc/rstudio_4.4.0_samtools:topushPushing images to Docker Hub
If we then want to share our images with someone else, or simply store them elsewhere for future use, we can push to Docker Hub.
Make sure you have an account on Docker Hub.
Use renv and Docker together
Make a lock file
renv and Docker can be used in tandem to easily recreate and R environment.
There is a renv lock file in the ‘r_course/data/renv_docker’ folder within the course materials. This lock file generated by renv shows the versions of R and the loaded packages that were part of my project.
Dockerfile with renv
We will now use renv within a Dockerfile. R still needs to be installed to use renv, so we use Rocker again to install a specific version of R to match the renv lock file.
Dockerfile with renv
When building the image the lock file is copied to the image into a directory that is created and set as the working directory with the WORKDIR command.
Building image and running container
Build the image with the build context (last argument) set to the directory containing the Dockerfile and the lock file, then launch a container.
Using Docker on HPC
Docker is generally not allowed on the HPC due to the need to have root access. You can use Docker images by using another containerization software called Apptainer.
Apptainer was designed to play nice with Docker and the Apptainer manual goes into detail about how to use Apptainer with Docker images.
Apptainer is not installed on the head nodes, so you can run interactively if your lab has it’s own node, or you can submit jobs to the HPC scheduler.
Exercises
Exercise on Reproducibility in R can be found here
Contact
Any suggestions, comments, edits or questions (about content or the slides themselves) please reach out to our GitHub and raise an issue.