Conda

This tutorial was adapted from Introduction to Conda for Data Scientists

Getting Started with Conda

Packages and Environment

When working with a programming language, such as Python, that can do almost anything, one has to wonder how this is possible. You download Python, it has about 25 MB, how can everything be included in this small data package. The answer is - it is not. Python, as well as many other programming languages use external libraries or packages for being able to doing almost anything. You can see this already when you start programming. After learning some very basics, you often learn how to import something into your script or session.

Modules, packages, libraries

  • Module: a collection of functions and variables, as in a script
  • Package: a collection of modules with an __init__.py file
  • Library: a collection of packages with related functionality

Library/Package are often used interchangeably.

Dependencies

A dependency is a package or piece of software that another package requires in order to work correctly. For instance, when you install the package pandas, it needs other dependencies to work properly. Conda’s job is not only to install the package you requested but to determine and install other required packages.

###Channels A Conda channel A location or repository where Conda searches for and downloads packages. Different channels can provide different packages, package versions, and builds. The :: notation lets us know which channel a package comes from. For example:

conda-forge::numpy
  • conda-forge: the channel
  • ::: separates the channel from the package
  • numpy: the package

A common channel for working with Bioinformatics is the bioconda channel, which lets us install thousands of software packages related to biomedical research using the conda package manager.

Environments

A separate workspace where you can install the packages and dependencies required for a particular project without affecting other projects or your main system. An environment management system allows to run isolated versions of the same packages on the same physical machine. Some of the common problems environment management systems can solve for data scientists:

  • An application you need for a research project requires different versions of your base programming language or different versions of various third-party packages from the versions that you are currently using.
  • An application you developed as part of a previous research project that worked fine on your system six months ago now no longer works.
  • Code that was written for a joint research project works on your machine but not on your collaborators’ machines.
  • An application that you are developing on your local machine doesn’t provide the same results when run on your remote cluster.

Environment management systems for Python

Conda is not the only way; Python for example has many more ways of working with environments:

  • virtualenv
  • pipenv
  • venv
  • pyenv

Package Management

A system for installing, updating, configuring, and removing software on a computer. Package managers also help manage the dependencies required by that software. Some examples are:

  • APT (apt): commonly used on Debian and Ubuntu Linux.
  • YUM (yum): traditionally used on Red Hat–based Linux distributions such as CentOS and RHEL.
  • Homebrew (brew): Commonly used on macOS, and also available for Linux

While package managers such as apt, yum, and brew manage software available to the entire operating system, Conda manages packages inside isolated environments.

Conda

Conda is an open source package and environment management system that runs on Windows, Mac OS and Linux.

  • Conda can quickly install, run, and update packages and their dependencies.
  • Conda can create, save, load, and switch between project specific software environments on your local computer.
  • Although Conda was created for Python programs, Conda can package and distribute software for any language such as R, Ruby, Lua, Scala, Java, JavaScript, C, C++, FORTRAN.

Conda as a package manager helps you find and install packages. If you need a package that requires a different version of Python, you do not need to switch to a different environment manager, because Conda is also an environment manager. With just a few commands, you can set up a totally separate environment to run that different version of Python, while continuing to run your usual version of Python in your normal environment.

Why use Conda?

Whilst there are many different package and environment management systems that solve either the package management problem or the environment management problem, Conda solves both of these problems and explicitly targeted at (data) science use cases.

  • Conda provides prebuilt packages, avoiding the need to deal with compilers, or trying to work out how exactly to set up a specific tool. Fields such as Astronomy use conda to distribute some of their most difficult-to-install tools such as IRAF. TensorFlow is another tool where to install it from source is near impossible, but Conda makes this a single step.

  • Conda is cross platform, with support for Windows, MacOS, GNU/Linux, and support for multiple hardware platforms, such as x86 and Power 8 and 9. In future lessons we will show how to make your environment reproducible (reproducibility being one of the major issues facing science), and Conda allows you to provide your environment to other people across these different platforms.

  • Conda allows for using other package management tools (such as pip) inside Conda environments, where a library or tools is not already packaged for Conda (we’ll show later how to get access to more conda packages via channels).

Key Points

  • Conda is a platform agnostic, open source package and environment management system.
  • Using a package and environment management tool facilitates portability and reproducibility of (data) science workflows.
  • Conda solves both the package and environment management problems and targets multiple programming languages. Other open source tools solve either one or the other, or target only a particular programming language.
  • Not only for Python.

Working with Environments

Workspace for Conda environments

We are going to change your working directory into introduction-conda for maintaining a consistent workspace for our Conda environments.

$ cd introduction-conda

Here, we are going to use as example a small DNA long-read sequencing dataset from Oxford Nanopore Technologies. This dataset consists of whole genome sequencing of three Canadian strains of Escherichia coli from biosolid samples. The dataset is already available for you in your introduction-conda directory.

For demonstration purposes, we are going to use SeqKit.

SeqKit is a cross-platform and ultrafast toolkit for FASTA/FASTQ file manipulation. It allows us to:

  • Obtain simple statistics on sequencing data.
  • Perform format conversion
  • Search through large files
  • Edit

Once you are inside your introduction-conda directory, we can try to obtain simple statistics (e.g. total number of reads, mean read length, min read length, max read length) using the command:

$ seqkit stats SRR23100672.fastq

After running it, you should see an error message saying that: seqkit is not available. But it can be installed if you contact your system admin. But instead of installing this tool system-wide, we can first create a conda environment to install whichever tools we need. These tools will only exist within the environment.

Creating a conda environment

$ conda create --name myenv

During the installation, we can already add software to be installed in our newly created environment. You do not need to create it with packages at this moment.

$ conda create --name myenv \
  python3.6 \
  numpy

Activating a Conda environment

After creating a Conda environment, it is not automatically activated. If you look at your terminal, you may notice (base) at the beginning of the prompt. This indicates that Conda’s base environment is currently active.

When an environment is active, commands you run in the terminal will use software available in that environment. This explains why our earlier attempt to run seqkit failed: seqkit was not installed in base.

We can now activate our newly created environment:

$ conda activate myenv

After activation, you should notice that (base) in your terminal prompt changes to (myenv):

(myenv) $

This indicates that myenv is now the active Conda environment. Commands you run will now have access to the software and dependencies installed in this environment.

At any time, you can also deactivate the current environment to return to base, or to activate another virtual environment. Note that you do not need to specify the name of the environment for deactivation.

$ conda deactivate

Installing a package in a Conda environment

Make sure that your environment is activated first.

$ conda activate myenv

The internet has extensive documentation on specific packages/software. However, we can use the search command to look for all the listed versions of a package as well as its channel of origin. Here, we want to install seqkit. Let’s investigate the available software. seqkit comes from the bioconda channel.

$ conda search bioconda::seqkit

It listed several versions of the tool. Sometimes the functionality that we need was modified or no longer exists in the most recent version of a package. We can explore the functionalities in the official documentation of the software.

During installation, if you do not specify a version, it will install the latest. For our use case, it is more than enough. Let’s install it:

$ conda install bioconda::seqkit

It will ask for your confirmation to install it:

The following NEW packages will be INSTALLED:
seqkit             bioconda/linux-64::seqkit-2.13.0-he881be0_0 
Proceed ([y]/n)?

You can type y and hit enter.

Now, seqkit is successfully installed in your conda environment and ready for use.

Using software from within a Conda environment

Once the Conda environment is activated and has the desired software installed, it is ready to be used. Let’s now try to use seqkit again:

$ seqkit stats SRR23100672.fastq
file               format  type  num_seqs     sum_len  min_len  avg_len  max_len
SRR23100672.fastq  FASTQ   DNA     40,436  68,795,095       51  1,701.3  170,914

It did not return the error message from when we tried to execute seqkit from the base. Let’s interpret the results:

  • file: The input file we executed the seqkit stats command.
  • format: The format of the input file.
  • type: The alphabet type in the input file (DNA, RNA, Protein).
  • num_seqs: The total number of sequences.
  • sum_len: Total length of all sequences combined.
  • min_len: Shortest sequence length.
  • avg_len: Average sequence length.
  • max_length: Longest sequence length.

Apart from the base execution of seqkit stats, there is an useful flag to get extra information:

  • -a,-all.

Activity

Let’s try to execute the seqkit stats with the -a,-all flag to see what happens.

How to find environments that exist in my machine

You can check all the environments that were created with conda in your machine. By default, environments live in the envs/ directory of your miniforge3 directory.

  • First, go to your home directory
$ cd /home/ubuntu
  • Then, you can ls into the miniforge3/envs
$ ls miniforge3/envs
  • This will list the names of all your created environments.

How to find out packages installed in an environment

If you ever forget what exactly has been installed in a particular Conda environment, you can run the conda list command. The `–name`` flag looks for the installed software in the listed environment.

$ conda list --name myenv

How to delete an environment

Occasionally, you will want to delete an entire environment. Perhaps, you were experimenting with conda commands and you created an environment you have no intention of using; perhaps, you no longer need an existing environment and just want to get rid of cruft on your machine. Whatever the reason, the command to delete an environment is:

By default, the conda remove command will delete packages. You can either specify the name of the package to be deleted:

$ conda remove --name myenv seqkit

If you add the flag --all, the entire environment is deleted.

$ conda remove --name myenv --all

However, the environment needs to be deactivated first.

$ conda deactivate

Sharing environments

When working on a collaborative research project it is often the case that your operating system might differ from the operating systems used by your collaborators. Similarly, the operating system used on a remote cluster to which you have access will likely differ from the operating system that you use on your local machine. In these cases it is useful to create an operating system agnostic environment file which you can share with collaborators or use to re-create an environment on a remote cluster.

How do I share my enviroment with others?

In order to make sure that your environment is truly shareable, you need to make sure that that the contents of your environment are described in such a way that the resulting environment file can be used to re-create your environment on Linux, Mac OS, and Windows. Conda uses YAML (“YAML Ain’t Markup Language”) for writing its environment files. YAML is a human-readable data-serialization language that is commonly used for configuration files and that uses Python-style indentation to indicate nesting.

Creating your project’s Conda environment from a single environment file is a Conda “best practice”. Not only do you have a file to share with collaborators but you also have a file that can be placed under version control which further enhancing the reproducibility of your research project and workflow.

The structure of a .yml file is:

name: myenv
dependencies:
  - ipython=7.13
  - matplotlib=3.1
  - pandas=1.0
  - pip=20.0
  - python=3.6
  - scikit-learn=0.22

Let’s create a environment.yml based on the existing software we have installed in our environment:

$ conda env export --name myenv --file environment.yml

Now your entire environment is represented as environment.yml. This does not mean this environment has already been created.

To create an environment from a .yml file, you can:

$ conda env create --file environment.yml