Module 0: Introductions and Environment Setup

Lecture

Environment Setup

This tutorial assumes use of a Linux computer with an ‘x86_64’ architecture. The rest of the tutorial should be conducted in a linux Terminal session. In other words you must already be logged into the Amazon EC2 instance as described before.

Before proceeding you must define a global working directory by setting the environment variable: ‘RNA_HOME’ Log into a server and SET THIS BEFORE RUNNING EVERYTHING.

Create a working directory and set the ‘RNA_HOME’ environment variable

mkdir -p ~/workspace/rnaseq/

export RNA_HOME=~/workspace/rnaseq

Make sure whatever the working dir is, that it is set and is valid

echo $RNA_HOME

You can place the RNA_HOME variable (and other environment variables) in your .bashrc and then logout and login again to avoid having to worry about it.

For simplicity, we are going to download a preconfigured .bashrc file to use with the following commands:

cd ~
wget http://genomedata.org/rnaseq-tutorial/bashrc_copy
mv bashrc_copy ~/.bashrc
source ~/.bashrc

Since all the environment variables we set up for the RNA-seq workshop start with ‘RNA’ we can easily view them all by combined use of the env and grep commands as shown below. The env command shows all environment variables currently defined and the grep command identifies string matches.

env | grep RNA

In order to view the contents of this file, you can type:

less ~/.bashrc

To exit the file, type q.

Environment variables used throughout this tutorial:

export RNA_HOME=~/workspace/rnaseq
export RNA_DATA_DIR=$RNA_HOME/data
export RNA_DATA_TRIM_DIR=$RNA_DATA_DIR/trimmed
export RNA_REFS_DIR=$RNA_HOME/refs
export RNA_REF_INDEX=$RNA_REFS_DIR/chr22_with_ERCC92
export RNA_REF_FASTA=$RNA_REF_INDEX.fa
export RNA_REF_GTF=$RNA_REF_INDEX.gtf
export RNA_ALIGN_DIR=$RNA_HOME/alignments/star
export RNA_PSEUDOALIGN_DIR=$RNA_HOME/alignments/kallisto

Now logout and login again.

Now if you run the following command, you should see the RNA environment variables present.

env | grep RNA

Alternatively, you could have add these enviroment variables manually if they were not part of your .bashrc. First, you can open your .bashrc file with nano by simply typing:

nano ~/.bashrc

You can now see the contents of this file. Then, you want to add the above environment variables to the bottom of the file. You can do this by copying and pasting. Once you have the variables in the file, you’ll want to type ctrl + o to save the file, then enter to confirm you want the same filename, then ctrl + x to exit nano.

Again, check all the RNA related environment variables to make sure things look right.

env | grep RNA

Note that if you are doing this course on the Google Cloud Platform instead of AWS, you should instead use this .bashrc file: http://genomedata.org/rnaseq-tutorial/bashrc_copy_gcp.sh

Tool Installation

Note

All tools are already installed in the Amazon EC2 instance. Thus, DO NOT run the installation instructions below. These are given for you to know how to install these tools on your own linux machine after the workshop. You can practice installing a tool by doing the Practical Exercise 1 at the end of this module.

Tools needed for this analysis are: samtools, bedtools, STAR, kallisto, featureCounts, FastQC, Fastp, MultiQC, gtfToGenePred, genePredToBed, how_are_we_stranded_here, R, BioConductor and other R packages. In the following installation example, the installs are local and will work whether you have root (i.e. admin) access or not. However, if root is available some binaries can/will be copied to system-wide locations (e.g., ~/bin/).

Set up tool installation location:

cd $RNA_HOME
mkdir student_tools
cd student_tools

FastQC

Installation type: download precompiled binary. Citation: s-andrews/FastQC.

cd $RNA_HOME/student_tools/
wget https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.12.1.zip
unzip fastqc_v0.12.1.zip
cd FastQC/
chmod 755 fastqc
./fastqc --help

Fastp

Installation type: download precompiled binary. Citation: PMID: 30423086

cd $RNA_HOME/student_tools/
mkdir fastp
cd fastp
wget http://opengene.org/fastp/fastp
chmod a+x ./fastp
./fastp

MultiQC

Installation type: use pip. Citation: PMID: 27312411.

Multiqc, a tool for assembling QC reports is a python package that can be installed using the python package manager pip.

pip3 install multiqc
export PATH=/home/ubuntu/.local/bin:$PATH
multiqc --help

STAR

Installation type: build C++ binary from source code using make. Citation: PMID: 23104886.

The following tool is installed by downloading a github repository git clone, and building the source code using make to run compiler commands in the “Makefile” provided with the tool. When make is run without options, it attempts the “default goal” in the make file which is the first “target” defined. In this case the first “target” is :all. Once the build is complete, we test that it worked by attempting to execute the STAR binary. Remember that the ./ in ./STAR tells the commandline that you want to execute the STAR binary in the current directory. We do this because there may be other STAR binaries in our PATH. Try which STAR to see the STAR binary that appears first in our PATH and therefore will be the one used when we specify STAR without specifying a particular location of the binary.

cd $RNA_HOME/student_tools/
git clone https://github.com/alexdobin/STAR.git
cd STAR/source
make STAR
./STAR

kallisto

Installation type: download a precompiled binary. Citation: PMID: 27043002.

The kallisto alignment free expression estimation tool is installed below simply by downloading an archive with wget, unpacking the archive with tar, and testing the binary to ensure it runs on our system.

cd $RNA_HOME/student_tools/
wget https://github.com/pachterlab/kallisto/releases/download/v0.44.0/kallisto_linux-v0.44.0.tar.gz
tar -zxvf kallisto_linux-v0.44.0.tar.gz
cd kallisto_linux-v0.44.0/
./kallisto

SAMtools

Installation type: build C++ binary from source code using make. Citation: PMID: 19505943.

The following tool is installed by downloading a compressed archive using wget, decompressing it using bunzip2, unpacking the archive using tar, and building the source code using make to run compiler commands in the “Makefile” provided with the tool. When make is run without options, it attempts the “default goal” in the make file which is the first “target” defined. In this case the first “target” is :all. Once the build is complete, we test that it worked by attempting to execute the samtools binary. Remember that the ./ in ./samtools tells the commandline that you want to execute the samtools binary in the current directory. We do this because there may be other samtools binaries in our PATH. Try which samtools to see the samtools binary that appears first in our PATH and therefore will be the one used when we specify samtools without specifying a particular location of the binary.

cd $RNA_HOME/student_tools/
wget https://github.com/samtools/samtools/releases/download/1.18/samtools-1.18.tar.bz2
bunzip2 samtools-1.18.tar.bz2
tar -xvf samtools-1.18.tar
cd samtools-1.18
make
./samtools

BEDtools

Installation type: build C++ binary from source code using make. Citation: PMID: 19505943.

The following tool is installed by downloading a compressed archive using wget, decompressing it using bunzip2, unpacking the archive using tar, and building the source code using make to run compiler commands in the “Makefile” provided with the tool. When make is run without options, it attempts the “default goal” in the make file which is the first “target” defined. In this case the first “target” is :all. Once the build is complete, we test that it worked by attempting to execute the samtools binary. Remember that the ./ in ./samtools tells the commandline that you want to execute the samtools binary in the current directory. We do this because there may be other samtools binaries in our PATH. Try which samtools to see the samtools binary that appears first in our PATH and therefore will be the one used when we specify samtools without specifying a particular location of the binary.

cd $RNA_HOME/student_tools/
git clone https://github.com/arq5x/bedtools2.git
cd bedtools2/
make
./bedtools

gtfToGenePred

Installation type: download precompiled binary.

cd $RNA_HOME/student_tools/
mkdir gtfToGenePred
cd gtfToGenePred
wget -c http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/gtfToGenePred
chmod a+x gtfToGenePred
./gtfToGenePred

genePredToBed

Installation type: download precompiled binary.

cd $RNA_HOME/student_tools/
mkdir genePredToBed
cd genePredToBed
wget -c http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/genePredToBed
chmod a+x genePredToBed
./genePredToBed

how_are_we_stranded_here

pip3 install git+https://github.com/kcotto/how_are_we_stranded_here.git
check_strandedness

Install R

#sudo apt-get remove r-base-core

#wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | sudo gpg --dearmor -o /usr/share/keyrings/r-project.gpg
#echo "deb [signed-by=/usr/share/keyrings/r-project.gpg] https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/" | sudo tee -a /etc/apt/sources.list.d/r-project.list
#sudo apt update
#sudo apt install --no-install-recommends r-base

Note, if X11 libraries are not available you may need to use --with-x=no during config, on a regular linux system you would not use this option. Also, linking the R-patched bin directory into your PATH may cause weird things to happen, such as man pages or git log to not display. This can be circumvented by directly linking the R* executables (R, RScript, RCmd, etc.) into a PATH directory.

R Libraries

Installation type: add new base R libraries to an R installation.

For this tutorial we require:

launch R (enter R at linux command prompt) and type the following at an R command prompt.

#R
#install.packages(c("devtools","dplyr","gplots","ggplot2","data.table","gprofiler2","msigdbr","ggVennDiagram","pheatmap"),repos="http://cran.us.r-project.org")
#quit(save="no")

Bioconductor

Installation type: add bioconductor libraries to an R installation. Citation: PMID: 15461798.

For this tutorial we require:

launch R (enter R at linux command prompt) and type the following at an R command prompt. If prompted, type “a” to update all old packages. NOTE: This has been pre-installed for you, so these commands can be skipped.

#R
#source("http://bioconductor.org/biocLite.R")
#biocLite(c("clusterProfiler","org.Hs.eg.db","enrichplot","pathview","KEGGREST","fgsea","DESeq2","Rsubread"))
#quit(save="no")

Installing tools from official ubuntu packages

Some useful tools are available as official ubuntu packages. These can be installed using the linux package management system apt. Most bioinformatic tools (especially the latest versions) are not available as official packages. Nevertheless, here is how you would update your apt library, upgrade existing packages, and install an Ubuntu tool called tree.

#sudo apt-get update
#sudo apt-get upgrade
#sudo apt-get install tree
#tree

Installing tools by Docker image

Some tools have complex dependencies that are difficult to reproduce across systems or make work in the same environment with tools that require different versions of the same dependencies. Container systems such as Docker and Singularity allow you to isolate a tool’s environment giving you almost complete control over dependency issues. For this reason, many tool developers have started to distribute their tools as docker images. Many of these are placed in container image repositories such as DockerHub. Here is an example tool installation using docker.

Install samtools:

docker pull biocontainers/samtools:v1.9-4-deb_cv1
docker run -t biocontainers/samtools:v1.9-4-deb_cv1 samtools --help

Install pvactools for personalized cancer vaccine designs:

#docker pull griffithlab/pvactools:latest
#docker run -t griffithlab/pvactools:latest pvacseq --help

Installing tools by Docker image (using Singularity)

Some systems do not allow docker to be run for various reasons. Sometimes singularity is used instead. The equivalent to the above but using singularity looks like the following:

#singularity pull docker://griffithlab/pvactools:latest
#singularity run docker://griffithlab/pvactools:latest pvacseq -h

Note that if you encounter errors with /tmp space usage or would like to control where singularity stores its temp files, you can set the environment variables:

#export SINGULARITY_CACHEDIR=/media/workspace/.singularity
#export TMPDIR=/media/workspace/temp

Add locally installed tools to your PATH

To use the locally installed version of each tool without having to specify complete paths, you could add the install directory of each tool to your ‘$PATH’ variable and set some other environment variables:

PATH=$RNA_HOME/student_tools/genePredToBed:$RNA_HOME/student_tools/gtfToGenePred:$RNA_HOME/student_tools/samtools-1.18:$RNA_HOME/student_tools/bam-readcount/bin:$RNA_HOME/student_tools/gffcompare-0.12.6.Linux_x86_64:$RNA_HOME/student_tools/kallisto_linux-v0.44.0:$RNA_HOME/student_tools/FastQC:$RNA_HOME/student_tools/fastp/home/ubuntu/bin/bedtools2/bin:/home/ubuntu/.local/bin:$PATH

echo $PATH

You can make these changes permanent by adding the above lines to your .bashrc file use a text editor to open your bashrc file. For example:

vi ~/.bashrc

Vi instructions

  1. Using your cursor, navigate down to the “export PATH” commands at the end of the file.
  2. Delete the line starting with PATH using the vi command “dd”.
  3. Press the “i” key to enter insert mode. Go to an empty line with you cursor and copy paste the new RNA_HOME and PATH commands into the file
  4. Press the “esc” key to exit insert mode.
  5. Press the “:” key to enter command mode.
  6. Type “wq” to save and quit vi

If you would like to learn more about how to use vi, try this tutorial/game: VIM Adventures

NOTE: If you are worried your .bashrc is messed up you can redownload as follows:

cd ~
wget http://genomedata.org/rnaseq-tutorial/bashrc_copy
mv bashrc_copy ~/.bashrc
source ~/.bashrc

PRACTICAL EXERCISE 1 - Software Installation

Assignment: Install bedtools on your own. Make sure you install it in your tools folder. Download, unpack, compile, and test the bedtools software. Citation: PMID: 20110278.

cd $RNA_HOME/student_tools/
  • Hint: google “bedtools” to find the source code
  • Hint: read the installation instructions for “Compiling from source via Github”
  • Hint: If your install has worked you should be able to run bedtools as follows:
$RNA_HOME/student_tools/bedtools2/bin/bedtools

Questions

  • What happens when you run bedtools without any options?
  • Where can you find detailed documentation on how to use bedtools?
  • How many general categories of analysis can you perform with bedtools? What are they?

Solution: When you are ready you can check your approach against the Solutions