This program will annotate, aggregate, and summarize clinical variant information from the IonTorrent suite
- A brief graphical overview of the pipeline can be found here.
Sequencing results are accessed directly from the lab's IonTorrent server via scripts using ssh
and rsync
. After downloading files for a given run to the local system, VCF formatted variant call files are annotated and summarized to identify mutations of known significance (ANNOVAR, bcftools
), while BAM formatted coverage files are visualized with IGV. An HTML formatted report is generated from variant summary information and IGV snapshots. Results can be easily emailed to clinicians for review with the provided script.
Development of the following items is currently planned for the future:
-
Per-sample reports showing variant summary table and clinical interpretation of variants supplied by the Weill Cornell Precision Medicine Knowledgebase.
-
Analysis review feedback system to mark sequencing artifacts and remove them from report output
-
Deposition of pipeline output in a central database (REDCap, or other)
[[ A full HTML version of the report can be previewed here or here. ]]
An analysis overview report displays the significant variants found across all samples in the run. 'SC' sensitivity control samples are shown in a separate table (hidden by default).
IGV snapshots shown for all significant variants. For low frequency variants, a "long view" snapshot is included to ensure mutations can be seen amongst reads. If available, 'NC' control sample is included on the lower track.
- First, clone this repo:
git clone --recursive https://github.com/stevekm/reportIT.git
cd reportIT
- Then, run the
dir_setup.sh
script;
./dir_setup.sh
This should set up the bin
and ref
directories, along with creating and symlinking the external input
, output
, and data
directories. You should verify these symlinks and directories, and then populate the data
directory with files required for the pipeline (see the 'Data directory' section, below). You should also set up automatic ssh for your IonTorrent server as described here.
- Review the items described in the Files & Directories and Pipeline Settings sections to make sure everything is set properly.
Before you can run the pipeline, you need to know which runs are available on your IonTorrent server. The following steps require that your data/server_info.txt
file is set correctly, as described below. It is also recommended to have ssh key authentication set up for your user account on the IonTorrent server.
If you only want to know which runs on the IonTorrent server are not present on your local system, you can use this script:
code/check_for_new_runs.py
By default, it will validate each missing run to make sure that IonTorrent sequencing data has been produced in the remote run directory. It will also automatically create an unpaired samplesheet for the missing runs. If you inlclude the -d
argument to the script, it will also download the missing runs entered on the sample sheet produced.
The following script will log into your IonTorrent server, and output a list of all run directories found:
code/get_server_run_list.sh
The best way to run the reportIT pipeline is by using a samplesheet. These are stored in the samplesheets
directory by default, and an example can be found here. A samplesheet must be in TSV (tab-separated) format, preferably with one run ID per line. If two runs should be treated as a 'pair', then both run ID's should be on the same line. Note that paired run processing only affects report and IGV snapshot generation, not downloading or annotation.
The best way to create a samplesheet is to use the make_samplesheet.py
script. This script can take any number of unpaired run ID's, and a single set of paired ID's.
code/make_samplesheet.py unpaired_run1 unpaired_run2 -p paired_run3.1 -p paired_run3.2
A samplesheet produced this way will look like this:
unpaired_run1
unpaired_run2
paired_run3.1 paired_run3.2
The simplest way to run the reportIT pipeline is to use the run_samplesheet.py
script, and specify which actions you would like to take on the analysis ID's specified in your sample sheet. The following actions are available:
- Download:
-d
- Annotate:
-a
- Report:
-r
The following modifiers are available:
- Submit to cluster with
qsub
:-q
Multiple actions can be combined in a single command:
# download all files, then annotate and report with qsub
$ code/run_samplesheet.py samplesheets/samplesheet.tsv -darq
NOTE: The -q
method has been configured to work with the phoenix SGE compute cluster at NYULMC, and might need to be reconfigured to work on other HPC systems.
After manually reviewing the HTML report output, pipeline results can be delivered in a pre-formatted email using the following script:
code/mail_analysis_report.sh <analysis_ID>
Multiple analysis_ID
's can be passed to email all results sequentially. All summary tables, VCF files, IGV snapshots, and the analysis overview report will be included as email attachments. Configuration for emailing is saved in the file mail_settings.sh
.
Rough estimates for pipeline completeion time are ~5-10 minutes to download all files and annotate variants, and ~5-15 minutes to create all IGV snapshots and generate reports. In total this comes to roughly 10 - 30 minutes per analysis, depending on the number of variants present.
Running the pipeline without the -q
argument will run all pipeline steps for all analyses in the current session; if you plan to do this, you should probably run the pipeline in screen
in order to allow it to run in the background indepedent of your terminal connection. Note that running with qsub
is currently disabled for the file download step, so all file downloads will always run in the current session. If you have a lot of analyses, this might take a while.
As a safety feature against undesired usage, the run_samplesheet.py
script includes self-validating features to make sure the following items are set correctly before running the pipeline:
- check that the proper
git
branch is currently in use - check that the proper output directory has been symlinked
These validations can be skipped by adding the --debug
argument to the script.
Input, output, and reference data for the program is stored external to the program's directory and is set by symlinks. These should be automatically created when you run the dir_setup.sh
script during initial installation.
The data
directory should contain the following items:
data/
|-- control_sample_IDs.txt
|-- control_sample_regex.txt
|-- email_recipients.txt
|-- actionable_genes.txt
|-- panel_genes.txt
|-- server_info.txt
`-- summary_fields.txt
Important files:
control_sample_IDs.txt
: ID's for IonTorrent samples which are used to denote 'control' samples, which should not be used for IGV snapshots. Example:
NC
SC
NTC
NC HAPMAP
HAPMAP
Sc
SC-ACROMETRIX
control_sample_regex.txt
: Regex patterns to use withgrep -F
in some scripts which try to identify NC control samples. Example:
^NC[[:space:]]
^NC[[:space:]]HAPMAP[[:space:]]
^HAPMAP[[:space:]]
SC_control_sample_IDs.txt
: ID's to identify SC control samples. Example:
SC
Sc
SC-ACROMETRIX
email_recipients.txt
: A list of email addresses to use in theTo:
field of the outgoing email with analysis results. The addresses must be on a single line, formatted as such:
email1@gmail.com, email2@gmail.com
panel_genes.txt
: A list of genes to be included in the gene panel. Example:
AKT1
ALK
APC
ATM
actionable_genes.txt
: A list of genes determined to be actionable. Example:
BRAF
EGFR
FLT3
server_info.txt
: The login info for the IonTorrent server. Must be formatted as such:
username@server_IP
This directory contains the following items:
ref
|-- cannonical_transcript_table.py
`-- hg19
|-- canonical_transcript_list.txt
|-- download_refs.txt
|-- kgXref.txt
|-- kgXref.txt.gz
|-- knownCanonical.txt
`-- knownCanonical.txt.gz
Important files:
hg19/canonical_transcript_list.txt
: A list of canonical transcripts to use with the given genome (hg19). Each gene for should have only one 'canonical transcript' given in the list. Example:
NR_026820
NM_001005484
NR_039983
NM_001005277
IDs_to_replace.csv
: Transcript ID's that should be replaced in the defaultcanonical_transcript_list.txt
list during setup, in the format ofOLD,NEW
. Example:
NM_001276760,NM_000546
Settings used by the pipeline have been saved in several files, for ease of access & modification.
-
filter_criteria.json
: Filtering criteria used identify quality variants for inclusion in the variant summary tables. The sample fileexample_filter_criteria.json
has been provided, and should be renamed tofilter_criteria.json
-
global_settings.sh
: Global pipeline locations and settings used bybash
scripts. References to hard-coded locations on your local system or IonTorrent system should be reviewed, and updated to match your criteria. -
global_settings.py
: Many of the same settings as set inglobal_settings.sh
, intended for use in Python scripts. -
mail_settings.sh
: Settings to use with thebash
email script(s).
An important aspect of the IonTorrent reporting pipeline is the ability to recognize control samples included in a run. Unlike patient samples, these samples are included in a run for quality control purposes. Since the IonTorrent system is agnostic to the nature of samples in a run, these control samples must be denoted by their sample ID entered in the system during run setup. Similarly, the reporting pipeline is only able to identify which samples are controls by their sample ID. This makes sample labeling of control samples important during wet-lab IonTorrent run setup. The following control samples are typically used:
-
SC
: Sensitivity control (positive control). This sample is expected to show a large number of mutations. Typically uses AcroMetrix Hotspot control sample. -
NC
: Negative control. DNA sample that should not have any mutations. Typically a HapMap sample. -
NTC
: No template control. No DNA included in the sample, only water.
The best practice is to label these control samples as SC
, NC
, and NTC
in every run. The ID's to be used should be entered in the appropriate settings files as described in the section Files & Directories.
These control samples have special treatment when running the pipeline. During processing, IGV snapshots will not be taken for any sample that has a label matching a control sample. Instead, the pipeline will attempt to identify the NC
control sample for a run, or pair of runs, and use it's corresponding .bam file as the lower track in IGV snapshots for all samples in the given run(s). During report generation, variants from the SC
sample will be excluded from the primary variant summary table displayed, since it is expected to have a large number of variants.
This program has been developed in a Linux environment running CentOS 6. Some scripts issue system commands which rely on standard GNU Linux utilities. The current list of all binary dependencies are contained in the file bin.txt
. Some download notes for obtaining these programs can be found in bin_downloads.txt
- Python 2.7
- pandoc version 1.12.3 or higher
- R 3.3.0 or higher
- IGV_2.3.81
- bcftools-1.3.1
- htslib-1.3.1
- samtools-1.3.1
- ANNOVAR version 2015-06-17 21:43:53 -0700 (Wed, 17 Jun 2015)
- GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
- numpy==1.11.0
- pandas==0.17.1
- rmarkdown
- optparse