This PR:

Makes some significant changes to the TGA workflow by aiming to use the UMIs during the deduplication stage. This is a feature that only exists on the later versions of Sentieon.

The PR is tested on Sentieon version: sentieon-genomics-202308.03 which has this feature.

As this new feature was implemented it required some further changes to the workflows which ended up becoming a quite large refactoring. But in terms of effects on the quality of the analysis the only relevant changes are probably:

• using UMIs in the dedup for all TGA workflows
• adding adapter-trimming to the UMI workflow
• updating Sentieon version

Some more info regarding using the UMIs in the dedup step. It could rescue a significant number of reads wrongfully discarded as duplicates with a purely position-based approach, and could probably replace the UMI workflow for many use-cases as error-correction with 3,1,1 approach may not be necessary to avoid false positives in the range of VAFs commonly looked for in the samples previously run in the UMI workflow (0.5 - 2% VAF)

For more info see user story: #1361

Further necessary features before merging this:

• Replace Vardict with TNscope (OR verify that VarDict is performing better / same with these changes): feat: replace vardict with tnscope #1429
• Rebuild TGA PONs with the new TGA UMI collapsed bams feat: update cnvkit pons #1465

Extra information:

Extra note:

• Cap base quality script was added because Manta crashes if it detects bases in the reads with a higher quality than 70, which some UMI reads get after the UMI collapse
• samtools collate + samtools fixmate was added to the TGA workflow because of bug in the new Sentieon dedup consensus tool
• awk script and samtools calmd added to fix further issues in bamfile, documented here: [Bug] Sentieon Dedup with UMI consensus BAM issues #1472 and in the google sheet: https://docs.google.com/spreadsheets/d/1IZo1hcfYte2Vd7TjzQ-SjdX1wOqk9rHSNMQXvsDJbUM/edit?gid=573365273#gid=573365273

Blocked by:

• Update Sentieon: [User Story] Update Sentieon #1384
  This update requires updating Sentieon as the current version in production: sentieon-genomics-202010.02
  Does not contain the consensus option for LocusCollector which is the step prior to Dedup:

Current prod:

Latest Sentieon version:

Issues to consider:

• Percent duplicate metrics:

After implementing this, we are no longer able to retrieve % Duplicates and Optical Duplicates info from the ".metrics" file from dedup. The values are "0". How can this be fixed? I have notified Sentieon about this issue and they say that they will add more useful statistics to the report in the next version.

At the moment I have added standard non UMI Sentieon Dedup just to get the PERCENT DUPLICATES stats. But is it worth it?

UPDATE: In the latest version of Sentieon 202308.03 they claim to have added support for % Duplicates from the --consensus mode from Dedup, so maybe this extra rule can be removed!

Comparing the percent duplicates between using Dedup with or without the UMI consensus options shows that they are a bit different, which makes sense as we're now using UMIs to identify duplicates as well as position. With roughly 4% less duplicates seen in this example case while using UMIs. Meaning that we're should be both getting a better calculation of duplicates which will be good for future evaluations of lab-methods, and at the same time saving more reads.

With this new version of sentieon we can safely remove the extra Sentieon dedup!

• Collapsed singleton or pairs?

Are overlapping mates in read-pairs maintained as Pairs or Singletons after UMI collapse (as in the other Sentieon UMI collapse tool)? I checked and they are singletons when the mates are overlapping.

I discussed with Sentieon and colleagues and it should be fine: #1361 But it would be good to add more SVs from TGA to the validation set and make sure we can find them.

Graph overview

TGA T+N

TGA T+N (3,1,1 UMI workflow)

TGA T+N (QC)

WGS T+N

TGA CNVkit PON

Added

• UMI extraction and deduplication to TGA workflow
• Adapter trimming of fastqs to UMI workflow
• Cap base quality in bam for Manta input

Changed

• Refactored multi workflow rule-files to separate files to decrease complexity
• Refactored output files to in general comply with format {sample_type}.{sample_name}
• Replaced Picard QC tools with matching Sentieon QC tools

Removed

• UMI specific rules for UMI-extraction and alignment (using new TGA-rules instead)
• Fastq and UMI trimming command-line options

Documentation

• N/A
• Updated Balsamic documentation to reflect the changes as needed for this PR.
  • [Document Name]

Tests

Feature Tests

• All QC metrics are still detectable in deliverables yaml for TGA T+N and WGS T+N
• [ ]

Aggregate QC metric comparison to v15.0.0

Sheet with QC metric comparison:
https://docs.google.com/spreadsheets/d/1phztyRT1VS8ud6bf0JWwFo5PB2yWB0YgFXz_ZnXSF7U/edit?gid=0#gid=0

Notes on QC metrics:

In general QC metrics are improved, but at the cost of somewhat longer run-times, increased by around 50%. This is due to the need of some extra rules for extracting UMIs and post-processing the bamfiles (the cap_base_qualities script and the Sentieon UMI extract rules, but there's no multi-threading option for the UMI extract). However hopefully this will be compensated by replacing VarDict with TNscope, we'll have to see, either way the TGA workflow is already quite fast so while it's a negative development it shouldn't be that bad.

On the positive side the QC metrics show a increase in coverage of between 2-20% after using the UMIs, which should increase the quality of the analysis and reduce the amount samples failing coverage thresholds in production.

There's also a quite significant change in the number of variants called by VarDict in the TGA T/N case, of around 17%. But I will investigate this change in the associated PR where VarDict will be replaced by TNscope: #1429

• QC metrics look good and we can proceed without further investigation.

Pipeline Integrity Tests

• Report deliver (generation of the .hk file)
  • N/A
  • Verified
• TGA T/O Workflow
  • N/A
  • Verified
• TGA T/N Workflow
  • N/A
  • Verified
• UMI T/O Workflow
  • N/A
  • Verified
• UMI T/N Workflow
  • N/A
  • Verified
• WGS T/O Workflow
  • N/A
  • Verified
• WGS T/N Workflow
  • N/A
  • Verified
• QC Workflow
  • N/A
  • Verified
• PON Workflow
  • N/A
  • Verified

Clinical Genomics Stockholm

Checklist

For Developers

• PR Description
  • Provided a comprehensive description of the PR.
  • Linked relevant user stories or issues to the PR.
• Documentation
  • Verified and updated documentation if necessary.
• Tests
  • Described and tested the functionality addressed in the PR.
  • Ensured integration of the new code with existing workflows.
  • Confirmed that meaningful unit tests were added for the changes introduced.
  • Checked that the PR has successfully passed all relevant code smells and coverage checks.
• Review
  • Addressed and resolved all the feedback provided during the code review process.
  • Obtained final approval from designated reviewers.

For Reviewers

• Code
  • Code implements the intended features or fixes the reported issue.
  • Code follows the project's coding standards and style guide.
• Documentation
  • Pipeline changes are well-documented in the CHANGELOG and relevant documentation.
• Tests
  • The author provided a description of their manual testing, including consideration of edge cases and boundary
    conditions where applicable, with satisfactory results.
• Review
  • Confirmed that the developer has addressed all the comments during the code review.