Andrea Telatin
Andrea Telatin Senior bioinformatician at the Quadram Institute Bioscience, Norwich.

Nextflow DSL2 tutorial

Nextflow DSL2 tutorial

The problem

Every bioinformatics analysis can be described as a set of interconnected tasks. A primitive approach to automate the execution of multiple commands is writing a script, but this leaves a broad range of untackled issues:

  • We need to manually manage the dependencies
  • We will tailor our script to the machine we are working on: how can we migrate the pipeline to a different machine (for example, an HPC cluster or to the cloud!)?
  • How can we orchestrate each task (some tasks require fewer resources and can run in parallel, other require more memory etc.)?
  • If we need to share our pipeline will the process easy?
  • We will need to version our script using a repository
  • The final script will be probably long, is it going to be readable, and updatable?

The solution

A workflow is more than a set of instructions, so a workflow language should allow to describe the connections between tasks, and also how to configure the execution of each of them.

There are dedicated system to generate workflows (design) and to execute them. Some popular alternatives are:

  • Nextflow (a workflow language based on Groovy)
  • Snakemake (a workflow language based on Python, and modeled after GNU Make, where each task is defined)
  • CWL (a language that does not ship a single executor/interpreter, but can be used in different setups, for example using Toil)

All the solutions are rapidly evolving, and features present in one might be adopted from the other in a few months. We picked Nextflow because it was born with several key features like scalability and reproducibility, and it’s widely adopted, but you might want to have a look around and see if Snakemake (for example) can work better for you. Specifically, we start directly introducing Nextflow latest syntax (DSL2), that introduced dramatic changes to the initial implementation (giving great advantages as a result).

The workshop

With this website and its companion repository, we will build a basic de novo assembly pipeline, using tools available from BioConda and building a Nextflow pipeline using the DSL2 syntax.

Workflow scheme

The repository

The code we will develop is stored in a dedicated repository: The repository contains several “snapshots” of the pipeline to make it easier to reconstruct how it was built.

The programme

Part of the CLIMB Workshop Bioinformatics skills for Microbial Genomics (02-02-2022)