v2.12 - Aluminium Wolf - [2024-01-29]

Template

• Add a Github Action Workflow to the pipeline template that tests a successful download with 'nf-core download' (#2618)
• Use pre-commit to lint files in GitHub CI (#2635)
• Use pdiff also on gitpod for nf-test (#2640)
• switch to new image syntax in readme (#2645)
• Add conda channel order to nextflow.config (#2094)
• Fix tyop in pipeline nextflow.config (#2664)
• Remove nfcore_external_java_deps.jar from lib directory in pipeline template (#2675)
• Add function to check -profile is well formatted (#2678)
• Add new pipeline error message pointing to docs when 'requirement exceeds available memory' error message (#2680)
• add 👀👍🏻🎉😕 reactions to fix-linting-bot action (#2692)

Linting

• Fix linting of a pipeline with patched custom module (#2669)
• linting a pipeline also lints the installed subworkflows (#2677)
• environment.yml name must be lowercase (#2676)
• allow ignoring specific files when template_strings (#2686)
• lint nextflow.config default values match the ones specified in nextflow_schema.json (#2684)

Modules

• Fix empty json output for nf-core list local (#2668)

General

• Run CI-pytests for nf-core tools on self-hosted runners (#2550)
• Add Ruff linter and formatter replacing Black, isort and pyupgrade (#2620)
• Set pdiff as nf-test differ in Docker image for Gitpod (#2642)
• Fix Renovate Dockerfile updating issues (#2648 and #2651)
• Add new subcommand nf-core tui, which launches a TUI (terminal user interface) to intuitively explore the command line flags, built using Trogon (#2655)
• Add new subcommand: nf-core logo-create to output an nf-core logo for a pipeline (instead of going through the website) (#2662)
• Handle api redirects from the old site (#2672)
• Remove redundanct v in pipeline version for emails (#2667)
• add function to check -profile is well formatted (#2678)
• Update pre-commit hook astral-sh/ruff-pre-commit to v0.1.14 (#2674)
• Update pre-commit hook pre-commit/mirrors-mypy to v1.8.0 (#2630)
• Update mshick/add-pr-comment action to v2 (#2632)
• update python image version in docker file (#2636)
• Update actions/cache action to v4 (#2666)
• Update peter-evans/create-or-update-comment action to v4 (#2683) and (#2695)

v2.11.1 - Magnesium Dragon Patch - [2023-12-20]

Template

• Rename release-announcments.yml to release-announcements.yml (#2610)
• Fix nextflow.config docker.runOptions (#2607)

General

• Only dump modules.json when it is modified (#2609)

v2.11 - Magnesium Dragon + [2023-12-19]

Highlights:

• nf-test is now used for testing modules and subworkflows. 🚀 This comes with updated templates and a changed commands (good-bye nf-core modules/subworkflows create-test-yml). For more information, you can refer to the revised documentation on modules and subworkflows .
  • Every module and subworkflow now contains a tests/ directory, which includes a main.nf.test file. By running nf-core modules test or nf-core subworkflows test, you can perform the tests and generate the main.nf.test.snap snapshot file. As a general guideline, we recommend testing all outputs of modules/subworkflows. For complex cases, please refer to the nf-test assertion tutorial.
  • To assist with the conversion of modules and subworkflows from pytest to nf-test, we've added the new --migrate-pytest option for nf-core modules test and nf-core subworkflows test, which does most of the work for you.
  • The nf-core modules lint and nf-core subworkflows lint commands now also check the nf-test configuration and warn you if you are using pytest (it's really not that hard to switch 🙂 ).
• We pinned the version of nf-validation, which should make offline usage of nf-core pipelines easier.
• In nf-core download, we now prefix all absolute container URIs with docker://, which should make more pipelines work out of the box.
• We started using type hinting in the nf-core/tools codebase. This will help us to find bugs earlier and make the code more readable.

The complete list of changes can be found below.

Template

• Fix writing files to a remote outdir in the NfcoreTemplate helper functions (#2465)
• Fancier syntax highlighting for example samplesheets in the usage.md template (#2503)
• Use closure for multiqc ext.args (#2509)
• Fix how the modules template references the conda environment file (#2540)
• Unset env variable JAVA_TOOL_OPTIONS in gitpod (#2569)
• Pin the version of nf-validation (#2579)
• Disable process selector warnings by default (#2161)
• Remove docker.userEmulation from nextflow.config in pipeline template (#2580)

Download

• Add docker:// prefix for absolute container URIs as well (#2576).
• Bugfix for AttributeError: ContainerError object has no attribute absoluteURI (#2543).

Linting

• Fix incorrectly failing linting if 'modules' was not found in meta.yml (#2447)
• Correctly pass subworkflow linting test if COMPONENT.out.versions is used in the script (#2448)
• Add pyupgrade to pre-commit config and dev requirements as mentioned in #2200
• Check for spaces in modules container URLs (#2452)
• Correctly ignore timeline.enabled, report.enabled, trace.enabled, dag.enabled variables when linting a pipeline. (#2507)
• Lint nf-test main.nf.test tags include all used components in chained tests (#2572)
• Don't fail linting if md5sum for empty files are found in a stub test (#2571)
• Check for existence of test profile (#2478)

Modules

• Added stub test creation to create_test_yml (#2476)
• Replace ModulePatch by ComponentPatch (#2482)
• Fixed nf-core modules lint to work with new module structure for nf-test (#2494)
• Add option --migrate-pytest to create a module with nf-test taking into account an existing module (#2549)
• When installing modules and subworkflows, automatically create the ./modules directory if it doesn't exist (#2563)
• When .nf-core.yml is not found create it in the current directory instead of the root filesystem (#2237)
• Modules --migrate-pytest copies template scripts (#2568)

Subworkflows

• Added stub test creation to create_test_yml (#2476)
• Fixed nf-core subworkflows lint to work with new module structure for nf-test (#2494)
• Add option --migrate-pytest to create a subworkflow with nf-test taking into account an existing subworkflow (#2549)

General

• Update schema build functionality to automatically update defaults which have changed in the nextflow.config(#2479)
• Change testing framework for modules and subworkflows from pytest to nf-test (#2490)
• bump_version keeps now the indentation level of the updated version entries (#2514)
• Add mypy to pre-commit config for the tools repo (#2545)
• Use Path objects for ComponentCreate and update the structure of components templates (#2551).
• GitPod base image: swap tool installation back to conda from mamba (#2566).
• Sort the installed_by list in modules.json (#2570).
• Unset env variable JAVA_TOOL_OPTIONS in gitpod (#2569)

New Contributors

• @fa2k made their first contribution in #2472
• @NovakApis made their first contribution in #2223
• @rernst made their first contribution in #2507
• @kedhammar made their first contribution in #2483

v2.10 - Nickel Ostrich + [2023-09-25]

This new release includes new commands, template updates, and several bug fixes. Thanks to all contributors for making this possible!

🌟 Highlights

• We are now linting subworkflows nf-core subworkflows lint.
• Use nf-core create-params-file params-file with documentation and default values for all parameters in a pipeline.
• A new GitHub Action will automatically post new pipeline release messages to Mastodon, Bluesky and Twitter
• Pipelines in sync with the new template, will dump all pipeline parameters to a file called params.json after every run

✖️ Deprecations

• Deprecation of the —no-git argument for nf-core create command
• Deprecation of the —template-yaml argument for the nf-core sync command - The same information will now be automatically stored in .nf-core.yml

Template

• Fix links in multiqc_config.yml (#2372 and #2412)
• Remove default false from nextflow_schema.json (#2376)
• Add module MULTIQC to modules.config (#2377)
• Add GitHub workflow for automated release announcements (#2382)
• Update the Code of Conduct (#2381)
• Save template information to .nf-core.yml and deprecate argument --template-yaml for nf-core sync (#2388 and #2389)
• (#2397) Remove fixed Ubuntu test and added to standard testing matrix
• (#2396) Reduce container finding error to warning since the registries are not consistent.
• (#2415) Add autoMounts for apptainer.
• Remove igenomes_base from the schema, so that nf-validation doesn't create a file path and throw errors offline for s3 objects.
• Modified devcontainer permissions so that singularity can be run in Codespaces/VS Code devcontainers (Commit a103f44)
• Update Gitpod profile resources to reflect base environment settings.
• (#747) Add to the template the code to dump the selected pipeline parameters into a json file.

Download

• Improved container image resolution and prioritization of http downloads over Docker URIs (#2364).
• Registries provided with -l/--container-library will be ignored for modules with explicit container registry specifications (#2403).
• Fix unintentional downloading of containers in test for the Tower download functionality. Bug reported by @adamrtalbot and @awgymer (#2434).

Linting

• Add new command nf-core subworkflows lint (#2379)

Modules

Subworkflows

• Fix bug: missing subworkflow name when using nf-core subworkflows create (#2435)

General

• Initialise docker_image_name to fix UnboundLocalError error (#2374)
• Fix prompt pipeline revision during launch (#2375)
• Add a create-params-file command to create a YAML parameter file for a pipeline containing parameter documentation and defaults. (#2362)
• Update the Code of Conduct (#2381)
• Remove --no-git option from nf-core create (#2394)
• Throw warning when custom workflow name contains special characters (#2401)
• Bump version of nf-test snapshot files with nf-core bump-version (#2410)

v2.9 - Chromium Falcon + [2023-06-30]

A new big release with new features, template updates and bug fixes. Thanks to all contributors!

🌟 Highlights

• A new nf-core downloadcommand compatible with Nextflow Tower.
• Implementation of the plugin nf-validation for parameter validation in pipeline templates.
• Nextflow version bump to 23.04.0.

Template

• params.max_multiqc_email_size is no longer required (#2273)
• Remove cleanup = true from test_full.config in pipeline template (#2279)
• Fix usage docs for specifying params.yaml (#2279)
• Added stub in modules template ([#2277])(#2277) [Contributed by @nvnieuwk]
• Move registry definitions out of profile scope ([#2286])(#2286)
• Remove aws_tower profile ([#2287])(#2287)
• Fixed the Slack report to include the pipeline name (#2291)
• Fix link in the MultiQC report to point to exact version of output docs (#2298)
• Updates seqeralabs/action-tower-launch to v2.0.0 (#2301)
• Remove schema validation from lib folder and use Nextflow nf-validation plugin instead (#1771)
• Fix parsing of container directive when it is not typical nf-core format (#2306)
• Add ability to specify custom registry for linting modules, defaults to quay.io (#2313)
• Add singularity.registry = 'quay.io' in pipeline template (#2305)
• Add apptainer.registry = 'quay.io' in pipeline template (#2352)
• Bump minimum required NF version in pipeline template from 22.10.1 -> 23.04.0 (#2305)
• Add ability to interpret docker.registry from nextflow.config file. If not found defaults to quay.io. (#2318)
• Add functions to dynamically include pipeline tool citations in MultiQC methods description section for better reporting. (#2326)
• Remove --tracedir parameter (#2290)
• Incorrect config parameter warnings when customising pipeline template (#2333)
• Use markdown syntax in the description for the meta map channels (#2358)

Download

• Introduce a --tower flag for nf-core download to obtain pipelines in an offline format suited for seqeralabs® Nextflow Tower (#2247).
• Refactored the CLI for --singularity-cache in nf-core download from a flag to an argument. The prior options were renamed to amend (container images are only saved in the $NXF_SINGULARITY_CACHEDIR) and copy (a copy of the image is saved with the download). remote was newly introduced and allows to provide a table of contents of a remote cache via an additional argument --singularity-cache-index (#2247).
• Refactored the CLI parameters related to container images. Although downloading other images than those of the Singularity/Apptainer container system is not supported for the time being, a generic name for the parameters seemed preferable. So the new parameter --singularity-cache-index introduced in #2247 has been renamed to --container-cache-index prior to release (#2336).
• To address issue #2311, a new parameter --container-library was created allowing to specify the container library (registry) from which container images in OCI format (Docker) should be pulled (#2336).
• Container detection in configs was improved. This allows for DSL2-like container definitions inside the container parameter value provided to process scopes #2346.
• Add apptainer to the list of false positve container strings (#2353).

Updated CLI parameters

In addition, -r / --revision has been changed to a parameter that can be provided multiple times so several revisions can be downloaded at once.

Linting

• Warn if container access is denied (#2270)
• Error if module container specification has quay.io as prefix when it shouldn't have ([#2278])(https://github.com/nf-core/tools/pull/2278/files)
• Detect if container is 'simple name' and try to contact quay.io server by default (#2281)
• Warn about null/None/empty default values in nextflow_schema.json (#3328)
• Fix linting when creating a pipeline skipping some parts of the template and add CI test (#2330)

Modules

• Don't update modules_json object if a module is not updated (#2323)

Subworkflows

General

• GitPod base image: Always self-update to the latest version of Nextflow. Add pre-commit dependency.
• GitPod configs: Update Nextflow as an init task, init pre-commit in pipeline config.
• Refgenie: Create nxf_home/nf-core/refgenie_genomes.config path if it doesn't exist (#2312)
• Add CI tests to test running a pipeline when it's created from a template skipping different areas

v2.8 - Ruthenium Monkey - [2023-04-27]

Template

• Explicitly disable conda when a container profile (#2140)
• Turn on automatic clean up of intermediate files in work/ on successful pipeline completion in full-test config (#2163) [Contributed by @jfy133]
• Add documentation to usage.md on how to use params.yml files, based on nf-core/ampliseq text (#2173) [Contributed by @jfy133, @d4straub]
• Make jobs automatically resubmit for a much wider range of exit codes (now 104 and 130..145) (#2170)
• Add a clean-up GHA which closes issues and PRs with specific labels (#2183)
• Remove problematic sniffer code in samplesheet_check.py that could give false positive 'missing header' errors ([https://github.com//pull/2194]) [Contributed by @Midnighter, @jfy133]
• Consistent syntax for branch checks in PRs (#2202)
• Fixed minor Jinja2 templating bug that caused the PR template to miss a newline
• Updated AWS tests to use newly moved seqeralabs/action-tower-launch instead of nf-core/tower-action
• Remove .cff files from .editorconfig [(#2145)[https://github.com//pull/2145]]
• Simplify pipeline README (#2186)
• Added support for the apptainer container engine via -profile apptainer. (#2244) [Contributed by @jfy133]
• Added config docker.registry to pipeline template for a configurable default container registry when using Docker containers. Defaults to quay.io (#2133)
• Add tower.yml file to the pipeline template (#2251)
• Add mastodon badge to README (#2253)
• Removed quay.io from all module Docker container references as this is now supplied at pipeline level. (#2249)
• Remove CITATION.cff file from pipeline template, to avoid that pipeline Zenodo entries reference the nf-core publication instead of the pipeline (#2059).

Linting

• Update modules lint test to fail if enable_conda is found (#2213)
• Read module lint configuration from .nf-core.yml, not .nf-core-lint.yml (#2221)
• nf-core schema lint now defaults to linting nextflow_schema.json if no filename is provided (#2225)
• Warn if /zenodo.XXXXXX is present in the Readme (#2254)
• Lint all labels in a module (#2227)

Modules

• Add an --empty-template option to create a module without TODO statements or examples (#2175 & #2177)
• Removed the nf-core modules mulled command and all its code dependencies (2199).
• Take into accout the provided --git_remote URL when linting all modules (2243).

Subworkflows

• Fixing problem when a module included in a subworkflow had a name change from TOOL to TOOL/SUBTOOL (#2177)
• Fix nf-core subworkflows test not running subworkflow tests (#2181)
• Add tests for nf-core subworkflows create-test-yml (#2219)

General

• Deprecate Python 3.7 support because it reaches EOL (#2210)
• nf-core modules/subworkflows info now prints the include statement for the module/subworkflow (#2182).
• Add a clean-up GHA which closes issues and PRs with specific labels (#2183)
• update minimum version of rich to 13.3.1 (#2185)
• Add the Nextflow version to Gitpod container matching the minimal Nextflow version for nf-core (according to nextflow.config) (#2196)
• Use nfcore/gitpod:dev container in the dev branch (#2196)
• Replace requests_mock with responses in test mocks (#2165).
• Add warning when installing a module from an org_path that exists in multiple remotes in modules.json (#2228 #2239).
• Add the possibility to translate refgenie asset aliases to the ones used in a pipeline with an alias_translations.yaml file (#2242).
• Add initial CHM13 support (1988)

v2.7.2 - Mercury Eagle Patch

Patch release solving some bugs introduced with v2.7

Template

• Fix the syntax of github_output in GitHub actions (#2114)
• Fix a bug introduced in 2.7 that made pipelines hang (#2132)

Linting

• Allow specifying containers in less than three lines (#2121)
• Run prettier after dumping a JSON schema file (#2124)

General

• Only check that a pipeline name doesn't contain dashes if the name is provided by prompt of --name. Don't check if a template file is used. (#2123)
• Deprecate --enable_conda parameter. Use conda.enable instead (#2131)
• Handle json.load() exceptions (#2134)

This release fixes a sync error with the new syntax of set_output on GitHub Actions.

• Patch release to fix pipeline sync (#2110)

Another big release with lots of new features and bug fixes. Thanks to all contributors!

🌟 Highlights

• New nf-core subworkflows subcommand for creating, removing, testing, updating and finding subworkflows, see the documentation for more information.
• Every pipeline has now it's own GitHub codespace template, which can be used to develop the pipeline directly in the browser.
• Improved handling of modules and subworkflows from other repos than nf-core/modules.
• Pre-commit is now installed as a dependency, which allows us, besides other things, to run prettier on the fly even if it is not manually installed.
• Shell completion for nf-core commands, more information here.

Template

Features

• Ignore files in bin/ directory when running prettier (#2080).
• Add GitHub codespaces template (#1957)
• nextflow run <pipeline> --version will now print the workflow version from the manifest and exit (#1951).
• Add profile for running docker with the ARM chips (including Apple silicon) (#1942 and #2034).
• Flip execution order of parameter summary printing and parameter validation to prevent 'hiding' of parameter errors (#2033).
• Change colour of 'pipeline completed successfully, but some processes failed' from red to yellow (#2096).

Bug fixes

• Fix lint warnings for samplesheet_check.nf module (#1875).
• Check that the workflow name provided with a template doesn't contain dashes (#1822)
• Remove CITATION.cff file from pipeline template, to avoid that pipeline Zenodo entries reference the nf-core publication instead of the pipeline (#2059).

Linting

Features

• Add --sort-by option to linting which allows ordering module lint warnings/errors by either test name or module name (#2077).

Bug fixes

• Don't lint pipeline name if manifest.name in .nf-core.yml (#2035)
• Don't check for docker pull commands in actions_ci lint test (leftover from DSL1) (#2055).

General

Features

• Use pre-commit run prettier if prettier is not available (#1983) and initialize pre-commit in gitpod and codespaces (#1957).
• Refactor CLI flag --hide-progress to be at the top-level group, like --verbose (#2016)
• nf-core sync now supports the template YAML file using -t/--template-yaml (#1880).
• The default branch can now be specified when creating a new pipeline repo #1959.
• Only warn when checking that the pipeline directory contains a main.nf and a nextflow.config file if the pipeline is not an nf-core pipeline #1964
• Bump promoted Python version from 3.7 to 3.8 (#1971).
• Extended the chat notifications to Slack (#1829).
• Don't print source file + line number on logging messages (except when verbose) (#2015)
• Automatically format test.yml content with Prettier (#2078)
• Automatically format modules.json content with Prettier (#2074)
• Add shell completion for nf-core tools commands(#2070)

Bug fixes, maintenance and tests

• Fix error in tagging GitPod docker images during releases (#1874).
• Fix bug when updating modules from old version in old folder structure (#1908).
• Don't remove local copy of modules repo, only update it with fetch (#1881).
• Improve test coverage of sync.py and __main__.py (#1936, #1965).
• Add file versions.yml when generating test.yml with nf-core modules create-test-yml but don't check for md5sum #1963.
• Mock biocontainers and anaconda api calls in modules and subworkflows tests #1967
• Run tests with Python 3.11 (#1970).
• Run test with a realistic version of git (#2043).
• Fix incorrect file deletion in nf-core launch when --params_in has the same name as --params_out (#1986).
• Updated GitHub actions (#1998, #2001)
• Code maintenance (#1818, #2032, #2073).
• Track from where modules and subworkflows are installed (#1999).
• Substitute ModulesCommand and SubworkflowsCommand by ComponentsCommand (#2000).
• Prevent installation with unsupported Python versions (#2075).
• Allow other remote URLs not starting with http (#2061)

Modules

• Update patch file paths if the modules directory has the old structure (#1878).
• Don't write to modules.json file when applying a patch file during nf-core modules update (#2017).

Subworkflows

• Add subworkflow commands create-test-yml, create and install (#1897).
• Update subworkflows install so it installs also imported modules and subworkflows (#1904).
• check_up_to_date() function from modules_json.py also checks for subworkflows (#1934).
• Add tests for nf-core subworkflows install command (#1996).
• Function create() from modules_json.py adds also subworkflows to modules.json file (#2005).
• Add nf-core subworkflows update command (#2019).

⚠️ Please read

This release goes together with a major update on nf-core/modules that restructures DSL2 module files and updates the syntax of modules.json.

These changes have been brought about to accommodate the imminent release of shared subworkflows.

Please be aware of some points:

• ♻️ You will need to run nf-core modules update on your pipeline which should move a lot of files around
• 💀 Any module .patch files from nf-core modules patch may be lost (if you have any). Please take care to regenerate them.

If you have any issues, please drop us a line on Slack on the #tools channel 👍🏻

Template

• Add actions/upload-artifact step to the awstest workflows, to expose the debug log file
• Add prettier as a requirement to Gitpod Dockerimage
• Bioconda incompatible conda channel setups now result in more informative error messages (#1812)
• Update MultiQC module, update supplying MultiQC default and custom config and logo files to module
• Add a 'recommend' methods description text to MultiQC to help pipeline users report pipeline usage in publications (#1749)
• Fix template spacing modified by JINJA (#1830)
• Fix MultiQC execution on template #1855
• Don't skip including base.config when skipping nf-core/configs

Linting

• Pipelines: Check that the old renamed lib files are not still present:
  • Checks.groovy -> Utils.groovy
  • Completion.groovy -> NfcoreTemplate.groovy
  • Workflow.groovy -> WorkflowMain.groovy

General

• Add function to enable chat notifications on MS Teams, accompanied by hook_url param to enable it.
• Schema: Remove allOf if no definition groups are left.
• Use contextlib to temporarily change working directories (#1819)
• More helpful error messages if nf-core download can't parse a singularity image download

Modules

• If something is wrong with the local repo cache, offer to delete it and try again (#1850)
• Restructure code to work with the directory restructuring in modules (#1859)
• Make label: process_single default when creating a new module

Full Changelog

2.5.1...2.6