Beta support for configurable dependency resolution & Biocontainers. #214

jmchilton · 2016-10-21T20:25:21Z

Consider the included tool seqtk_seq.cwl. It includes the following SoftwareRequirement hint:

hints:
  SoftwareRequirement:
    packages:
    - package: seqtk
      version:
      - r93

I'm not happy that version is a list - but I can live with it for now I guess.

If cwltool is executed with the hidden --beta-conda-dependencies flag, this requirement will be processed by galaxy-lib, Conda will be installed, and seqtk will be installed, and a Conda environment including seqtk will be setup for the job.

virtualenv .venv
. .venv/bin/activate
python setup.py install
pip install galaxy-lib
cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json

Additional flags are available to configure dependency resolution in a more fine grained way - using Conda however has a number of advantages that make it particularily well suited to CWL. Conda packages are distributed as binaries that work across Mac and Linux and work on relatively old version of Linux (great for HPC). Conda also doesn't require root and supports installation of multiple different versions of a package - again these factors make it great for HPC and non-Docker targets.

The Biocontainers project (previously Biodocker) dovetails nicely with this. Every version of every Bioconda package has a corresponding best-practice (very lightweight, very small) Docker container on quay.io (assembled by @bgruening and colleagues). There are over 1800 such containers currently.

Continuing with the example above, the new --beta-use-biocontainers flag instructs cwltool to fetch the corresponding Biocontainers container from quay.io automatically or build one to use locally (required for instance for tools with multiple software requirements - fat tools).

cwltool --beta-use-biocontainers tests/seqtk_seq.cwl tests/seqtk_seq_job.json

These containers contain the same binaries that the package would use locally (outside of Docker). Therefore this technique allows cross platform reproducibility/remixability across CWL, Galaxy, and CLI - both inside and outside of Docker.

My sincerest hope is that we move away from CWL-specific Dockerfiles. For less effort, a community bioconda package can be made and the result can be used in many more contexts. The Docker image will then be maintained by the community Biocontainer project.

tetron · 2016-10-21T21:35:22Z

This is awesome! I will review at the soonest opportunity (likely early next week).

"CWL-specific Dockerfiles" I think you mean Docker-specific CWL files?

tetron · 2016-10-22T02:37:02Z

I definitely want this feature, but I'd like to see some more work before merging. Are you able to continue working on this branch in response to comments? Otherwise can see about working off of this branch.

The intended design of SoftwareRequirement is to avoid picking favorites with the packaging system. The specs field lists URIs for one or more concrete packages to fulfill the abstract requirement. For example:

hints:
  SoftwareRequirement:
    packages:
    - package: seqtk
      version:
      - r93
      specs:
      - https://anaconda.org/bioconda/seqtk
      - https://packages.debian.org/jessie/seqtk

Resolution should work something like this:

Determine which resolvers (conda, pip, apt, yum, etc.) are available.
Pass the packages list to each resolver
Each resolver looks at specs for each package and matches the prefix (eg https://anaconda.org/bioconda/, https://packages.debian.org/)
For each resolver that matched one or more packages, generate a fragment of shell script that installs the package.
Combine the shell script fragments with the tool execution command to get the job script.

tetron · 2016-10-22T03:02:55Z

DependenciesConfigruation is misspelled.

Reviewing #212, I'm not sure why a separate PYTHON_RUN_SCRIPT is necessary. Everything that it does can be accomplished generating shell code?

jmchilton · 2016-10-24T17:55:25Z

@tetron I don't think the specs should strictly be needed - I can look at seqtk and find it Conda, Brew, and Debian. Same is going to hold for most packages in our experience.

Back in #93 I made the same argument passionately - that time about bwa.

do we have a long, long list of IDs in every tool definition?

This remains a false dichotomy and a misstatement of my position. The proposed solution works fine in 95% of cases - it should surprise no one to learn that Homebrew, Conda, MSI's enviornment modules, Debian, and Nix all call bwa... bwa. Small variations will of course pop up - but those can be solved with site or package-manager specific mappings on the backend - small tweaks - there is no need for long lists of IDs in tools.

That comment was +1'd by @pvanheus who has a lot more experience adapting software inventories these platforms - Galaxy and cwl.

The tool author shouldn't need to know every last package manager that this may be deployed against - forcing them to know this a priori is a mistake and is a recipe for non-future facing, brittle tools.

If instead you want to use the spec field as an optional hint when the answer isn't the intuitive, natural one, then I think this idea is acceptable and I'm more than happy to add it to the galaxy-lib stuff myself. The resulting improvement would benefit Galaxy at the margins also. But it is the margins - I assure you.

If you ignore the spec thing however - --beta-dependency-resolvers-configuration can set up cascades of resolvers the way you described. The conda flag is just a shortcut to say - install and enable only the conda resolver. Which is nice because it is the Galaxy communities' best practice and it has reproducibility with the Biocontainers.

jmchilton · 2016-10-24T17:57:34Z

@tetron If you can accomplish #212 without the Python script that would be awesome sauce. I think that would be a big improvement over my attempt. I didn't really know how to accomplish the sort of safe command handling inside of a shell script.

jmchilton · 2016-10-24T19:49:50Z

@tetron Rebased with noted spelling fix - thanks for the catch!

tetron · 2016-10-24T22:04:00Z

If instead you want to use the spec field as an optional hint when the answer isn't the intuitive, natural
one, then I think this idea is acceptable and I'm more than happy to add it to the galaxy-lib stuff myself.
The resulting improvement would benefit Galaxy at the margins also. But it is the margins - I assure you.

Ok, this seems reasonable. In that case, the behavior should be something like, check specs for an explicit package identifier for the resolver, if found, use that; if not, then try the default name from the package field.

Let me see if I'm following: the strategy is to generate shell script which does the package installation prior to running the command. This may run on a separate node (or VM or container environment) from the workflow engine. How does this work with multiple package managers or package versions? Does it generate an if-then-else chain?

jmchilton · 2016-11-02T15:22:56Z

@tetron So a limitation of the current implementation is that it does the install or generates the shell code to execute runs on the node running cwltool. (This works on fine on clusters that have a certain directory mounted across them and in Galaxy land our remote job execution tool Pulsar knows how to resolve dependencies so for jobs without a shared file system this would happen closer to the job.) I guess cwltool is somewhat explicitly single node so this shouldn't be a problem right? Explicit support for something like Toil would be great - I'd be happy to field PRs to galaxy-lib to enable that or do it myself if I sense enough eagerness for it.

If multiple "package managers" are used (the term is a misnomer since something like Galaxy packages or environment modules are not package mangers) - each may resolve packages individually and produce its own string for injecting the environment into the shell.

This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can. To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this: ``` <requirement type="package" version="2.2.31" name="blast+"> ``` Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both. This introduces the following syntax for tools with profile >= 16.10: ``` <requirement type="package" version="2.2.31" name="blast+"> <specification uri="https://anaconda.org/bioconda/blast" /> <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" /> </requirement> ``` This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so). This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10: ``` <requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/[email protected],https://packages.debian.org/jessie/[email protected]">blast+</requirement> ``` This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies. For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.

cwl-bot · 2016-11-15T11:20:02Z

Can one of the admins verify this patch?

This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can. To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this: ``` <requirement type="package" version="2.2.31" name="blast+"> ``` Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both. This introduces the following syntax for tools with profile >= 16.10: ``` <requirement type="package" version="2.2.31" name="blast+"> <specification uri="https://anaconda.org/bioconda/blast" /> <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" /> </requirement> ``` This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so). This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10: ``` <requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/[email protected],https://packages.debian.org/jessie/[email protected]">blast+</requirement> ``` This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies. For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage. Conflicts: lib/galaxy/tools/deps/__init__.py test/functional/tools/samples_tool_conf.xml

jmchilton · 2017-06-27T08:24:19Z

And we are up and running on this branch again at the GCC 2017 Hackathon. Just summarize what I have done and some conversations...

So I have rebased this branch - there were minor updates on both sides that needed to be adjusted for. The biggest changes were do to modifications to the way default container was handled in this repository. I think what is here is a bit hacky now but it was in my previous attempt as well. The previous variant of this is backed up and can viewed at https://github.com/common-workflow-language/cwltool/compare/master...jmchilton:galaxy_deps_old?expand=1.
I reran the biocontainers and the conda examples in the test above - they both still work properly on my new devbox.
Specs have been implemented upstream in Implement Dependency Resolver Mapping galaxyproject/galaxy#3444 - after a big debate we have decided not to expose that concept in Galaxy tools for now - but they should be available for cwltool.

So in response to @tetron comment:

In that case, the behavior should be something like, check specs for an explicit package identifier for the resolver, if found, use that; if not, then try the default name from the package field.

This is in fact how it works now I think - I will follow up with additional tests for that today to demonstrate the functionality.
Implement Dependency Resolver Mapping galaxyproject/galaxy#3444 also implemented mapping for environment modules. I will try to writeup some documentation about how to leverage that from cwltool. This could be a powerful tool for adapting CWL tools for HPC centers. (Though Conda is even better 😉 ).
@mr-c requested this be more of a plugin for non-Conda stuff again - I restated that I would definitely field PRs for integrating Debian resolution and we support many non-Conda package approaches - including Homebrew, custom scripts that modify the environment (i.e. "Galaxy packages"), and environment modules.
galaxy-lib now can build Singularity containers on-demand from these dependency specifications - if I have time I'll try to implement Singularity support in cwltool to utilize this as well.

So my TODO list for the rest of the hackathon is:

Add example for specs.*
Add example for environment modules.*
Redo the type signatures for galaxy-lib - I think they are broken and breaking the build.*
Check if this branch against toil - see if the concepts work.
Implement singularity support in cwltool.
Implement more Debian support in galaxy-lib.

These last two won't actually happen - but they sound fun.

jxtx · 2017-06-27T13:39:28Z

Tested using --strict --no-container --beta-conda-dependencies on https://github.com/bxlab/vision-workflows/blob/master/chipseq_tf.cwl, works great!

mr-c · 2017-06-27T15:40:38Z

@jmchilton https://travis-ci.org/common-workflow-language/cwltool/jobs/247570204#L195

mr-c · 2017-06-29T08:20:37Z

typeshed/2.7/galaxy/tools/deps/__init__.pyi

+log = ...  # type: Any
+CONFIG_VAL_NOT_FOUND = ...  # type: Any
+
+def build_dependency_manager(config): ...


This needs a full type signature. It returns a DependencyManager, but I am unsure of what type config is.

mr-c · 2017-06-29T08:21:44Z

typeshed/2.7/galaxy/tools/deps/__init__.pyi

+    def __init__(self, default_base_path, conf_file: Optional[Any] = ..., app_config: Any = ...) -> None: ...
+    def get_resolver_option(self, resolver, key, explicit_resolver_options: Any = ...): ...
+    def get_app_option(self, key, default: Optional[Any] = ...): ...
+    def dependency_shell_commands(self, requirements, **kwds): ...


This needs a full type signature.

mr-c · 2017-06-29T08:22:24Z

typeshed/2.7/galaxy/tools/deps/requirements.pyi

+    def to_dict(self): ...
+    def copy(self): ...
+    @staticmethod
+    def from_dict(dict): ...


This needs a full type description

mr-c · 2017-06-29T08:23:03Z

typeshed/2.7/galaxy/tools/deps/requirements.pyi

+    tool_requirements = ...  # type: Any
+    def __init__(self, tool_requirements: Optional[Any] = ...) -> None: ...
+    @staticmethod
+    def from_list(requirements): ...


This needs a full type description

mr-c · 2017-06-29T08:30:28Z

cwltool/draft2tool.py

    def __init__(self, toolpath_object, **kwargs):
        # type: (Dict[Text, Any], **Any) -> None
        super(CommandLineTool, self).__init__(toolpath_object, **kwargs)
+        self.find_default_container = kwargs["find_default_container"]


This line is causing multiple tests to fail.

mr-c · 2017-06-29T09:16:20Z

cwltool/main.py

        _logger.addHandler(defaultStreamHandler)


+COMMAND_WITH_DEPENDENCIES_TEMPLATE = string.Template("""#!/bin/bash


Can the rest of this be moved to an external file?

jmchilton · 2017-07-03T12:26:16Z

I have refactored the dependency handling adapter code out of main.py and into a new software_requirements module.
I have refined the type signatures for galaxy-lib for the recent changes to cwltool and galaxy-lib.

@bgruening

Consider the included tool ``seqtk_seq.cwl``. It includes the following SoftwareRequirement hint: ``` hints: SoftwareRequirement: packages: - package: seqtk version: - 1.2 ``` I'm not happy that ``version`` is a list - but I can live with it for now I guess. If cwltool is executed with the hidden ``--beta-conda-dependencies`` flag, this requirement will be processed by galaxy-lib, Conda will be installed, and seqtk will be installed, and a Conda environment including seqtk will be setup for the job. ``` virtualenv .venv . .venv/bin/activate python setup.py install pip install galaxy-lib cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json ``` Additional flags are available to configure dependency resolution in a more fine grained way - using Conda however has a number of advantages that make it particularily well suited to CWL. Conda packages are distributed as binaries that work across Mac and Linux and work on relatively old version of Linux (great for HPC). Conda also doesn't require root and supports installation of multiple different versions of a package - again these factors make it great for HPC and non-Docker targets. The Biocontainers project (previously Biodocker) dovetails nicely with this. Every version of every Bioconda package has a corresponding best-practice (very lightweight, very small) Docker container on quay.io (assembled by @bgruening and colleagues). There are over 1800 such containers currently. Continuing with the example above, the new ``--beta-use-biocontainers`` flag instructs cwltool to fetch the corresponding Biocontainers container from quay.io automatically or build one to use locally (required for instance for tools with multiple software requirements - fat tools). ``` cwltool --beta-use-biocontainers tests/seqtk_seq.cwl tests/seqtk_seq_job.json ``` These containers contain the same binaries that the package would use locally (outside of Docker). Therefore this technique allows cross platform reproducibility/remixability across CWL, Galaxy, and CLI - both inside and outside of Docker. My sincerest hope is that we move away from CWL-specific Dockerfiles. For less effort, a community bioconda package can be made and the result can be used in many more contexts. The Docker image will then be maintained by the community Biocontainer project. Rebased with correct spelling of DependenciesConfiguration thanks to @tetron.

jmchilton · 2017-07-07T19:44:28Z

@mr-c This branch's README now contains lots of documentation about SoftwareRequirements including how to resolve them with environment modules, custom package scripts, and Conda. It covers mapping to local infrastructure (for admins) as well mapping to multiple package managers from the tools themselves (for tool developers). It has many example calls that should work without external dependencies covering most of the test data I added and that can be run to try these things out today. It also contains some external links and information about installing the optional dependencies required to run these.

Here are the docs - https://github.com/jmchilton/cwltool/tree/galaxy_deps#leveraging-softwarerequirements-beta.

jmchilton · 2017-07-07T19:45:04Z

@mr-c Consider this comment documentation that I agree to allow uploading of galaxy-lib stubs to typeshed. Hope this is good enough?

mr-c · 2017-07-07T20:01:40Z

README.rst

+
+  $ pip install 'cwltool[deps]'
+
+Installing cwltool in this fashion enables several new command line options.


Hmm.. should we hide them when the deps extra isn't modified? What happens if they are called and galaxy-lib is missing?

I have clarified this behavior with #459 and improved it a bit.

mr-c · 2017-07-07T20:04:28Z

Merging as this is tremendous; we can add additional polish later. Thank you @jmchilton !

jmchilton force-pushed the galaxy_deps branch from cfc9da0 to 1625611 Compare October 24, 2016 19:49

jmchilton mentioned this pull request Nov 2, 2016

Allow unambiguous specification of requirement names galaxyproject/galaxy#3117

Closed

mr-c mentioned this pull request Mar 2, 2017

support plugins for third party CWL registry APIs, etc #314

Open

8 tasks

jmchilton force-pushed the galaxy_deps branch from 1625611 to 7414ee0 Compare June 27, 2017 07:31

mr-c reviewed Jun 29, 2017

View reviewed changes

mr-c requested changes Jun 29, 2017

View reviewed changes

mr-c reviewed Jun 29, 2017

View reviewed changes

jmchilton force-pushed the galaxy_deps branch 3 times, most recently from 450f646 to d25285c Compare July 3, 2017 11:42

Merge branch 'master' into galaxy_deps

fd2ac01

mr-c approved these changes Jul 7, 2017

View reviewed changes

mr-c merged commit 9900de0 into common-workflow-language:master

		_logger.addHandler(defaultStreamHandler)


		COMMAND_WITH_DEPENDENCIES_TEMPLATE = string.Template("""#!/bin/bash


		$ pip install 'cwltool[deps]'

		Installing cwltool in this fashion enables several new command line options.

Uh oh!

Beta support for configurable dependency resolution & Biocontainers. #214

Beta support for configurable dependency resolution & Biocontainers. #214

Uh oh!

Conversation

jmchilton commented Oct 21, 2016

Uh oh!

tetron commented Oct 21, 2016

Uh oh!

tetron commented Oct 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tetron commented Oct 22, 2016

Uh oh!

jmchilton commented Oct 24, 2016

Uh oh!

jmchilton commented Oct 24, 2016

Uh oh!

jmchilton commented Oct 24, 2016

Uh oh!

tetron commented Oct 24, 2016

Uh oh!

jmchilton commented Nov 2, 2016

Uh oh!

cwl-bot commented Nov 15, 2016

Uh oh!

jmchilton commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jxtx commented Jun 27, 2017

Uh oh!

mr-c commented Jun 27, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmchilton commented Jul 3, 2017

Uh oh!

jmchilton commented Jul 7, 2017

Uh oh!

jmchilton commented Jul 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mr-c commented Jul 7, 2017

Uh oh!

tetron commented Oct 22, 2016 •

edited

Loading

jmchilton commented Jun 27, 2017 •

edited

Loading