Skip to content

Conversation

@KevinMenden
Copy link
Contributor

This PR updates nf-core modules lint to deal with the module versioning to address #1114
A modules.json template file is also added to the pipeline template.

The following was implemented:

  • check for modules.json and load it (ask to create if not there)
  • for each module check whether a git_sha exists in the modules.json
  • look whether a new commit sha is available and warn the user
  • pass the git_sha to the module_changes test, which then compares the local file contents to the specific commit hash
  • added modules.json to the warning list for the file_exists lint check

It's still a bit of a work in progress, mostly because for some reason the tests for modules are being super slow now ...

PR checklist

  • This comment contains a description of changes (with reason)
  • CHANGELOG.md is updated
  • If you've fixed a bug or added code that should be tested, add tests!
  • Documentation in docs is updated

@KevinMenden KevinMenden added the WIP Work in progress label Jun 23, 2021
@codecov
Copy link

codecov bot commented Jun 23, 2021

Codecov Report

Merging #1125 (22de883) into dev (4c5720d) will decrease coverage by 0.73%.
The diff coverage is 91.48%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev    #1125      +/-   ##
==========================================
- Coverage   70.57%   69.83%   -0.74%     
==========================================
  Files          48       49       +1     
  Lines        4849     4883      +34     
==========================================
- Hits         3422     3410      -12     
- Misses       1427     1473      +46     
Impacted Files Coverage Δ
nf_core/lint/files_exist.py 81.81% <ø> (ø)
nf_core/modules/lint/module_tests.py 71.05% <ø> (ø)
nf_core/modules/lint/module_version.py 86.95% <86.95%> (ø)
nf_core/modules/modules_command.py 76.76% <90.00%> (-1.02%) ⬇️
nf_core/modules/lint/__init__.py 76.21% <100.00%> (+0.25%) ⬆️
nf_core/modules/lint/main_nf.py 81.37% <100.00%> (+0.12%) ⬆️
nf_core/modules/lint/meta_yml.py 62.16% <100.00%> (ø)
nf_core/modules/lint/module_changes.py 80.00% <100.00%> (+1.73%) ⬆️
nf_core/modules/nfcore_module.py 100.00% <100.00%> (ø)
nf_core/modules/module_utils.py 31.52% <0.00%> (-44.57%) ⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c5720d...22de883. Read the comment docs.

@ewels
Copy link
Member

ewels commented Jun 25, 2021

I think maybe don't prompt to create the meta if it's not there? Just fail. As this command is primarily used by automated CI I'm a bit wary of adding user interactivity.

Could potentially create it with --fix testname if we want? To be consistent with the other linting. But to be honest, we already have a commands to create the file so I think a flat fail is fine.

@ewels
Copy link
Member

ewels commented Jun 25, 2021

Sorry, for consistency with other tests I think the pattern is to ignore the test if the file isn't there. But then the files_exist test can warn / fail if it's not there.

Maybe we can have a hard fail if there are module files there and no JSON though? That's a bit of a special case.

@KevinMenden
Copy link
Contributor Author

Okay, leaving the prompt out 👍
Yes we need to somehow annoy the users so that they just go ahead and create the file (which just means running nf-core modules install).

The files_exist test only runs when using nf-core lint, not when running nf-core modules lint, so maybe we still check for the file and then just fail telling the user to create the modules.json file first.

@KevinMenden
Copy link
Contributor Author

Okay so now it just fails when no modules.json is available, and I actually think that's good because this is an essential file.

And nf-core modules lint should only be run on DSL2 pipelines, which again should have the file.

@KevinMenden KevinMenden added linting and removed WIP Work in progress labels Jun 29, 2021
Copy link
Contributor

@ErikDanielsson ErikDanielsson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Seems to work for me locally as well. However, do we want the lint to fail on missing meta.yaml files? Quite a few modules are failing due to this when running the command on the most recent version of rnaseq.

@KevinMenden
Copy link
Contributor Author

Thanks for the review @ErikDanielsson !
Yes the lint should fail in these cases. The meta.yml is actually an essential part of every module (not really used, but every module should have it). The modules that don't have meta.yml files are due to us being to lazy to actually add them 👀
So it's good that the linter reminds us to update them.

@KevinMenden
Copy link
Contributor Author

Alright I also added the load_lint_config() function to the ModulesCommand, who knows maybe it's useful to some other command in the future. Really liking this new code structure! 👍

@KevinMenden KevinMenden merged commit b155d0d into nf-core:dev Jun 30, 2021
@KevinMenden KevinMenden deleted the update-modules-lint branch June 30, 2021 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants