remove language_modeling #14192
Merged
remove language_modeling #14192
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
…om/NVIDIA/NeMo into dpykhtar/remove_language_modelling
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Check notice
Code scanning / CodeQL
Cyclic import Note
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Comment on lines
+29
to
+32
| from nemo.collections.nlp.data.language_modeling.megatron.gpt_sft_chat_dataset import ( | ||
| _get_header_conversation_type_mask_role, | ||
| get_prompt_template_example, | ||
| ) |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 6 months ago
To fix the problem:
- Remove the unused import
get_prompt_template_examplefrom thenemo.collections.nlp.data.language_modeling.megatron.gpt_sft_chat_datasetmodule. - Ensure that the removal does not affect the functionality of the code, as no references to
get_prompt_template_exampleexist in the file.
Detailed steps:
- Locate the import statement starting on line 29.
- Remove the specific
get_prompt_template_examplefrom the import list while keeping any other imports intact (_get_header_conversation_type_mask_role).
Suggested changeset
1
nemo/collections/nlp/modules/common/text_generation_server.py
| @@ -28,7 +28,6 @@ | ||
| try: | ||
| from nemo.collections.nlp.data.language_modeling.megatron.gpt_sft_chat_dataset import ( | ||
| _get_header_conversation_type_mask_role, | ||
| get_prompt_template_example, | ||
| ) | ||
|
|
||
| HAVE_NLP = True |
Copilot is powered by AI and may make mistakes. Always verify output.
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
chtruong814
approved these changes
Oct 10, 2025
chtruong814
requested changes
Oct 10, 2025
Collaborator
chtruong814
left a comment
There was a problem hiding this comment.
@dimapihtar this is all nemo1 code we're removing?
Collaborator
|
@dimapihtar I think that last test is failing because the path for helpers.cpp file was renamed. |
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Collaborator
Author
no, we have nemo.nlp.modules left which will be removed in the next separate PR. It's just complicated to remove everything in a single PR. |
Collaborator
Author
It was failing because I forgot to move Makefile in addition to helpers.cpp |
Contributor
|
[🤖]: Hi @dimapihtar 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully. So it might be time to merge this PR or get some approvals. |
chtruong814
approved these changes
Oct 13, 2025
yaoyu-33
approved these changes
Oct 14, 2025
dimapihtar
added a commit
that referenced
this pull request
Oct 15, 2025
* remove language_modeling Signed-off-by: dimapihtar <[email protected]> * fix imports Signed-off-by: dimapihtar <[email protected]> * fix imports Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * fix import Signed-off-by: dimapihtar <[email protected]> * fix imports Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * remove multimodal data unit tests Signed-off-by: dimapihtar <[email protected]> * remove nlp Signed-off-by: dimapihtar <[email protected]> * remove import checks for nlp Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * remove check imposrts for nlp Signed-off-by: dimapihtar <[email protected]> * remove check imports for nlp Signed-off-by: dimapihtar <[email protected]> * fix nlp imports Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * guard imports Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * move MegatronPretrainingBatchSampler Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix import Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * fix import Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix imports Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix import Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * remove multimodal test Signed-off-by: dimapihtar <[email protected]> * fix import Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * remove files Signed-off-by: dimapihtar <[email protected]> * resolve merge conflicts Signed-off-by: dimapihtar <[email protected]> * revert changes Signed-off-by: dimapihtar <[email protected]> * fix nlp data imports Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * move list_available_models to nlp.modules Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * get rid of language modelling modules Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * add Makefile Signed-off-by: dimapihtar <[email protected]> * add Makefile Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]>
chtruong814
pushed a commit
that referenced
this pull request
Oct 15, 2025
* remove language_modeling * fix imports * fix imports * Apply isort and black reformatting * fix style * fix style * fix import * fix imports * Apply isort and black reformatting * fix style * fix style * fix style * remove multimodal data unit tests * remove nlp * remove import checks for nlp * Apply isort and black reformatting * remove check imposrts for nlp * remove check imports for nlp * fix nlp imports * Apply isort and black reformatting * guard imports * Apply isort and black reformatting * fix style * move MegatronPretrainingBatchSampler * Apply isort and black reformatting * fix import * Apply isort and black reformatting * fix style * Apply isort and black reformatting * fix style * fix style * fix import * Apply isort and black reformatting * fix imports * Apply isort and black reformatting * fix import * Apply isort and black reformatting * fix style * remove multimodal test * fix import * Apply isort and black reformatting * remove files * resolve merge conflicts * revert changes * fix nlp data imports * Apply isort and black reformatting * fix style * Apply isort and black reformatting * fix style * move list_available_models to nlp.modules * Apply isort and black reformatting * get rid of language modelling modules * Apply isort and black reformatting * fix style * add Makefile * add Makefile --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Removes nlp/language_modelling.
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information