How to start training again from last checkpoint #4488
Replies: 8 comments 10 replies
-
|
If you're using exp manager, it has two flags for resuming training v Without it, you can try loading your checkpoint using load_from_checkpoinr() and then calling trainer.train() again but we haven't tested that |
Beta Was this translation helpful? Give feedback.
-
|
What is involved to restart training using speech_to_text_finetune.py for model type EncDecHybridRNNTCTCBPEModel? The script states: "Currently for simplicity of single script for all model types, we only support |
Beta Was this translation helpful? Give feedback.
-
|
Are you talking about this one? create_checkpoint_callback. It is set to True @titu1994 |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Note that you will need to fix eithr |
Beta Was this translation helpful? Give feedback.
-
|
We also some documentation here: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/core/exp_manager.html |
Beta Was this translation helpful? Give feedback.
-
|
You could continue training from the latest checkpoint following the steps:
|
Beta Was this translation helpful? Give feedback.
-
|
This worked!!!! |
Beta Was this translation helpful? Give feedback.
-
|
if training is interrupted within an epoch, then no new checkpoint |
Beta Was this translation helpful? Give feedback.
-
|
Training on a-last.ckpt starts training on already seen data samples. How do I avoid it? |
Beta Was this translation helpful? Give feedback.
-
|
it did not happen to me. Maybe change another random seed? @titu1994 are you aware of this observation? |
Beta Was this translation helpful? Give feedback.
-
|
If you have not turned on shuffle, or have not used shuffle_n in train_ds, then data loader will not shuffle the data (if using tarred dataset). |
Beta Was this translation helpful? Give feedback.
-
|
@FatimaArshad-DS Do you have any other questions about this? |
Beta Was this translation helpful? Give feedback.
-
|
Its possible save checkpoint from Moe models? i got some issues trying to save one the only way i se now its saving weights but doesnt work for resume training |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
My training got interrupted and Im trying to restart training from last checkpoint. However, training starts from the beginning. How do I make it start from last checkpoint?
Beta Was this translation helpful? Give feedback.
All reactions