Skip to content

Refactor training loop shardv2 dev #7225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 79 commits into from
Closed

Refactor training loop shardv2 dev #7225

wants to merge 79 commits into from

Conversation

liuzhenhai93
Copy link
Contributor

PR types

Others

PR changes

Others

Description

  • sharding reshard supoort v1 -> v2, v2->v1, v2->v2, v1->v1

Meiyim and others added 30 commits June 16, 2023 08:23
@Meiyim
refactor training loop support
6b690ec
@ForFishes
fix trainer
78d764d
@Meiyim
Merge pull request PaddlePaddle#1 from ForFishes/add_seed_order_nlp
aa738a6 
[Distributed]Fix trainer for distributed training
@haohongxiang
support pp delay_scale_loss and dp_comm_overlap
9cfd157
@Meiyim
Merge pull request PaddlePaddle#3 from haohongxiang/support_pp_new_st…
28cb000 
…rategy

[Dygraph] Support PP new strategy - delay_scale_loss & dp_comm_overlap
@wuhuachaocoding
support sharding stage1 in hybrid parallel.
3bec988 
update.
@Meiyim
pipeline 兼容非master-grad
dfbd574
@Meiyim
Merge pull request PaddlePaddle#5 from Meiyim/fix-master-grad
c382dc0 
pipeline 兼容非master-grad
@Meiyim
Merge pull request PaddlePaddle#4 from wuhuachaocoding/sharding_eb4
499413b 
support sharding stage1 in hybrid parallel.
@Meiyim
Bump paddle to new-model-7
fc8f1ab
@Meiyim
fix sharding + overlap hangs under progressive batching
49c9e5c
@GhostScreaming
Add time statistics for nccl-connection.
40f3a91
@GhostScreaming
Polish code.
dc59752
@Meiyim
Merge pull request PaddlePaddle#7 from GhostScreaming/refactor-traini…
9539fdb 
…ng-loop

Add time statistics for nccl-connection.
@Meiyim
allow to use main-grad under TF32/FP32
0f29ea4
@Meiyim
Merge pull request PaddlePaddle#8 from Meiyim/tf32-main-grad
9a71dd7 
allow to use `main-grad` under TF32/FP32
@Meiyim
[hot-fix] resume from accumulation-step wrong
3124e91
@xysheng-baidu
[fix] pp非mp情况下只存了pp01 model
afebd95
@Meiyim
Merge pull request PaddlePaddle#9 from Meiyim/hot-fix-resume-when-acc…
1985aeb 
…umulate

[hot-fix] resume from accumulation-step wrong
@Meiyim
Merge pull request PaddlePaddle#10 from bo-ke/fix_args
55ee7d6 
[fix] pp非mp情况下只存了pp01 model
@Meiyim
online hot fix, rasie Error when optimizer/lr scheduler no show in re…
16e685e 
…sume, warning when model-weight has missing keys
@Meiyim
Merge pull request PaddlePaddle#11 from Meiyim/hotfix
56c001a 
online hot fix, rasie Error when optimizer/lr scheduler no show in re…
sharding save and load for bf16
7798f93
fix
cf5e5bb
polish
854036c
polish
2024390
add reshard
e182e0b
polish
d5e4512
bf16 save and load demo
519ab65
polish
2366c32
liuzhenhai93 and others added 28 commits July 14, 2023 19:30
@liuzhenhai93
broadcast (PaddlePaddle#6397)
dce1593 
* broadcast

* polish

* polish

* polish

* polish

* polish
@GuoxiaWang
fix ClipGradByAdaptiveNorm load state dict (PaddlePaddle#6409)
6c1d9bc 
* fix ClipGradByAdaptiveNorm load state dict

* fix code style
@sneaxiy
remove too many parameter logs (PaddlePaddle#6407)
fa76862
@Meiyim
Add profile timer (PaddlePaddle#6441)
5517ca5 
* add profile timer
@liuzhenhai93
polish (PaddlePaddle#6492)
02ae3fc
@ForFishes
[Distributed]Add dp/sharding overlap for pipeline (PaddlePaddle#6504)
47a71a1
@ForFishes
[Distributed]Support pipelineparallel in accumulation_steps (PaddlePa…
083f8d8 
…ddle#6509)
@SylarTiaNII
[LLM] Fix asynchronize save (PaddlePaddle#6624)
28d4e0c
@ZHUI
[LLM] Support master grad on dp. (PaddlePaddle#6650)
ed8ca95 
* support master grad on dp.
@ForFishes
[Distributed] Add hang mode to avoid overlap (PaddlePaddle#6838)
583bb33 
* add debug mode
@ZHUI
Add tensor parallel conversion. (PaddlePaddle#6844)
0a808d3
@QimingPeng
[Trainer] Support main grad for pure sharding PaddlePaddle#7013 (Padd…
2e53d60 
…lePaddle#7014)
@pangengzheng
improve load sharding model (PaddlePaddle#6976)
0ed3828
@Meiyim
[Trainer] Support for MoE model training (PaddlePaddle#7089)
1191c40
@QimingPeng
[Trainer] Update load/save sharded model for mp model (PaddlePaddle#7180
8def3e7 
)
@xysheng-baidu
[LLM] use pin memory save optimizer (PaddlePaddle#7185)
939224a
@liuzhenhai93
basic func
cb7efe1
@liuzhenhai93
basic func
2772950
@liuzhenhai93
polish
06123fa
@liuzhenhai93
code ok
d6400c9
@liuzhenhai93
basic func
283371f
@liuzhenhai93
polish
b4f0034
@liuzhenhai93
polish
05da413
Merge branch 'refactor-training-loop' of https://github.com/PaddlePad…
d0509e4 
…dle/PaddleNLP into refactor-training-loop_shardv2
@liuzhenhai93
polish
bdf9aa9
@liuzhenhai93
paddlenlp/trainer/utils/reshard/sharding_v2.py
d6d1dcf
@liuzhenhai93
polish
6f8e808
@liuzhenhai93
polish
50950c2
@CLAassistant
Copy link

CLAassistant commented Oct 14, 2023 •
edited
Loading

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
11 out of 13 committers have signed the CLA.

✅ Meiyim
✅ haohongxiang
✅ GhostScreaming
✅ ForFishes
✅ SylarTiaNII
✅ wuhuachaocoding
✅ GuoxiaWang
✅ liuzhenhai93
✅ sneaxiy
✅ xysheng-baidu
✅ ZHUI
❌ pangengzheng
❌ QimingPeng

pangengzheng seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.