Skip to content

[LLM] Support prefix tuning and lora for qwen2 #8601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ZHUI merged 15 commits into from
Jun 20, 2024
Merged

[LLM] Support prefix tuning and lora for qwen2 #8601

ZHUI merged 15 commits into from
Jun 20, 2024

Conversation

@DrownFish19
Copy link
Collaborator

@DrownFish19 DrownFish19 commented Jun 13, 2024 •
edited
Loading

PR types

Function optimization

PR changes

Models

Description

  1. support prefix tuning and lora;
  2. fix modeling and tokenizer when tie_word_embedding=True (Qwen1.5-0.5B, Qwen2-0.5B, Qwen2-1.5B);
  3. fix pipeline and sequence parallel;
  4. add unittest;
  5. add llm unittest;
  6. upload pretrain, sft, and lora configs.

@paddle-bot
Copy link

paddle-bot bot commented Jun 13, 2024

Thanks for your contribution!

@codecov
Copy link

codecov bot commented Jun 14, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 52.32558% with 41 lines in your changes missing coverage. Please review.

Project coverage is 54.73%. Comparing base (970b868) to head (cb19d31).
Report is 237 commits behind head on develop.

Files with missing lines Patch % Lines
paddlenlp/transformers/qwen2/modeling_pp.py 34.37% 21 Missing ⚠️
paddlenlp/transformers/qwen2/modeling.py 68.42% 12 Missing ⚠️
paddlenlp/transformers/qwen2/tokenizer.py 41.66% 7 Missing ⚠️
paddlenlp/transformers/model_utils.py 50.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #8601      +/-   ##
===========================================
+ Coverage    54.18%   54.73%   +0.55%     
===========================================
  Files          625      625              
  Lines        98942    98985      +43     
===========================================
+ Hits         53612    54180     +568     
+ Misses       45330    44805     -525

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@DrownFish19 DrownFish19 changed the title [LLM] Add unittest for Qwen2 [LLM] suport prefix tuning and lora for Qwen2 Jun 17, 2024
@DrownFish19 DrownFish19 changed the title [LLM] suport prefix tuning and lora for Qwen2 [LLM] Support prefix tuning and lora for qwen2 Jun 17, 2024
ZHUI
ZHUI reviewed Jun 18, 2024
View reviewed changes
paddlenlp/transformers/model_utils.py
# However, when state dict only contains the one piece of shared parameters, the shared parameters
# will be different from the original shared parameters.

if isinstance(model, PipelineLayer):
Copy link
Contributor

@ZHUI ZHUI Jun 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

赞👍🏻

@DrownFish19 DrownFish19 mentioned this pull request Jun 20, 2024
Closed
1 task
ZHUI
ZHUI approved these changes Jun 20, 2024
View reviewed changes
Copy link
Contributor

@ZHUI ZHUI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ZHUI ZHUI merged commit da8b9ac into PaddlePaddle:develop Jun 20, 2024
@DrownFish19 DrownFish19 deleted the dev_add_tests_qwen2 branch June 20, 2024 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ZHUI ZHUI ZHUI approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@DrownFish19 @ZHUI