Skip to content

[LLM INFER] Append attn #9244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ZHUI merged 51 commits into from
Oct 23, 2024
Merged

[LLM INFER] Append attn #9244

ZHUI merged 51 commits into from
Oct 23, 2024

Conversation

@yuanlehome
Copy link
Collaborator

@yuanlehome yuanlehome commented Oct 11, 2024 •
edited
Loading

PR types

New features

PR changes

Others

Description

大模型推理attention组网重构，新的append_attn方案相比旧方案有10%到90%的性能提升。

目前已支持了llama/qwen/qwen-moe/mixtral的推理。

使用方式，原推理脚本的 --block_attn选项改为--append_attn即可。

TODO：

  • fp8推理适配
  • 性能数据补充，稍后见llm docs

yuanlehome and others added 30 commits September 14, 2024 14:05
@yuanlehome
append_attention 0914
b072465
@yuanlehome
paddle::empty to phi::allocator
b915f95
@yuanlehome
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
9b1e1d8 
…nto append_attn
@yuanlehome
append_attn 0919
140a509
@yuanlehome
0920 fix split_kv_block
5272b6f
@yuanlehome
my change for merge 4 to 1
a42157d
@yuanlehome
fix prev
bec8eef
@yuanlehome
merge zhenyun 0923
8dab056
@yuanlehome
fix prev
d5047b5
@yuanlehome
fix var name
006a467
@yuanlehome
update
73e2c06
@yuanlehome
fix config
a8acb2b
@yuanlehome
fix
ec46a89
@lizhenyun01
fix append_attn
cb02ee5 
Change-Id: Ibe8920ba41ea9775e676b05b12dc01cb9da95b5e
@yuanlehome
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
83a19a6 
…nto append_attn
@yuanlehome
fix --use_fake_parameter
37fc7da
@yuanlehome
refine paddle::empty(), fix memory error, support multi_stream for at…
a3b265b 
…tention
@yuanlehome
fix and rename attention as append_attention
68a09b6
@yuanlehome
rename file
2bcd939
@yuanlehome
fix
74941a0
@lizhenyun01
encoder GQANEOX rope support
19a0bdb
@lizhenyun01
decoder a8w8c8 GQANEOX rope support
a9078cb
@yuanlehome
merge get_block_shape and split_kv_block
f64f962
@lizhenyun01
bf16 neox rope support
7ba73f8
@lizhenyun01
fix diff
6837c23
@lizhenyun01
separate compilation
0a5ae96
@lizhenyun01
manual destroy stream
e9cfc55
@yuanlehome
fix multi stream
478c517
@yuanlehome
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
aa1e96a 
…nto append_attn
@yuanlehome
qwen/llama support weightonly
e8ddfe8
@paddle-bot
Copy link

paddle-bot bot commented Oct 11, 2024

Thanks for your contribution!

@yuanlehome yuanlehome mentioned this pull request Oct 11, 2024
Closed
@codecov
Copy link

codecov bot commented Oct 11, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 60 lines in your changes missing coverage. Please review.

Project coverage is 52.74%. Comparing base (fe8b527) to head (84a6864).
Report is 264 commits behind head on develop.

Files with missing lines Patch % Lines
...erimental/transformers/fused_transformer_layers.py 0.00% 38 Missing ⚠️
...dlenlp/experimental/transformers/qwen2/modeling.py 0.00% 8 Missing ⚠️
...dlenlp/experimental/transformers/llama/modeling.py 0.00% 7 Missing ⚠️
...enlp/experimental/transformers/mixtral/modeling.py 0.00% 5 Missing ⚠️
...lp/experimental/transformers/qwen2_moe/modeling.py 0.00% 1 Missing ⚠️
paddlenlp/experimental/transformers/utils.py 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@           Coverage Diff            @@
##           develop    #9244   +/-   ##
========================================
  Coverage    52.73%   52.74%           
========================================
  Files          661      661           
  Lines       107422   107371   -51     
========================================
- Hits         56653    56630   -23     
+ Misses       50769    50741   -28

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.