Skip to content

[LLM INFER] Append attn #9244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ZHUI merged 51 commits into from
Oct 23, 2024
Merged

[LLM INFER] Append attn #9244

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
b072465
append_attention 0914
yuanlehome Sep 14, 2024
b915f95
paddle::empty to phi::allocator
yuanlehome Sep 14, 2024
9b1e1d8
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
yuanlehome Sep 19, 2024
140a509
append_attn 0919
yuanlehome Sep 20, 2024
5272b6f
0920 fix split_kv_block
yuanlehome Sep 20, 2024
a42157d
my change for merge 4 to 1
yuanlehome Sep 23, 2024
bec8eef
fix prev
yuanlehome Sep 23, 2024
8dab056
merge zhenyun 0923
yuanlehome Sep 23, 2024
d5047b5
fix prev
yuanlehome Sep 23, 2024
006a467
fix var name
yuanlehome Sep 23, 2024
73e2c06
update
yuanlehome Sep 23, 2024
a8acb2b
fix config
yuanlehome Sep 24, 2024
ec46a89
fix
yuanlehome Sep 24, 2024
cb02ee5
fix append_attn
lizhenyun01 Sep 27, 2024
83a19a6
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
yuanlehome Sep 27, 2024
37fc7da
fix --use_fake_parameter
yuanlehome Sep 27, 2024
a3b265b
refine paddle::empty(), fix memory error, support multi_stream for at…
yuanlehome Sep 29, 2024
68a09b6
fix and rename attention as append_attention
yuanlehome Sep 29, 2024
2bcd939
rename file
yuanlehome Sep 29, 2024
74941a0
fix
yuanlehome Sep 29, 2024
19a0bdb
encoder GQANEOX rope support
lizhenyun01 Oct 8, 2024
a9078cb
decoder a8w8c8 GQANEOX rope support
lizhenyun01 Oct 8, 2024
f64f962
merge get_block_shape and split_kv_block
yuanlehome Oct 8, 2024
7ba73f8
bf16 neox rope support
lizhenyun01 Oct 9, 2024
6837c23
fix diff
lizhenyun01 Oct 9, 2024
0a5ae96
separate compilation
lizhenyun01 Oct 9, 2024
e9cfc55
manual destroy stream
lizhenyun01 Oct 9, 2024
478c517
fix multi stream
yuanlehome Oct 10, 2024
aa1e96a
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
yuanlehome Oct 10, 2024
e8ddfe8
qwen/llama support weightonly
yuanlehome Oct 10, 2024
8798938
fix multi stream
yuanlehome Oct 10, 2024
f6a64d0
qwen-moe and mixtral support append_attn
yuanlehome Oct 10, 2024
2292780
refine code
yuanlehome Oct 11, 2024
036fb73
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
yuanlehome Oct 11, 2024
b85782d
decoder neox_rope_c4 support
lizhenyun01 Oct 11, 2024
9814578
instantiation of append_attn with float16
lizhenyun01 Oct 11, 2024
7a1f591
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
yuanlehome Oct 12, 2024
5c126ad
optimize cpu performance
yuanlehome Oct 12, 2024
2ef7c11
format code
yuanlehome Oct 12, 2024
4a4a4b4
c16/c8/c4 分离编译 加快编译速度
yuanlehome Oct 15, 2024
0e35a1e
fix bug
yuanlehome Oct 15, 2024
c5b4633
gqa_group_size -> kv_num_heads
yuanlehome Oct 15, 2024
ea8c07e
support speculate_attn
lizhenyun01 Oct 15, 2024
3789175
adjust network
yuanlehome Oct 16, 2024
6eacbca
cache_int4 -> cache_int4_zp
yuanlehome Oct 16, 2024
358115d
fix use_fake_parameter multi cards
yuanlehome Oct 17, 2024
30ac44c
fix speculate_decoder
lizhenyun01 Oct 17, 2024
4011d89
delete comment
lizhenyun01 Oct 17, 2024
7efff99
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
yuanlehome Oct 21, 2024
c30c112
Merge branch 'append_attn' of https://github.com/yuanlehome/PaddleNLP…
yuanlehome Oct 21, 2024
84a6864
fix ci
yuanlehome Oct 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i… 
…nto append_attn
  • Loading branch information
@yuanlehome
yuanlehome committed Sep 27, 2024
commit 83a19a6465bfb774ff3c3c57febc5f53b2281462
1 change: 1 addition & 0 deletions csrc/setup_cuda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ def get_gencode_flags():
"./gpu/append_attn/append_attention_kernel.cu",
"./gpu/append_attn/encoder_write_cache_with_rope_kernel.cu",
"./gpu/append_attn/decoder_write_cache_with_rope_kernel.cu",
"./gpu/sample_kernels/top_p_sampling_reject.cu",
]

cutlass_dir = "third_party/cutlass"
Expand Down
26 changes: 10 additions & 16 deletions paddlenlp/experimental/transformers/fused_transformer_layers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2050,21 +2050,18 @@ def compute_mmha(self, qkv_out, caches, attn_mask, seq_lens, rotary_embs, rotary
)[0]

def compute_out_linear(self, fmha_out, i):
if paddle.is_compiled_with_rocm():
out_linear_out = paddle.matmul(fmha_out, self.linear_weights[i])
out_linear_out = dequant_int8(out_linear_out, self.linear_out_scales[i], self._dtype)
else:
try:
from paddlenlp_ops import gemm_dequant
out_linear_out = gemm_dequant(fmha_out, self.linear_weights[i], self.linear_out_scales[i], self._dtype)
except:
if self.skip_quant("out_linear_weight_scale", i):
if paddle.is_compiled_with_rocm():
out_linear_out = paddle.matmul(fmha_out, self.linear_weights[i])
else:
out_linear_out = paddle.matmul(fmha_out, self.linear_weights[i], False, True)
else:
if paddle.is_compiled_with_rocm():
out_linear_out = paddle.matmul(fmha_out, self.linear_weights[i])
out_linear_out = dequant_int8(out_linear_out, self.linear_out_scales[i], self._dtype)
else:
try:
from paddlenlp_ops import gemm_dequant
out_linear_out = gemm_dequant(
fmha_out, self.linear_weights[i], self.linear_out_scales[i], self._dtype
)
Expand Down Expand Up @@ -2113,21 +2110,18 @@ def compute_ffn1(self, tmp_out, i):
return paddle.matmul(tmp_out, self.ffn1_weights[i], False, True)

def compute_ffn2(self, ffn1_out, i):
if paddle.device.is_compiled_with_rocm():
ffn2_out = paddle.matmul(ffn1_out, self.ffn2_weights[i])
ffn2_out = dequant_int8(ffn2_out, self.ffn2_out_scales[i], self._dtype)
else:
try:
from paddlenlp_ops import gemm_dequant
ffn2_out = gemm_dequant(ffn1_out, self.ffn2_weights[i], self.ffn2_out_scales[i], self._dtype)
except:
if self.skip_quant("ffn2_weight_scale", i):
if paddle.device.is_compiled_with_rocm():
ffn2_out = paddle.matmul(ffn1_out, self.ffn2_weights[i])
else:
ffn2_out = paddle.matmul(ffn1_out, self.ffn2_weights[i], False, True)
else:
if paddle.device.is_compiled_with_rocm():
ffn2_out = paddle.matmul(ffn1_out, self.ffn2_weights[i])
ffn2_out = dequant_int8(ffn2_out, self.ffn2_out_scales[i], self._dtype)
else:
try:
from paddlenlp_ops import gemm_dequant
ffn2_out = gemm_dequant(ffn1_out, self.ffn2_weights[i], self.ffn2_out_scales[i], self._dtype)
except:
ffn2_out = paddle.matmul(ffn1_out, self.ffn2_weights[i], False, True)
Expand Down
27 changes: 8 additions & 19 deletions paddlenlp/experimental/transformers/llama/modeling.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1237,7 +1237,10 @@ def set_state_dict(self, state_dict):

for k, v in cache_scales_loader.scale.items():
for i_layer, weight_scale in enumerate(v):
weight_scale = weight_scale.astype("float32")
if self.config.append_attn:
weight_scale = paddle.to_tensor(weight_scale).cast(paddle.get_default_dtype())
else:
weight_scale = weight_scale.astype("float32")
if k == "cache_k_scale":
self.transformer_block.cache_k_scales[i_layer].set_value(weight_scale)
elif k == "cache_v_scale":
Expand All @@ -1247,24 +1250,10 @@ def set_state_dict(self, state_dict):
else:
self.transformer_block.cache_v_out_scales[i_layer].set_value(weight_scale)

for k, v in cache_scales_loader.scale.items():
for i_layer, weight_scale in enumerate(v):
if self.config.append_attn:
weight_scale = paddle.to_tensor(weight_scale).cast(paddle.get_default_dtype())
else:
weight_scale = weight_scale.astype("float32")
if k == "cache_k_scale":
self.transformer_block.cache_k_scales[i_layer].set_value(weight_scale)
elif k == "cache_v_scale":
self.transformer_block.cache_v_scales[i_layer].set_value(weight_scale)
elif k == "cache_k_out_scale":
self.transformer_block.cache_k_out_scales[i_layer].set_value(weight_scale)
else:
self.transformer_block.cache_v_out_scales[i_layer].set_value(weight_scale)

for k, v in weight_scales_loader.scale.items():
if "qkv_" in k:
for i_layer, weight_scale in enumerate(v):
for k, v in weight_scales_loader.scale.items():
if "qkv_" in k:
for i_layer, weight_scale in enumerate(v):
if not np.all(weight_scale == -1):
tmp = paddle.to_tensor(
weight_scale
/ (
Expand Down
27 changes: 8 additions & 19 deletions paddlenlp/experimental/transformers/qwen2/modeling.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -829,7 +829,10 @@ def set_state_dict(self, state_dict):

for k, v in cache_scales_loader.scale.items():
for i_layer, weight_scale in enumerate(v):
weight_scale = weight_scale.astype("float32")
if self.config.append_attn:
weight_scale = paddle.to_tensor(weight_scale).cast(paddle.get_default_dtype())
Copy link
Contributor

@ZHUI ZHUI Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为什么 append_attn 下 可以不同 fp32？

Copy link
Collaborator Author

@yuanlehome yuanlehome Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

因为kernel实现里是要求half精度的，访存 不同

else:
weight_scale = weight_scale.astype("float32")
if k == "cache_k_scale":
self.transformer_block.cache_k_scales[i_layer].set_value(weight_scale)
elif k == "cache_v_scale":
Expand All @@ -839,24 +842,10 @@ def set_state_dict(self, state_dict):
else:
self.transformer_block.cache_v_out_scales[i_layer].set_value(weight_scale)

for k, v in cache_scales_loader.scale.items():
for i_layer, weight_scale in enumerate(v):
if self.config.append_attn:
weight_scale = paddle.to_tensor(weight_scale).cast(paddle.get_default_dtype())
else:
weight_scale = weight_scale.astype("float32")
if k == "cache_k_scale":
self.transformer_block.cache_k_scales[i_layer].set_value(weight_scale)
elif k == "cache_v_scale":
self.transformer_block.cache_v_scales[i_layer].set_value(weight_scale)
elif k == "cache_k_out_scale":
self.transformer_block.cache_k_out_scales[i_layer].set_value(weight_scale)
else:
self.transformer_block.cache_v_out_scales[i_layer].set_value(weight_scale)

for k, v in weight_scales_loader.scale.items():
if "qkv_" in k:
for i_layer, weight_scale in enumerate(v):
for k, v in weight_scales_loader.scale.items():
if "qkv_" in k:
for i_layer, weight_scale in enumerate(v):
if not np.all(weight_scale == -1):
tmp = paddle.to_tensor(
weight_scale
/ (
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.