Add Gradient Cache&Recompute into Neural Search #3697
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wawltor
reviewed
Nov 10, 2022
| * `corpus_file`: 召回库数据 corpus_file | ||
| * `use_recompute`: 使用Recompute策略，用于节省显存，是一种以时间换空间的技术 | ||
| * `use_gradient_cache`: 使用Gradient Cache策略，用于节省显存，是一种以时间换空间的技术 | ||
| * `chunk_numbers`: 使用Gradient Cache策略的参数，表示的是同一个批次的样本分几次执行 |
There was a problem hiding this comment.
建议recompute和gradient_cache的这种分布式能力建议收口到Trainer里面，后续@ZHUI来加入
| title_cls_embedding, | ||
| transpose_y=True) | ||
|
|
||
| # substract margin from all positive samples cosine_sim() |
| # substract margin from all positive samples cosine_sim() | ||
| margin_diag = paddle.full(shape=[query_cls_embedding.shape[0]], | ||
| fill_value=self.margin, | ||
| dtype=paddle.get_default_dtype()) |
There was a problem hiding this comment.
这里的逻辑比较奇怪，为什么不拿一个变量来赋值，可以直接拿cosine_sim dtype
There was a problem hiding this comment.
我试了一下，效果是一样的，已经改成cosine_sim dtype
| all_grads): | ||
|
|
||
| sub_query_input_ids, sub_query_token_type_ids, sub_title_input_ids, sub_title_token_type_ids = sub_batch | ||
| paddle.framework.random.set_cuda_rng_state(CUDA_state) |
There was a problem hiding this comment.
这里还原cuda的随机种子，是为了还原dropout？
There was a problem hiding this comment.
由于Gradient Cache需要计算两次前向，设置随机种子是为了使dropout等随机种子一致，确保模型的中间状态一样，使得2次前向得到的结果一致。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
PR changes
Description