🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/101683

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 9ff21d5:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Hey @janEbert, thanks for the PR - is there an associated issue for this?

For my understanding, is this change related to the porting of RMSNorm from this comment?

Hey @jbschlosser, I tried querying issues and PRs matching "LayerNorm without bias" and similar, but didn't find anything. I haven't opened an issue for this but can do it if it makes administration easier.
Also if you have a tip for searching in very large open-source projects like PyTorch, I'd really appreciate it for the future!

The reason to implement this PR is T5-style models as discussed in @mikaylagawarecki's linked issue (which I didn't find in my search). The PaLM paper also mentioned more stable scaling when disabling LayerNorm bias for large models.

@albanD mentioned:

yes bias is not great as it is usually just the bias weight but here it also means the centering is also removed (you don't remove the average bias). Do you think rms_only=True would be a better name?

Should this be discussed? I personally think bias is clear enough, considering the consistency with other PyTorch layers and that centering in the statistical sense is usually "mean".

Hey, any new opinions on this? I'd be really happy to see this merged so that the PyTorch Transformers API becomes more flexible for scaling up. :)

@janEbert Apologies for the delay, so my understanding (which perhaps you were getting at) is that

LayerNorm is given by

$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$
where the $\epsilon$ is added for numerical stability

This PR is doing LayerNorm(elementwise_affine=True, bias=False) is

$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma$

and RMSNorm is given by

$y = \frac{x}{ RMS(x)} * \gamma$ (with maybe an optional $+ \beta$ per this comment)

Where

$Var(x) = \frac{1}{n}\sum\limits_{i=1}^{n} (x_i - E[x])^2$

and

$RMS(x) = \sqrt{\frac{1}{n}\sum\limits_{i=1}^{n} (x_i)^2}$ ($\epsilon$ could also be used for numerical stability in this calculation)

so RMSNorm defers from LayerNorm on 3 counts

1. don't subtract expectation from numerator
2. use RMS(x) rather than $\sqrt{Var(x) + \epsilon}$ in the denominator
3. (perhaps) don't learn an elementwise affine bias

And this change is completely separate from RMSNorm. In conclusion, we are okay with adding a bias argument here to disable the affine bias for consistency with other modules (e.g. Bilinear, Linear) :)

@pytorchbot rebase

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

Successfully rebased layernorm_nobias onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout layernorm_nobias && git pull --rebase)

Thank you so much @mikaylagawarecki, that's an amazing summary that clears up any misunderstandings!

@pytorchbot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team