Nothing Special   »   [go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve numerical stability of LayerNorm #59987

Closed
wants to merge 1 commit into from
Closed

Improve numerical stability of LayerNorm #59987

wants to merge 1 commit into from

Conversation

xiaomengy
Copy link
Contributor
@xiaomengy xiaomengy commented Jun 14, 2021 •
edited
Loading

Summary: Similar as GroupNorm, improve numerical stability of LayerNorm by Welford algorithm and pairwise sum.

Test Plan: buck test mode/dev-nosan //caffe2/test:nn -- "LayerNorm"

Differential Revision: D29115235

Similar as #54921.

For input = torch.rand(4, 1024, 1024, 72, dtype=torch.float32) and normalized_shape = [1024, 1024, 72]
Previously the absolute error on CPU is over 0.03 and after this PR it will be less than 1e-6
The single-thread forward time change on CPU is 1209.96ms -> 1274.34ms
The forward time change on CUDA is 184.61ms -> 208.27ms.

@facebook-github-bot
Copy link
Contributor
facebook-github-bot commented Jun 14, 2021 •
edited
Loading

💊 CI failures summary and remediations

As of commit c635ad4 (more details on the Dr. CI page and at hud.pytorch.org/pr/59987):

  • 2/2 failures possibly* introduced in this PR
    • 1/2 non-scanned failure(s)

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_linux_xenial_py3_clang5_asan_test2 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jun 25 05:51:55 RuntimeError: test_unary_ufuncs failed! 
Jun 25 05:51:55     #172 0x55a9faeec196 in main /home/builder/ktietz/cos6/ci_cos6/python_1622833237666/work/Programs/python.c:69
Jun 25 05:51:55     #173 0x7f4ea3dc883f in __libc_start_main /build/glibc-S7Ft5T/glibc-2.23/csu/../csu/libc-start.c:291
Jun 25 05:51:55     #174 0x55a9faf7c33d in _start (/opt/conda/bin/python3.6+0x1a733d)
Jun 25 05:51:55 
Jun 25 05:51:55 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/native/Math.h:217:17 in 
Jun 25 05:51:55 Traceback (most recent call last):
Jun 25 05:51:55   File "test/run_test.py", line 1310, in <module>
Jun 25 05:51:55     main()
Jun 25 05:51:55   File "test/run_test.py", line 1289, in main
Jun 25 05:51:55     raise RuntimeError(err_message)
Jun 25 05:51:55 RuntimeError: test_unary_ufuncs failed!
Jun 25 05:51:56 + cleanup
Jun 25 05:51:56 + retcode=1
Jun 25 05:51:56 + set +x
Jun 25 05:51:56 =================== sccache compilation log ===================
Jun 25 05:51:56 =========== If your build fails, please take a look at the log above for possible reasons ===========
Jun 25 05:51:56 Compile requests                      2
Jun 25 05:51:56 Compile requests executed             0
Jun 25 05:51:56 Cache hits                            0
Jun 25 05:51:56 Cache misses                          0
Jun 25 05:51:56 Cache timeouts                        0
This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

1 similar comment
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@xiaomengy xiaomengy requested a review from ngimel June 14, 2021 23:17
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

ngimel
ngimel approved these changes Jun 16, 2021
View reviewed changes
Copy link
Collaborator
@ngimel ngimel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

1 similar comment
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@xiaomengy @facebook-github-bot
Improve numerical stability of LayerNorm (pytorch#59987)
c635ad4 
Summary:
Pull Request resolved: pytorch#59987

Similar as GroupNorm, improve numerical stability of LayerNorm by Welford algorithm and pairwise sum.

Test Plan: buck test mode/dev-nosan //caffe2/test:nn -- "LayerNorm"

Reviewed By: ngimel

Differential Revision: D29115235

fbshipit-source-id: 376dac89a4e14bd340aaaf169fef8d0d4ca4a1c4
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D29115235

@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 963c983.

This was referenced Jun 26, 2021
Closed
Closed
Closed
Closed
Closed
asuhan pushed a commit to asuhan/pytorch that referenced this pull request Jun 28, 2021
@xiaomengy @asuhan
Improve numerical stability of LayerNorm (pytorch#59987)
f6a1866 
Summary:
Pull Request resolved: pytorch#59987

Similar as GroupNorm, improve numerical stability of LayerNorm by Welford algorithm and pairwise sum.

Test Plan: buck test mode/dev-nosan //caffe2/test:nn -- "LayerNorm"

Reviewed By: ngimel

Differential Revision: D29115235

fbshipit-source-id: 5183346c3c535f809ec7d98b8bdf6d8914bfe790
@ysiraichi ysiraichi mentioned this pull request Jun 30, 2021
Closed
asuhan pushed a commit that referenced this pull request Jun 30, 2021
@xiaomengy @asuhan
Improve numerical stability of LayerNorm (#59987)
9936433 
Summary:
Pull Request resolved: #59987

Similar as GroupNorm, improve numerical stability of LayerNorm by Welford algorithm and pairwise sum.

Test Plan: buck test mode/dev-nosan //caffe2/test:nn -- "LayerNorm"

Reviewed By: ngimel

Differential Revision: D29115235

fbshipit-source-id: 5183346c3c535f809ec7d98b8bdf6d8914bfe790
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ngimel ngimel ngimel approved these changes

Assignees
No one assigned
Labels
cla signed fb-exported Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xiaomengy @facebook-github-bot @ngimel