Skip to content

MPS test failures on MacOS Tahoe #164712

@malfet

Description

@malfet

🐛 Describe the bug

Running on M4 Pro:

======================================================================
ERROR: test_output_match_signal_windows_kaiser_mps_float32 (__main__.TestConsistencyMPS.test_output_match_signal_windows_kaiser_mps_float32)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_device_type.py", line 1144, in test_wrapper
    return test(*args, **kwargs)
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 12434, in test_output_match
    self.assertEqual(cpu_out, mps_out, atol=atol, rtol=rtol)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 4168, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
    ...<4 lines>...
    )
AssertionError: Tensor-likes are not close!

Mismatched elements: 2 / 2 (100.0%)
Greatest absolute difference: nan at index (0,) (up to 1e-05 allowed)
Greatest relative difference: nan at index (0,) (up to 1.3e-06 allowed)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 3213, in wrapper
    method(*args, **kwargs)
    ~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_device_type.py", line 426, in instantiated_test
    result = test(self, **param_kwargs)
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 1644, in wrapper
    fn(*args, **kwargs)
    ~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_device_type.py", line 1156, in test_wrapper
    raise e_tracked from e
Exception: Tensor-likes are not close!

Mismatched elements: 2 / 2 (100.0%)
Greatest absolute difference: nan at index (0,) (up to 1e-05 allowed)
Greatest relative difference: nan at index (0,) (up to 1.3e-06 allowed)

Caused by sample input at index 4: SampleInput(input=2, args=(), kwargs={'sym': 'True', 'device': "'mps:0'", 'dtype': 'torch.float32', 'requires_grad': 'True', 'beta': '12.0'}, broadcasts_input=False, name='')

To execute this test, run the following from the base repo dir:
    PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=4 python test/test_mps.py TestConsistencyMPS.test_output_match_signal_windows_kaiser_mps_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_exp1 (__main__.TestMPS.test_exp1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 3213, in wrapper
    method(*args, **kwargs)
    ~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 809, in test_exp1
    self.assertEqual(output, output_cpu, atol=1e-8, rtol=1e-8)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 4168, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
    ...<4 lines>...
    )
AssertionError: Tensor-likes are not close!

Mismatched elements: 2 / 4 (50.0%)
Greatest absolute difference: 2.384185791015625e-07 at index (1,) (up to 1e-08 allowed)
Greatest relative difference: 8.770929582624376e-08 at index (1,) (up to 1e-08 allowed)

To execute this test, run the following from the base repo dir:
    python test/test_mps.py TestMPS.test_exp1

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_inverse (__main__.TestMPS.test_inverse)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 3213, in wrapper
    method(*args, **kwargs)
    ~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 7290, in test_inverse
    helper(1025, atol=1e-4)
    ~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 7284, in helper
    self.assertEqual(cpu_result, mps_result, atol=atol, rtol=rtol)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 4168, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
    ...<4 lines>...
    )
AssertionError: Tensor-likes are not close!

Mismatched elements: 51 / 1050625 (0.0%)
Greatest absolute difference: 0.00012379884719848633 at index (959, 943) (up to 0.0001 allowed)
Greatest relative difference: 0.00029734105919487774 at index (563, 132) (up to 1e-06 allowed)

To execute this test, run the following from the base repo dir:
    python test/test_mps.py TestMPS.test_inverse

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_linalg_cholesky (__main__.TestMPS.test_linalg_cholesky)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 3213, in wrapper
    method(*args, **kwargs)
    ~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 5789, in test_linalg_cholesky
    run_cholesky_test(size, batch_size, upper=upper)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 5779, in run_cholesky_test
    self.assertEqual(output_cpu, output_mps, atol=2e-5, rtol=1e-6)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 4168, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
    ...<4 lines>...
    )
AssertionError: Tensor-likes are not close!

Mismatched elements: 480 / 16384 (2.9%)
Greatest absolute difference: 1552.9036865234375 at index (0, 95, 81) (up to 2e-05 allowed)
Greatest relative difference: inf at index (0, 1, 0) (up to 1e-06 allowed)

The failure occurred for item [0]

To execute this test, run the following from the base repo dir:
    python test/test_mps.py TestMPS.test_linalg_cholesky

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_log1p_float32 (__main__.TestMPS.test_log1p_float32)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 3213, in wrapper
    method(*args, **kwargs)
    ~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 552, in instantiated_test
    test(self, **param_kwargs)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 6076, in test_log1p
    self.assertEqual(log_result, log_result_cpu, atol=0, rtol=2e-7)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 4168, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
    ...<4 lines>...
    )
AssertionError: Tensor-likes are not close!

Mismatched elements: 88 / 30000 (0.3%)
Greatest absolute difference: 2.384185791015625e-07 at index (27668,) (up to 0 allowed)
Greatest relative difference: 2.571402148987545e-07 at index (20045,) (up to 2e-07 allowed)

To execute this test, run the following from the base repo dir:
    python test/test_mps.py TestMPS.test_log1p_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_mps_allocator_module (__main__.TestMPS.test_mps_allocator_module)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 3213, in wrapper
    method(*args, **kwargs)
    ~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/test/test_mps.py", line 7705, in test_mps_allocator_module
    self.assertEqual(current_alloc_before, 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malfet/git/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 4168, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
    ...<4 lines>...
    )
AssertionError: Scalars are not equal!

Expected 0 but got 7680.
Absolute difference: 7680
Relative difference: inf

To execute this test, run the following from the base repo dir:
    python test/test_mps.py TestMPS.test_mps_allocator_module

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
Ran 7159 tests in 837.441s

FAILED (failures=5, errors=1, skipped=246, expected failures=835)

Versions

ci

cc @mruberry @kulinseth @DenisVieriu97 @jhavukainen

Metadata

Metadata

Assignees

No one assigned

    Labels

    module: mpsRelated to Apple Metal Performance Shaders frameworkmodule: testsIssues related to tests (not the torch.testing module)triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions