Skip to content

Disable running thread-unsafe hypothesis tests in state_test.py #27058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Disable running thread-unsafe hypothesis tests in state_test.py #27058

wants to merge 1 commit into from

Conversation

vfdev-5
Copy link
Collaborator

@vfdev-5 vfdev-5 commented Mar 11, 2025 •
edited
Loading

In this PR we skip running tests with hypothesis in free-threading.

Tests like this one

    @hp.given(set_vmap_params())
    @hp.settings(deadline=None, print_blob=True,
                 max_examples=jtu.NUM_GENERATED_CASES.value)
    def test_addupdate_vmap(self, set_vmap_param: SetVmapParams):
         pass

executed as following python3 -m pytest -vvv -s tests/state_test.py -k test_addupdate_vmap --parallel-threads=20 --iterations=15 under TSAN show the following data race due to thread-unsafe hypothesis:

tests/state_test.py::StateHypothesisTest::test_addupdate_vmap ==================
WARNING: ThreadSanitizer: data race (pid=377990)
  Write of size 8 at 0x7fffd22e0678 by thread T33:
    #0 dict_popitem_impl /project/cpython/Objects/dictobject.c:4484:23 (python3.13+0x276a7a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #1 dict_popitem /project/cpython/Objects/clinic/dictobject.c.h:220:20 (python3.13+0x276a7a)
    #2 method_vectorcall_NOARGS /project/cpython/Objects/descrobject.c:447:24 (python3.13+0x200b61) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #3 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #4 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #5 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e35db) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #6 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x21b3c2) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #7 gen_send_ex2 /project/cpython/Objects/genobject.c:229:24 (python3.13+0x21b3c2)
    #8 gen_iternext /project/cpython/Objects/genobject.c:589:9 (python3.13+0x218bd9) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #9 PyIter_Next /project/cpython/Objects/abstract.c:2919:14 (python3.13+0x1c1d7d) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #10 PySequence_Tuple /project/cpython/Objects/abstract.c:2076:26 (python3.13+0x1c1d7d)
    #11 tuple_new_impl /project/cpython/Objects/tupleobject.c:695:16 (python3.13+0x2e3034) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #12 tuple_vectorcall /project/cpython/Objects/tupleobject.c:713:16 (python3.13+0x2e13ec) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #13 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #14 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #15 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e35db) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #16 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df70a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #17 _PyEval_Vector /project/cpython/Python/ceval.c:1813:12 (python3.13+0x3df70a)
    #18 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #19 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb482) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #20 PyObject_CallOneArg /project/cpython/Objects/call.c:395:12 (python3.13+0x1eb482)
    #21 property_descr_get /project/cpython/Objects/descrobject.c:1692:12 (python3.13+0x20226f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #22 _PyObject_GenericGetAttrWithDict /project/cpython/Objects/object.c:1665:19 (python3.13+0x2962c1) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #23 PyObject_GenericGetAttr /project/cpython/Objects/object.c:1751:12 (python3.13+0x296112) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #24 PyObject_GetAttr /project/cpython/Objects/object.c:1261:18 (python3.13+0x294a77) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #25 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:3766:28 (python3.13+0x3ef070) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #26 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df70a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #27 _PyEval_Vector /project/cpython/Python/ceval.c:1813:12 (python3.13+0x3df70a)
    #28 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #29 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #30 method_vectorcall /project/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #31 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #32 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #33 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #34 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e57c2) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #35 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df70a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #36 _PyEval_Vector /project/cpython/Python/ceval.c:1813:12 (python3.13+0x3df70a)
    #37 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #38 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #39 method_vectorcall /project/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #40 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #41 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #42 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #43 thread_run /project/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x567ef2) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #44 pythread_wrapper /project/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c0e67) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)

  Previous atomic read of size 8 at 0x7fffd22e0678 by thread T28:
    #0 _Py_atomic_load_ptr_relaxed /project/cpython/./Include/cpython/pyatomic_gcc.h:359:18 (python3.13+0x260a5c) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #1 compare_generic_threadsafe /project/cpython/Objects/dictobject.c:1431:26 (python3.13+0x260a5c)
    #2 do_lookup /project/cpython/Objects/dictobject.c:1082:23 (python3.13+0x260a5c)
    #3 dictkeys_generic_lookup_threadsafe /project/cpython/Objects/dictobject.c:1460:12 (python3.13+0x260a5c)
    #4 _Py_dict_lookup_threadsafe /project/cpython/Objects/dictobject.c:1523:14 (python3.13+0x260a5c)
    #5 dict_subscript /project/cpython/Objects/dictobject.c:3311:10 (python3.13+0x275be6) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #6 PyObject_GetItem /project/cpython/Objects/abstract.c:158:26 (python3.13+0x1b8e3c) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #7 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:446:23 (python3.13+0x3e1cdf) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #8 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df70a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #9 _PyEval_Vector /project/cpython/Python/ceval.c:1813:12 (python3.13+0x3df70a)
    #10 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #11 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x2f5197) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #12 vectorcall_unbound /project/cpython/Objects/typeobject.c:2569:12 (python3.13+0x2f5197)
    #13 vectorcall_method /project/cpython/Objects/typeobject.c:2600:24 (python3.13+0x2f5197)
    #14 slot_mp_subscript /project/cpython/Objects/typeobject.c:9329:1 (python3.13+0x300255) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #15 PyObject_GetItem /project/cpython/Objects/abstract.c:158:26 (python3.13+0x1b8e3c) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #16 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:446:23 (python3.13+0x3e1cdf) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #17 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x21b3c2) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #18 gen_send_ex2 /project/cpython/Objects/genobject.c:229:24 (python3.13+0x21b3c2)
    #19 gen_iternext /project/cpython/Objects/genobject.c:589:9 (python3.13+0x218bd9) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #20 PyIter_Next /project/cpython/Objects/abstract.c:2919:14 (python3.13+0x1c1d7d) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #21 PySequence_Tuple /project/cpython/Objects/abstract.c:2076:26 (python3.13+0x1c1d7d)
    #22 tuple_new_impl /project/cpython/Objects/tupleobject.c:695:16 (python3.13+0x2e3034) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #23 tuple_vectorcall /project/cpython/Objects/tupleobject.c:713:16 (python3.13+0x2e13ec) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #24 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #25 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #26 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e35db) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #27 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df70a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #28 _PyEval_Vector /project/cpython/Python/ceval.c:1813:12 (python3.13+0x3df70a)
    #29 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #30 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb482) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #31 PyObject_CallOneArg /project/cpython/Objects/call.c:395:12 (python3.13+0x1eb482)
    #32 property_descr_get /project/cpython/Objects/descrobject.c:1692:12 (python3.13+0x20226f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #33 _PyObject_GenericGetAttrWithDict /project/cpython/Objects/object.c:1665:19 (python3.13+0x2962c1) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #34 PyObject_GenericGetAttr /project/cpython/Objects/object.c:1751:12 (python3.13+0x296112) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #35 PyObject_GetAttr /project/cpython/Objects/object.c:1261:18 (python3.13+0x294a77) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #36 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:3766:28 (python3.13+0x3ef070) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #37 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df70a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #38 _PyEval_Vector /project/cpython/Python/ceval.c:1813:12 (python3.13+0x3df70a)
    #39 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #40 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #41 method_vectorcall /project/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #42 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #43 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #44 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #45 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e57c2) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #46 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df70a) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #47 _PyEval_Vector /project/cpython/Python/ceval.c:1813:12 (python3.13+0x3df70a)
    #48 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #49 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #50 method_vectorcall /project/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #51 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #52 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #53 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #54 thread_run /project/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x567ef2) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)
    #55 pythread_wrapper /project/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c0e67) (BuildId: 3ae84a424a863e898f8ae5899a1f37386e6a2faa)

Running other tests (without hypothesis), TSAN does not report any other races.

cc @hawkinsp

hawkinsp
hawkinsp reviewed Mar 11, 2025
View reviewed changes
Copy link
Collaborator

@hawkinsp hawkinsp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That race is a CPython race, right and not particularly associated with hypothesis, correct?

Shouldn't we reproduce it in a minimal repro and report it upstream?

@vfdev-5
Copy link
Collaborator Author

vfdev-5 commented Mar 11, 2025 •
edited
Loading

That race is a CPython race, right and not particularly associated with hypothesis, correct?

The race is in CPython and it is hypothesis is responsible for calling somewhere dict.popitem() and not JAX. If I run the empty test with hypothesis decorators, there is a data race:

    @hp.given(set_vmap_params())
    @hp.settings(deadline=None, print_blob=True,
                 max_examples=jtu.NUM_GENERATED_CASES.value)
    def test_addupdate_vmap(self, set_vmap_param: SetVmapParams):
         pass

Shouldn't we reproduce it in a minimal repro and report it upstream?

yes, this can be my next step => python/cpython#131113

@vfdev-5
Copy link
Collaborator Author

vfdev-5 commented Mar 12, 2025

Another race seen in tests/state_test.py::RunStateHypothesisTest::test_jvp: https://gist.github.com/vfdev-5/e80f9b4528993eb1543e45a9216350e3#file-repro-log

Reported to cpython: python/cpython#131141

hawkinsp
hawkinsp requested changes Apr 7, 2025
View reviewed changes
Copy link
Collaborator

@hawkinsp hawkinsp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a comment linking the bugs that were responsible (and not fixed in 3.13, I guess?) and skip this on 3.13 only?

@vfdev-5
Copy link
Collaborator Author

vfdev-5 commented Apr 8, 2025

Closing this one in favor of #27804

@vfdev-5 vfdev-5 closed this Apr 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hawkinsp hawkinsp hawkinsp requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vfdev-5 @hawkinsp