Segfault in test_embedded_r.py #1111
Description
Describe the issue or bug
Our Linux distribution (NixOS) started seeing rpy2 test failures under Python 3.11.9 after upgrading to R 4.4.0. The following test case, minimized from tests/Interface/test_embedded.py:test_parse_error_when_evaluating, segfaults when R tries to run rpy2's _handler_wrap callback via cffi under API mode.
from rpy2 import rinterface
rinterface.initr()
rinterface.parse("list(''=1)")
Python doesn't emit a traceback due to the crash, but gdb reports the following stack trace which points to the _handler_wrap function at _rinterface_capi:690.
#0 take_ownership (frame=0x7ffff6dac390, f=0x7ffff6dac360) at Python/frame.c:97
#1 _PyFrame_Clear (frame=frame@entry=0x7ffff7e48320) at Python/frame.c:133
#2 0x00007ffff7b2b4f6 in _PyEvalFrameClearAndPop (frame=0x7ffff7e48320, tstate=0x7ffff7de7b18 <_PyRuntime+166328>) at Python/ceval.c:6406
#3 _PyEval_Vector (tstate=0x7ffff7de7b18 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>)
at Python/ceval.c:6439
#4 0x00007ffff663422f in general_invoke_callback (decode_args_from_libffi=0, result=0x7fffffff7c90, args=<optimized out>, userdata=0x7ffff6d6cdb0) at src/c/_cffi_backend.c:6201
#5 0x00007ffff6634451 in cffi_call_python (externpy=0x7ffff75b4c60 <_cffi_externpy___handler_wrap>, args=0x7fffffff7c90 "\370\255", <incomplete sequence \341>) at src/c/call_python.c:276
#6 0x00007ffff759cd8d in _handler_wrap (a0=<optimized out>, a1=<optimized out>) at build/temp.linux-x86_64-cpython-311/_rinterface_cffi_api.c:1351
#7 0x00007ffff69a535d in do_internal (call=<optimized out>, op=<optimized out>, args=<optimized out>, env=0x286f5b8) at names.c:1409
#8 0x00007ffff695ea00 in Rf_eval (e=e@entry=0x2868248, rho=rho@entry=0x286f5b8) at eval.c:1237
#9 0x00007ffff6960767 in R_execClosure (call=call@entry=0x9fb218, newrho=newrho@entry=0x286f5b8, sysparent=<optimized out>, rho=rho@entry=0x286c220, arglist=arglist@entry=0x286f628,
op=op@entry=0x286a2b8) at eval.c:2398
#10 0x00007ffff69615a7 in applyClosure_core (call=call@entry=0x9fb218, op=op@entry=0x286a2b8, arglist=arglist@entry=0x286f628, rho=rho@entry=0x286c220, suppliedvars=<optimized out>,
unpromise=unpromise@entry=TRUE) at eval.c:2311
#11 0x00007ffff69446e9 in Rf_applyClosure (unpromise=TRUE, suppliedvars=<optimized out>, rho=<optimized out>, arglist=0x286f628, op=0x286a2b8, call=0x9fb218) at eval.c:2333
#12 bcEval_loop (ploc=ploc@entry=0x7fffffff9050) at eval.c:8112
#13 0x00007ffff695e26a in bcEval (rho=0x286cbf8, body=0x9e8578) at eval.c:7524
#14 bcEval (body=0x9e8578, rho=0x286cbf8) at eval.c:7509
#15 0x00007ffff695e5d3 in Rf_eval (e=e@entry=0x9e8578, rho=rho@entry=0x286cbf8) at eval.c:1167
#16 0x00007ffff6960767 in R_execClosure (call=call@entry=0x286ce28, newrho=newrho@entry=0x286cbf8, sysparent=<optimized out>, rho=rho@entry=0x286a2f0, arglist=arglist@entry=0x286cd80,
op=op@entry=0x9e86c8) at eval.c:2398
#17 0x00007ffff69615a7 in applyClosure_core (call=call@entry=0x286ce28, op=op@entry=0x9e86c8, arglist=0x286cd80, rho=rho@entry=0x286a2f0, suppliedvars=<optimized out>,
unpromise=unpromise@entry=TRUE) at eval.c:2311
#18 0x00007ffff695e70e in Rf_applyClosure (unpromise=TRUE, suppliedvars=<optimized out>, rho=0x286a2f0, arglist=<optimized out>, op=0x9e86c8, call=0x286ce28) at eval.c:2333
#19 Rf_eval (e=e@entry=0x286ce28, rho=rho@entry=0x286a2f0) at eval.c:1285
#20 0x00007ffff68d05a0 in do_docall (call=0x286ce28, op=<optimized out>, args=0xe1a9b8, rho=<optimized out>) at coerce.c:2761
#21 0x00007ffff69443e9 in bcEval_loop (ploc=ploc@entry=0x7fffffffa6b0) at eval.c:8141
#22 0x00007ffff695e26a in bcEval (rho=0x286d170, body=0x176d4d0) at eval.c:7524
#23 bcEval (body=0x176d4d0, rho=0x286d170) at eval.c:7509
#24 0x00007ffff695e5d3 in Rf_eval (e=e@entry=0x176d4d0, rho=rho@entry=0x286d170) at eval.c:1167
#25 0x00007ffff6960767 in R_execClosure (call=call@entry=0x2866a00, newrho=newrho@entry=0x286d170, sysparent=<optimized out>, rho=rho@entry=0x286a2f0, arglist=arglist@entry=0x286d2f8,
op=op@entry=0x17711b0) at eval.c:2398
#26 0x00007ffff69615a7 in applyClosure_core (call=call@entry=0x2866a00, op=op@entry=0x17711b0, arglist=0x286d2f8, rho=rho@entry=0x286a2f0, suppliedvars=<optimized out>,
unpromise=unpromise@entry=TRUE) at eval.c:2311
#27 0x00007ffff695e70e in Rf_applyClosure (unpromise=TRUE, suppliedvars=<optimized out>, rho=0x286a2f0, arglist=<optimized out>, op=0x17711b0, call=0x2866a00) at eval.c:2333
#28 Rf_eval (e=0x2866a00, rho=rho@entry=0x286a2f0) at eval.c:1285
#29 0x00007ffff6962320 in do_begin (call=0x2868088, op=0x65aec0, args=0x28669c8, rho=0x286a2f0) at ../../src/include/Rinlinedfuns.h:90
#30 0x00007ffff695ea00 in Rf_eval (e=e@entry=0x2868088, rho=rho@entry=0x286a2f0) at eval.c:1237
#31 0x00007ffff6960767 in R_execClosure (call=call@entry=0x2866728, newrho=newrho@entry=0x286a2f0, sysparent=<optimized out>, rho=rho@entry=0x688cc8, arglist=arglist@entry=0x2866680,
op=op@entry=0x2866878) at eval.c:2398
#32 0x00007ffff69615a7 in applyClosure_core (call=call@entry=0x2866728, op=op@entry=0x2866878, arglist=0x2866680, rho=rho@entry=0x688cc8, suppliedvars=<optimized out>,
unpromise=unpromise@entry=TRUE) at eval.c:2311
#33 0x00007ffff695e70e in Rf_applyClosure (unpromise=TRUE, suppliedvars=<optimized out>, rho=0x688cc8, arglist=<optimized out>, op=0x2866878, call=0x2866728) at eval.c:2333
#34 Rf_eval (e=0x2866728, rho=0x688cc8) at eval.c:1285
#35 0x00007ffff6931856 in evalKeepVis (rho=<optimized out>, e=<optimized out>) at errors.c:43
#36 R_tryCatch (body=body@entry=0x7ffff759cdb0 <_parsevector_wrap>, bdata=bdata@entry=0x7ffff6db3e80, conds=conds@entry=0x284dfd8, handler=handler@entry=0x7ffff759cd60 <_handler_wrap>,
hdata=hdata@entry=0x0, finally=finally@entry=0x0, fdata=0x0) at errors.c:2476
#37 0x00007ffff69319bf in R_tryCatchError (body=body@entry=0x7ffff759cdb0 <_parsevector_wrap>, bdata=0x7ffff6db3e80, handler=handler@entry=0x7ffff759cd60 <_handler_wrap>, hdata=0x0)
at errors.c:2399
#38 0x00007ffff75a778f in _cffi_f_R_tryCatchError (self=<optimized out>, args=<optimized out>) at build/temp.linux-x86_64-cpython-311/_rinterface_cffi_api.c:3338
#39 0x00007ffff7a4a7b2 in cfunction_call (func=0x7ffff6dd4540, args=<optimized out>, kwargs=<optimized out>) at Objects/methodobject.c:553
#40 0x00007ffff7a015ad in _PyObject_MakeTpCall (tstate=0x7ffff7de7b18 <_PyRuntime+166328>, callable=0x7ffff6dd4540, args=<optimized out>, nargs=4, keywords=0x0) at Objects/call.c:214
#41 0x00007ffff78f7c5c in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4769
#42 0x00007ffff7b2b57a in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7e48110, tstate=0x7ffff7de7b18 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#43 _PyEval_Vector (tstate=0x7ffff7de7b18 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>)
at Python/ceval.c:6434
#44 0x00007ffff78fa99c in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffff7002e00, callargs=0x7ffff71c6350, func=0x7ffff6d99760, tstate=<optimized out>) at Python/ceval.c:7349
#45 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#46 0x00007ffff7b2bc9c in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7e48020, tstate=0x7ffff7de7b18 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#47 _PyEval_Vector (tstate=tstate@entry=0x7ffff7de7b18 <_PyRuntime+166328>, func=func@entry=0x7ffff71d6020, locals=locals@entry=0x7ffff71f6dc0, kwnames=0x0, argcount=0, args=0x0)
at Python/ceval.c:6434
#48 0x00007ffff7b2be3a in PyEval_EvalCode (co=0x7ffff71c10d0, globals=<optimized out>, locals=0x7ffff71f6dc0) at Python/ceval.c:1148
#49 0x00007ffff7b56a10 in run_eval_code_obj (locals=0x7ffff71f6dc0, globals=0x7ffff71f6dc0, co=0x7ffff71c10d0, tstate=0x7ffff7de7b18 <_PyRuntime+166328>) at Python/pythonrun.c:1741
#50 run_mod (mod=mod@entry=0x4a66c0, filename=filename@entry=0x7ffff71ff410, globals=globals@entry=0x7ffff71f6dc0, locals=locals@entry=0x7ffff71f6dc0, flags=flags@entry=0x7fffffffbe58,
arena=arena@entry=0x7ffff711b7b0) at Python/pythonrun.c:1762
#51 0x00007ffff7b76752 in pyrun_file (flags=0x7fffffffbe58, closeit=1, locals=0x7ffff71f6dc0, globals=0x7ffff71f6dc0, start=257, filename=0x7ffff71ff410, fp=0x4064d0)
at Python/pythonrun.c:1657
#52 _PyRun_SimpleFileObject (fp=0x4064d0, filename=0x7ffff71ff410, closeit=1, flags=0x7fffffffbe58) at Python/pythonrun.c:440
#53 0x00007ffff7b76ff1 in _PyRun_AnyFileObject (fp=0x4064d0, filename=0x7ffff71ff410, closeit=1, flags=0x7fffffffbe58) at Python/pythonrun.c:79
#54 0x00007ffff7b7990f in pymain_run_file_obj (skip_source_first_line=0, filename=0x7ffff71ff410, program_name=0x7ffff71967b0) at Modules/main.c:360
#55 pymain_run_file (config=0x7ffff7dcdb60 <_PyRuntime+59904>) at Modules/main.c:379
#56 pymain_run_python (exitcode=0x7fffffffbe54) at Modules/main.c:601
#57 Py_RunMain () at Modules/main.c:680
#58 0x00007ffff763d10e in __libc_start_call_main (main=main@entry=0x401040 <main>, argc=argc@entry=2, argv=argv@entry=0x7fffffffc068) at ../sysdeps/nptl/libc_start_call_main.h:58
#59 0x00007ffff763d1c9 in __libc_start_main_impl (main=0x401040 <main>, argc=2, argv=0x7fffffffc068, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7fffffffc058) at ../csu/libc-start.c:360
#60 0x0000000000401075 in _start ()
If I compile rpy2 with -O0 and run the testcase under valgrind, the following (likely bogus due to memory corruption, since it makes no sense) traceback is reported:
Exception ignored from cffi callback <function _handler_wrap at 0x6594220>:
Traceback (most recent call last):
File "/nix/store/pwi3r33c8xhic5011a5kcxp7q4h6ic6q-python3-3.11.9-env/lib/python3.11/site-packages/rpy2/rinterface_lib/_rinterface_capi.py", line 690, in _handler_wrap
AttributeError: 'set' object has no attribute 'rlib'
If I compile both rpy2 and cffi with -O0, then the test runs successfully (that is, it correctly reports a parser error). The test also works with the standard optimization levels if I force RPY2_CFFI_MODE="ABI".
Confusingly, the same test succeeds under Python 3.12 (or with Python 3.11.9 + R 4.3.3). I noticed the latest CI run at #1051 segfaults at the same file (test_embedded_r.py) under Python 3.12, however. The fact that it works with -O0 makes me think the problem is not related to a particular version, but rather due to unpredictable consequences of undefined behavior.
python -m rpy2.situation output:
rpy2 version:
3.5.16
Python version:
3.11.9 (main, Apr 2 2024, 08:25:04) [GCC 13.2.0]
Looking for R's HOME:
Environment variable R_HOME: None
Calling `R RHOME`: /nix/store/fi1xm60r31a9v3bp0xy1gmjpfvny7hdk-R-4.4.0/lib/R
Environment variable R_LIBS_USER: None
R's value for LD_LIBRARY_PATH:
/nix/store/fi1xm60r31a9v3bp0xy1gmjpfvny7hdk-R-4.4.0/lib/R/lib:/nix/store/x012fzpyg0vxza2iiajdfp0437b11bli-gfortran-13.2.0-lib/lib:/nix/store/lrwdy6y3yr4nc4324awflxl832qxvz6a-gfortran-wrapper-13.2.0/bin:/nix/store/kv5ph9jwim2aar8p87in3x3hx0wc9yd0-gfortran-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0:/nix/store/kv5ph9jwim2aar8p87in3x3hx0wc9yd0-gfortran-13.2.0/lib:/nix/store/sdyn5834dw2w6nkijzn66zj280wwl82x-blas-3/lib:/nix/store/c853p3yib0ydda48hvb5az81icj2rnr2-lapack-3/lib:/nix/store/p16s3zg96afj68mfp17m03svhwmdgcz8-tcl-8.6.13/lib:/nix/store/lmci3q5pnjkas9d2x81kd05jgmiffkn3-tk-8.6.13/lib:/nix/store/2vwkssqpzykk37r996cafq7x63imf4sp-openjdk-21+35/lib/openjdk/lib/server:/nix/store/is7dfka7vihiig3w789s1b0az40cgmrj-pipewire-1.0.6-jack/lib
R version:
In the PATH: R version 4.4.0 (2024-04-24) -- "Puppy Cup"
Loading R library from rpy2: OK
Additional directories to load R packages from:
None
C extension compilation:
include:
['/nix/store/fi1xm60r31a9v3bp0xy1gmjpfvny7hdk-R-4.4.0/lib/R/include']
libraries:
['R', 'pcre2-8', 'deflate', 'lzma', 'bz2', 'z', 'rt', 'dl', 'm', 'icuuc', 'icui18n']
library_dirs:
['/nix/store/fi1xm60r31a9v3bp0xy1gmjpfvny7hdk-R-4.4.0/lib/R/lib', '/nix/store/g3vi60zgyjsvij7xkk6dxky1hkwh0ynd-pcre2-10.43/lib']
extra_compile_args:
['-std=c99']
extra_link_args:
['-Wl,--export-dynamic', '-fopenmp']
Directory for the R shared library:
lib
CFFI extension type
Environment variable: RPY2_CFFI_MODE
Value: CFFI_MODE.ANY
ABI: PRESENT
API: PRESENT
To be honest, I am not sure whether this is a problem with cffi or rpy2. Please let me know if I can provide any extra debugging information.