ValueError: filedescriptor out of range in select() #559
Description
On glibc based Linux systems select() can monitor only file descriptor numbers that are less than FD_SETSIZE (1024).
This is an unreasonably low limit for many modern applications.
When running Spark applications (via pyspark, which uses py4j) we frequently observe:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/py4j/java_gateway.py", line 2283, in run
readable, writable, errored = select.select(
^^^^^^^^^^^^^^
ValueError: filedescriptor out of range in select()