CT Test Results

    3 files    135 suites   49m 22s ⏱️
1 655 tests 1 598 ✅ 57 💤 0 ❌
2 293 runs  2 217 ✅ 76 💤 0 ❌

Results for commit c89cbee.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

This looks promising. I pushed two minor fixups.
Does this PR fix your regression?

@sverker Yes, we tested the patches today and all the issues we were seeing appear to have been resolved.

I rebased-squashed it for inclusion into maint (OTP-28.2).

@sverker Some additional information: we've noticed a latency regression when at high CPU utilization (90%+) that seems to be due to scheduler slowdown from the large number of pollset entries per scheduler.

I'll draw some ASCII art in a separate comment...

@sverker Alright, so before scheduler pollsets, my understanding is that this is roughly what was happening:

┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐
│ S0      │  │ S1      │  │ S2      │  │ S3      │  │ I/O     │
└─────────┘  └─────────┘  └─────────┘  └─────────┘  │ PS a..h │
                                                    └─────────┘

At high load, all of the pollset operations are still isolated to the I/O event loop and won't uniformly slow down the schedulers when there are a really large number of file descriptors in the pollsets.

With the new scheduler pollset feature, we have this:

┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐
│ S0      │  │ S1      │  │ S2      │  │ S3      │
│ PS a..b │  │ PS c..d │  │ PS e..f │  │ PS g..h │
└─────────┘  └─────────┘  └─────────┘  └─────────┘

First off: is this an accurate view of how this works?

We tested a version of OTP where ERTS_POLL_USE_SCHEDULER_POLLING was defined as (0) to disable the feature entirely and the latency regression at high CPU went away.

If all of this is correct, would it make sense to have a socket option that we could use to selectively prevent them from being included in the scheduler pollset optimization? We think this might help us in very large number of sockets scenarios.

For small number of sockets, the scheduler pollset optimization should provide more of a benefit, I would think.

Maybe something like:

socket:setopt(Socket, {otp, scheduler_polling}, false).

What do you think?

There is one scheduler pollset, namely sched_pollset in erl_check_io.c. Only one scheduler at a time waits for events from the pollset and not until all outstanding triggered events have been handled. There are some internal docs written about this at https://www.erlang.org/doc/apps/erts/checkio.html.

You can disable scheduler pollset with erl +IOs false. If you don't get any performance gain from scheduler pollset then that seems to be the easy solution.

If an option would be necessary, I would prefer something less implementation specific than {otp, scheduler_polling} if possible.