[erlang-questions] max fds for epoll incorrectly reduced to FD_SETSIZE

[Steve Vinoski]

Doing some load testing on Linux a couple days ago, I started getting
the "Driver select called with too large file descriptor N" error,
despite the fact that ulimit -n was 64k and I had passed the "+K true"
option to an R12B-5 system built with kernel poll enabled. After
staring at the code for awhile, I determined the problem to be the
same issue Joel Reymont reported back in August last year:

<http://erlang.org/pipermail/erlang-questions/2008-August/037574.html>

though he was on OS X and I'm on Linux (and I sure wish I would have
found that link *before* digging through and independently finding it
myself!).

Since kernel poll is not on by default, select() is used as the
fallback/default, and the following code from the erts_poll_init()
function in erts/emulator/sys/common/erl_poll.c limits the max fds to
FD_SETSIZE:

#if ERTS_POLL_USE_SELECT && defined(FD_SETSIZE)
    if (max_fds > FD_SETSIZE)
        max_fds = FD_SETSIZE;
#endif

The ERTS_POLL_USE_SELECT is true because ERTS_POLL_USE_FALLBACK gets
set in erts/emulator/sys/common/erl_poll.h for non-Win32 systems when
ERTS_POLL_USE_EPOLL is true:

#define ERTS_POLL_USE_FALLBACK (ERTS_POLL_USE_KQUEUE || ERTS_POLL_USE_EPOLL)

#if !ERTS_POLL_USE_KERNEL_POLL || ERTS_POLL_USE_FALLBACK
#  if defined(ERTS_USE_POLL)
#    undef ERTS_POLL_USE_POLL
#    define ERTS_POLL_USE_POLL 1
#  elif !defined(__WIN32__)
#    undef ERTS_POLL_USE_SELECT
#    define ERTS_POLL_USE_SELECT 1
#  endif
#endif

We can't undef FD_SETSIZE to fix this because it's needed for cases
where "+K true" isn't passed, and we can't set it to a super-high
value because setting FD_SETSIZE beyond 2k doesn't work so well with
select().

I believe the proper fix here is to make the erts_poll_init() function
check for kernel polling enabled and not limit max_fds to FD_SETSIZE
for that case, but I haven't yet come up with a good way to implement
that fix -- any hints appreciated. But I agree with Joel that this a
bug.

--steve