[erlang-bugs] enif_make_double returns invalid term for 'nan'

[Mikael Pettersson]

Heinz Nikolaus Gies writes:
 > When enif_make_double gets a nan value passed as the double argument it returns a invalid term, this term can neither be printed (via the fmt*) functions nor can it be deserialized by binary_to_term when serializing it the following binary is generated:
 > 
 > 6> io:format("~p~n", [term_to_binary(V)]).
 > <<131,99,110,97,110,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>
 > 
 > V is the double returned by enif_make_double

There are two issues here:
1. the VM disallows INFs and NANs, but NIF authors may not be aware of that
2. enif_make_double() SHOULD reject non-finite doubles but it doesn't, and
   its documentation doesn't mention any failure modes; returning THE_NON_VALUE
   may not be detected as a failure by the NIF and so this non-value may leak
   into aggregates (lists or tuples) where it will totally break things

When looking at this I noticed that enif_make_atom() also has problems: it
may fail and return THE_NON_VALUE if the resulting atom would be too long,
but the documentation doesn't mention this possibility so the NIF might
continue and store this non-value in aggregates before returning.

The OTP folks need to straighten up error handling in the enif_make_*()
functions.