[erlang-patches] don't create oversize bignums in binary matching

[Mikael Pettersson]

Bignums are artifically restricted in size.  Arithmetic and logical
operations check the sizes of resulting bignums, and turn oversize
results into system_limit exceptions.

However, this check is not performed when bignums are constructed by
binary matching.  The consequence is that such matchings can construct
oversize bignums that satisfy is_integer/1 yet don't work.  Performing
arithmetic such as Term - 0 fails with a system_limit exception.  Worse,
performing a logical operation such as Term band Term results in [].
The latter occurs because the size checking (e.g. in erts_band()) is
a simple ASSERT(is_not_nil(...)) on the result of the bignum operation,
which internally is [] (NIL) in the case of oversize results.  However,
ASSERT is a no-op in release builds, so the error goes unnoticed and []
is returned as the result of the band/2.

This patch addresses this by preventing oversize bignums from entering
the VM via binary matching:

- the internal bytes_to_big() procedure is augmented to return NIL for
  oversize results, just like big_norm()
- callers of bytes_to_big() are augmented to check for NIL returns and
  signal errors in those cases
- erts_bs_get_integer_2() can only fail with badmatch, so that is the
  Erlang-level result of oversize bignums from binary matches
- big_SUITE.erl is extended with a test case that fails without this
  fix (no error signalled) and passes with it (badmatch occurs)

Credit goes to Nico Kruber for the initial bug report.

Signed-off-by: Mikael Pettersson <mikpelinux@REDACTED>

Links:

git fetch git://github.com/mikpe/otp.git avoid-oversize-bignums

https://github.com/mikpe/otp/compare/erlang:maint...avoid-oversize-bignums
https://github.com/mikpe/otp/compare/erlang:maint...avoid-oversize-bignums.patch

https://github.com/erlang/otp/pull/603