[erlang-bugs] Erlang problems with big binaries

[Mikael Pettersson]

Mikael Pettersson writes:
 > Mikael Pettersson writes:
 >  > Nico Kruber writes:
 >  >  > I'd like to 'bor' (binary or) a big binary but the result is an empty list(?!)
 >  >  > Smaller binaries work as expected.
 >  >  > 
 >  >  > Steps to reproduce (at least on Erlang 17.4):
 >  >  > 
 >  >  > A=erlang:term_to_binary(lists:seq(1000000,2200000)).
 >  >  > B=erlang:term_to_binary(lists:seq(1000001,2200001)).
 >  >  > ASize=erlang:bit_size(A).
 >  >  > BSize=erlang:bit_size(B).
 >  >  > <<ANr:ASize>> = A.
 >  >  > <<BNr:BSize>> = B.
 >  >  > XNr = ANr bor BNr.
 >  >  > <<XNr:BSize>>.
 >  >  > 
 >  >  > 
 >  >  > Please note that printing the actual values of A or B is not a good idea ;)
 >  >  > and so would XNr but including it directly into a binary fails with the same 
 >  >  > error: "** exception error: bad argument"
 >  >  > <<(ANr bor BNr):BSize>>.
 >  > 
 >  > The problem is that erts/emulator/beam/big.c:big_norm() contains:
 >  > 
 >  >     if ((arity = BIG_NEED_SIZE(xl)-1) > BIG_ARITY_MAX)
 >  >       return NIL;  /* signal error (too big) */
 >  > 
 >  > where BIG_ARITY_MAX is 0x7ffff, but:
 >  > 
 >  > 1) this is an implementation restriction which can, as this
 >  > case proves, be exceeded by simple application code, and
 >  > 
 >  > 2) erts_{band,bor,bxor}() check for this with ASSERT(is_not_nil(...)),
 >  > but that's wrong since this is about input validation and implementation
 >  > limits not fundamental logic, and it's ineffective for release builds.
 >  > 
 >  > These three functions need to turn this case into a system limit
 >  > exception, just like a lot of other places in erl_arith.c already do.
 >  > (Well, at least _bor() and _bxor() do, perhaps _band() doesn't need to.)
 > 
 > Actually, it's worse than I thought.  Observe:
 > 
 > 1>  A=erlang:term_to_binary(lists:seq(1000000,2200000)).
 > <<131,108,0,18,79,129,98,0,15,66,64,98,0,15,66,65,98,0,15,
 >   66,66,98,0,15,66,67,98,0,15,...>>
 > 2> ASize=erlang:bit_size(A).
 > 48000096
 > 3> <<ANr:ASize>> = A.
 > <<131,108,0,18,79,129,98,0,15,66,64,98,0,15,66,65,98,0,15,
 >   66,66,98,0,15,66,67,98,0,15,...>>
 > 4> ANr band ANr.
 > big_norm: xl 750002 arity 0xb71b2 > 0x7ffff
 > beam/erl_arith.c:1050:erts_band() Assertion failed: ((arg1) != ((~((Uint) 0) << 6) | ((0x3 << 4) | ((0x2 << 2) | 0x3))))
 >                 Abort
 > 
 > So band:ing a bignum with itself (which by definition requires no more
 > bits than the input bignum) triggers the limit check.  I also get errors
 > for trivial ops like ANr - 1.
 > 
 > Looks like <<ANr:ASize>> = A can construct invalid bignums.

In the general case, this calls erl_bits.c:erts_bs_get_integer_2(), which
calls big.c:bytes_to_big().  Nothing here enforces the BIG_ARITY_MAX limit.
However, most arithmetic does, via big_norm().

Also, I personally think BIG_ARITY_MAX is way too small -- we can build
both tuples and binaries much larger than that.

I'm thinking bytes_to_big() needs to mirror big_norm()'s error checking
behaviour, and erts_bs_get_integer_2() needs to check for an error
return (NIL) from bytes_to_big() and return THE_NON_VALUE in that case
(after restoring p->htop).