[erlang-questions] Problem with pattern matching in large binaries

[Edwin Fine]

Thanks for the info. I rebuilt my Erlang VM after making Per's change,
and the behavior is now as follows:

1> {ok,B} = file:read_file("large_test_data.txt").
{ok,<<"...">>}
2> byte_size(B).
1037563663
3> Offset = byte_size(B) - 100.
1037563563
4> <<_Skip:Offset/binary,Last100/binary>> = B.
<<"...">>
5> byte_size(Last100).
100
6> byte_size(_Skip).
1037563563 %% This is now correct. Before the 1-line change I made to
erl_bits.h, this was incorrect

Everything I was trying to do now works correctly. All that is left is
to regression test my applications, just in case. Maybe you could make
this available as a patch - it's a trivial change.

Thanks again.

Regards
Edwin Fine
Fine Computer Consultants, Inc.

On Tue, May 6, 2008 at 5:09 AM, Sverker Eriksson
<sverker@REDACTED> wrote:
> Edwin Fine wrote:
>  > I have a text file that is 1,037,563,663 bytes in length. In the
>  >  shell, I read it all into memory as follows:
>  >
>  >  >{ok,B} = file:read_file("/tmp/data").
>  [...]
>
>
>  > > <<_Skip:Offset/binary,Last100/binary>> = B.
>  > > byte_size(Last100).
>  > 100
>  > > byte_size(_Skip).
>  > 500692651
>  >
>  > WTF??? Checking Last100 showed that it was indeed the data from offset
>  > 500693651, not the last 100 bytes. Where did the other 500MB-odd go?
>  >
>  [...]
>
>  Per Gustafsson wrote:
>  > I took a look at this and it seems that in the BYTE_OFFSET macro on line
>  >   153 in erl_bits.h there is a cast to unsigned that really should be to
>  > Uint that probably causes this problem, but I do not have a good machine
>  > for testing this change. (Our amd64 does not have enough memory to build
>  > a 1 Gb binary)
>  >
>  > Per
>  >
>
>  I will take a look at this with my Intel Quad, 64bit, 4 Gb.
>
>  If Per is right, a fix will be released to R12B-3.
>
>  /Sverker, Erlang/OTP, Ericsson
>
>  _______________________________________________
>  erlang-questions mailing list
>  erlang-questions@REDACTED
>  http://www.erlang.org/mailman/listinfo/erlang-questions
>
>