[erlang-questions] Binaries problem

[Per Gustafsson]

Francesco Pierfederici wrote:
> Hi,
> 
> I have encountered a problem with handling binaries and I do not know
> what the cause might be. In a nutshell, I seem to be unable to
> assemble binaries larger than 2097149 bytes:
> 
> -module(test).
> -export([test/2]).
> 
> 
> test(FileName, NumBytes) ->
>     {ok, Fd} = file:open(FileName, [raw, binary]),
>     {ok, Data} = file:read(Fd, NumBytes),
>     file:close(Fd),
> 
>     NumRestBits = NumBytes * 8 - 1,
>     <<Bit:1, Rest:NumRestBits>> = Data,
> 
>     NewData = <<Rest:NumRestBits, 0:1>>,
>     ok.
> 
> 
> And then
> 
> inanna> erl
> Erlang (BEAM) emulator version 5.5.4 [source] [async-threads:0] [hipe]
> [kernel-poll:false]
> 
> Eshell V5.5.4  (abort with ^G)
> 1> c(test).
> ./test.erl:11: Warning: variable 'Bit' is unused
> ./test.erl:13: Warning: variable 'NewData' is unused
> {ok,test}
> 2> test:test('/Users/fpierfed/Desktop/software/xcode_2.4.1_8m1910_6936315.dmg',
> 2097148).
> ok
> 3> test:test('/Users/fpierfed/Desktop/software/xcode_2.4.1_8m1910_6936315.dmg',
> 2097149).
> 
> =ERROR REPORT==== 1-May-2007::16:33:57 ===
> Error in process <0.37.0> with exit value:
> {badarg,[{test,test,2},{erl_eval,do_apply,5},{shell,exprs,6},{shell,eval_loop,3}]}
> 
> ** exited: {badarg,[{test,test,2},
>                     {erl_eval,do_apply,5},
>                     {shell,exprs,6},
>                     {shell,eval_loop,3}]} **
> 
> 
> Is this a bug? Am I doing something wrong?
> 
> Thank you!
> Francesco
> 
> P.S. I am running this on a Mac OS X box using both R11B-4 and the
> latest snapshot (R11B_2007-05-01) with identical results.
> P.P.S. I get the same results with different files so there seems to
> be nothing peculiar about the 900+ MB XCode file.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions

I think that the problem you are running in to is that you are creating 
an integer that is too big for the erlang runtime system. I'm not sure 
exactly where this limit is but it should be in the order of what you 
are creating.

You can sidestep this problem (and have a much more efficient 
implementation by writing:


test(FileName, NumBytes) ->
   {ok, Fd} = file:open(FileName, [raw, binary]),
   {ok, Data} = file:read(Fd, NumBytes),
   file:close(Fd),

   <<Bit:1, Pad:7, Rest/binary>> = Data,

   NewData = <<Pad:7,Rest/binary, 0:1>>,
   ok.

Or if you use the following compiler option: [bitlevel_binaries]

test(FileName, NumBytes) ->
   {ok, Fd} = file:open(FileName, [raw, binary]),
   {ok, Data} = file:read(Fd, NumBytes),
   file:close(Fd),

   <<Bit:1, Rest/bitstr>> = Data,

   NewData = <<Rest/bitstr, 0:1>>,
   ok.

Of course this syntax is only experimental.

Per