Streaming Input

[Joe Armstrong (AL/EAB)]

   use binaries - that's what they are for

   First write something like this:

	extract(BinIn, Need, BinAcc) ->
	    Got = size(BinIn),
	    if
	       Got > Need ->
			{Before, After} = split_binary(BinIn, Need),
		      Result = concat_binary([BinAcc, Before]),
		      {done, Result, After};
		 Got == Need ->
		      Result = concat_binary([BinAcc,BinIn]),
		      {done, Result, <<>>};
		 Got < Need ->
			BinAcc1 = concat_binary([BinAcc, BinIn]),
			{more, Need - Got, BinAcc1}
	    end.

	<aside>
	 Organising the code like this should make it pretty clear
	 what's going on ie write the "if" clearly with three branches

		if
			Got > Need ->
				%% too much data - have to split it
				...
			Got == Need ->
				%% exactly right no need to split
				...
			Got < Need ->
				%% not enough. no need to split
				...
		end
	</aside>

	here

	in extract(BinIn, Need, BinAcc) 
  		More and Sofar are binaries
		Need is the required block length

	if size(BinIn) > Need we split the block into two chunks
 	and return {done, Bin, After} Bin = is the data you need
	otherwise {more, Need-Got, BinAcc}

	BinAcc is a binary accumulator containing all the data received so far.

	Then just arrange so code to call this

	Cheers

	/Joe





> -----Original Message-----
> From: owner-erlang-questions@REDACTED
> [mailto:owner-erlang-questions@REDACTED]On Behalf Of orbitz
> Sent: den 27 februari 2005 07:49
> To: erlang-questions@REDACTED
> Subject: Streaming Input
> 
> 
> I am working with a protocol where the size of the following block is 
> told to me so I can just convert the next N bytes to, say, a string.  
> The problem is though, I'm trying to write this so it handles 
> a stream 
> properly, so in the binary I have could be all N bytes that I 
> need, or 
> something less than N. So at first I tried:
> 
> extract_string(Tail, 0, Res) ->
>   {ok, {string, Res}, Tail};
> extract_string(<<H, Tail/binary>>, Length, Res) ->
>   extract_string(Tail, Length - 1, lists:append(Res, [H]));
> extract_string(<<>>, Length, Res) ->
>   case dispatch_message() of
>     {decode, _, Data} ->
>       extract_string(Data, Length, Res)
>   end.
> 
> When the binary is empty but I still need more data it waits 
> for more.  
> I don't know if this is the proper idiom (it seems gross to 
> me but I am 
> unsure of how to do it otherwise).  This is incredibly slow though.  
> With a long string that I need to extract it takes a lot of 
> CPU and far 
> too long.  So I decided to do:
> 
> extract_string(Data, Length, _) ->
>   <<String:Length/binary, Tail/binary>> = Data,
>   {ok, {string, binary_to_list(String)}, Tail}.
> 
> In terms of CPU and time this is much much better, but if I 
> don't have 
> all N bytes it won't work.  Any suggestions?
>