[erlang-questions 34] Re: [erlang-questions] Binary no-copy

Bob Cowdery <>
Sat Mar 26 20:57:41 CET 2011


Paul

Thank you for the explanations. I have data running right through the
app now using a simulator in Erlang pretending to be the hardware. Next
up is to plug in the DSP. I've done a linked-in driver before so hoping
that won't be too difficult.

Bob

On 23/03/2011 01:50, Per Gustafsson wrote:
> So essentially everytime this function is called one of the four
> clauses will match
>
> 1. Will match if the state is exactly a full block right now. It will
> send that frame to the dsp module, and then it will continue to decode
> the current frame using a new state.
>
> 2. If the current state is larger than a full block we will split out
> the full block and send that to the dsp module, we will then continue
> to decode the current frame using whatever was left in the old state
> as the new state.
>
> 3. If the frame still has samples in it and neither of the first two
> cases were true we will take out one set of samples, convert to them
> to floats and add them to the end of the state and continue to decode
> the rest of the frame.
>
> 4 If the frame is empty we will return the current state.
>
> The difference between this version and the version that uses a
> comprehension is that we are checking for every sample whether the
> state is now a full frame or not.
>
> This allows us to write the new samples directly into the state
> binary, rather than creating a temporary binary. That we then copy
> into the state binary.
>
>
> 2011/3/18 Bob Cowdery <
> <mailto:>>
>
>     Humour me. I'm still learning erlang and it still takes me ages to
>     write a few lines at times. Comments in-line.
>
>     Bob
>
>
>     On 17/03/2011 20:54, Per Gustafsson wrote:
>>     You're right it should be as below to get the same behaviour, but
>>     I think it will work the same for input that makes sense.
>>
>     Straight forward
>
>>     do_decode_frame(Frame, State) ->
>>        %% Separate the command bytes from the frame data
>>        <<C1:8, C2:8, C3:8, C4:8, C5:8, Rest/binary>> = Frame,
>>        decode_samples(Rest, State).
>>
>     First clause will match a full block where frames and blocks are
>     aligned. Never happen in practice as 63 will never be a multiple
>     of 64 but it's right that the code should not know this.
>
>>     decode_samples(Frame, FullBlock) when bytesize(FullBlock) ==
>>     ?BinSize ->
>>       dsp:process_block(FullBlock),
>>       decode_samples(Frame, <<>>);
>     Second clause will match when a FullBlock has > ?BinSize and will
>     split into ?BinSize and Rest
>>     decode_samples(Frame, <<FullBlock:?BinSize/binary, Rest/binary>>) ->
>>       dsp:process_block(FullBlock),
>>       decode_samples(Frame, Rest);
>     Third clause. Now I get lost. This will match pretty much any
>     binary > 8 bytes and any State. So this will get called from first
>     and second clause. The next two clauses seem to run into each
>     other. Don't understand that. Is the binary comprehension hidden
>     somewhere in there?
>
>
> Sorry the third clause is totally wrong. It should have been:
> decode_samples(<<I:24/big-signed, Q:24/big-signed, _M:16/big-signed,
> Rest/binary>>, State) ->
>   decode_samples(Rest, <<State/binary, (I/?S):32/float,
> (Q/?S):32/float>>);
>
>>     decode_samples(<<I:24/big-signed, Q:24/big-signed, _M:16/big-signed,
>>     Rest/binary>>, State) ->
>>       decode_samples(<<State/binary, (I/?S):32/float, (Q/?S):32/float>>;
>>     decode_samples(<<>>, State) ->
>>      {ok, State}.
>>
>     Would this work as the last two clauses:
>     decode_samples(Frame, State) ->
>         decode_samples(<<>>, << State/binary, <<(I/?S):32/float,
>     (Q/?S):32/float>> || <<I:24/big-signed, Q:24/big-signed,
>     _M:16/big-signed>> <= Frame >>)
>
>     decode_samples(<<>>, State) ->
>          {ok, State}.
>>
>>     2011/3/17 Bob Cowdery <
>>     <mailto:>>:
>>     > Thanks. More or less what I thought that the only copy is in
>>     the append
>>     > operation. The one liner is the big one in terms of time but I will
>>     > profile the whole thing eventually. I don't think your solution
>>     does
>>     > quite the same thing as the sent block must be exactly the
>>     correct size
>>     > not >= but I appreciate the kind of structure you are using
>>     with small
>>     > clauses is more elegant.
>>     >
>>     > Bob
>>     >
>>     > On 17/03/2011 17:11, Per Gustafsson wrote:
>>     >> It will copy binaries, but essentially the state binary will
>>     act like
>>     >> a growable array, pre-allocating space to grow in, so there
>>     will be
>>     >> some copying , but it will be linear in the size of the resulting
>>     >> array.
>>     >> It might be a little bit faster to rewrite this as shown
>>     below. The
>>     >> reason I think this might be faster is that you do not create a
>>     >> temporary samples array that just gets copied. Whether this is
>>     faster
>>     >> or not in reality is of course a matter of benchmarking.
>>     >>
>>     >> %% Constants
>>     >> %% Input scale factor
>>     >> -define (S, (1 bsl 23) - 1).
>>     >> %% Block size for DSP (interleaved I and Q)
>>     >> -define (BlockSz, 1024).
>>     >> %% Samples in a Metis frame (interleaved I and Q)
>>     >> -define (MetisSmpls, 63).
>>     >> %% Bytes in a float
>>     >> -define (FloatSz, 4).
>>     >> %% Size of binary for given block size
>>     >> -define (BinSz, ?BlockSz*2*?FloatSz*?MetisSmpls).
>>     >>
>>     >> do_decode_frame(Frame, State) ->
>>     >>    %% Separate the command bytes from the frame data
>>     >>    <<C1:8, C2:8, C3:8, C4:8, C5:8, Rest/binary>> = Frame,
>>     >>    decode_samples(Rest, State).
>>     >>
>>     >> decode_samples(Frame, State) when size(State) >= ?BinSz ->
>>     >>   dsp:process_block(State),
>>     >>   decode_samples(Frame, <<>>);
>>     >> decode_samples(<<I:24/big-signed, Q:24/big-signed,
>>     _M:16/big-signed,
>>     >> Rest/binary>>, State) ->
>>     >>   decode_samples(<<State/binary, (I/?S):32/float,
>>     (Q/?S):32/float>>;
>>     >> decode_samples(<<>>, State) ->
>>     >>   {ok, State}.
>>     >>
>>     >> 2011/3/17 Bob Cowdery <
>>     <mailto:>>
>>     >>> Hi,
>>     >>>
>>     >>> Am I correct in thinking that the code fragments below will
>>     not copy any
>>     >>> binary data. I've not tested the function yet so it may have
>>     bugs but
>>     >>> I'm interested in whether it will be optimal in efficiency.
>>     State starts
>>     >>> off as an empty binary. Frames are decoded and placed in a
>>     block. There
>>     >>> are not an integral number of frames to a block. The function
>>     >>> dsp:process_block() is in another gen-server.
>>     >>>
>>     >>> Thanks
>>     >>> Bob
>>     >>>
>>     >>> %% Constants
>>     >>> %% Input scale factor
>>     >>> -define (S, (1 bsl 23) - 1).
>>     >>> %% Block size for DSP (interleaved I and Q)
>>     >>> -define (BlockSz, 1024).
>>     >>> %% Samples in a Metis frame (interleaved I and Q)
>>     >>> -define (MetisSmpls, 63).
>>     >>> %% Bytes in a float
>>     >>> -define (FloatSz, 4).
>>     >>> %% Size of binary for given block size
>>     >>> -define (BinSz, ?BlockSz*2*?FloatSz*?MetisSmpls).
>>     >>>
>>     >>> do_decode_frame(Frame, State) ->
>>     >>>    %% Separate the command bytes from the frame data
>>     >>>    <<C1:8, C2:8, C3:8, C4:8, C5:8, Rest/binary>> = Frame,
>>     >>>    %% What's left now is <<I:24, Q:24, M:16 ..repeated 63
>>     times.. >>
>>     >>>    %% This one liner converts the I and Q to scaled floats
>>     >>>    Samples = << <<(I/?S):32/float, (Q/?S):32/float>> ||
>>     >>> <<I:24/big-signed, Q:24/big-signed, _M:16/big-signed>> <=
>>     Rest >>,
>>     >>>    CurrentSz = size(State),
>>     >>>    FrameSz = size(Samples),
>>     >>>    if
>>     >>>        (CurrentSz + FrameSz) < ?BinSz ->
>>     >>>            NewState = <<State/binary, Samples/binary>>;
>>     >>>        true ->
>>     >>>            %% Binary part of this transfer
>>     >>>            CutAt = ?BinSz - CurrentSz - 1,
>>     >>>            Bp = binary_part(Samples, {0, CutAt}),
>>     >>>            %% Binary part of remainder
>>     >>>            Br = binary_part(Samples, {CutAt, FrameSz - CutAt}),
>>     >>>            %% Process this block
>>     >>>            dsp:process_block(<<State/binary, Bp/binary>>),
>>     >>>            NewState = Br
>>     >>>    end,
>>     >>>    {ok, NewState}.
>>     >>>
>>     >>> ________________________________________________________________
>>     >>> erlang-questions (at) erlang.org <http://erlang.org> mailing
>>     list.
>>     >>> See http://www.erlang.org/faq.html
>>     >>> To unsubscribe;
>>     mailto:
>>     <mailto:>
>>     >>>
>>     >
>>     >
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110326/f4a38a98/attachment.html>


More information about the erlang-questions mailing list