[erlang-questions] Binary manipulation

[Hynek Vychodil]

Processing your binary in way

decode(Bin) -> [ {I,Q} || <<I:24/little, Q:24/little>> <= Bin ].

encode(Data) -> << <<I:24/little, Q:24/little>> || {I,Q} <- Data >>.

or roughly equivalent

decode(<<>>) -> [];
decode(<<I:24/little, Q:24/little, Rest/binary>>) -> [ {I,Q} | decode(Rest) ].

encode(Data) -> encode(Data, <<>>).

encode([], Result) -> Result;
encode([{I,Q}|T], Acc) -> encode(T, <<Acc/binary, I:24/little, Q:24/little>>).

should be good enough especially compiled with HiPE unless you
*measure* it is not.

On Mon, Mar 14, 2011 at 10:44 AM, Bob Cowdery <bob@REDACTED> wrote:
> Thanks. Sounds like I should move all the processing down into a
> threaded C NIF. So I would pass back, in the case of decoding an empty
> binary or a binary which is the multiple of 64 required. This can easily
> be done on a separate thread as there are several calls to build one
> response. Opposite on the encoding bit, pass back the largest list of
> binaries that can be encoded from the multiple of 64 passed in. However,
> this amounts to a polling loop from the erlang process waiting for the
> result.  That doesn't sound so great. Another alternative is to do all
> that in a linked-in driver as it is a big chuck of C code that is
> processing these large binaries anyway. Downside is that it breaks good
> program structure.
>
> Bob
>
> On 13/03/2011 10:22, Lukas Larsson wrote:
>> If you do write a nif call which could potentially take a long time you should use an async thread to do the work in it rather than doing it in the erlang vm thread. If not you could get some strange behaviour out of the vm depending on how unlucky you are with processes scheduling and unrelated work which other processes do. See http://www.erlang.org/cgi-bin/ezmlm-cgi?4:mss:55980:201101:jconogbffcaogeijbdkl for details.
>>
>> Lukas
>>
>>
>> ----- Original Message -----
>> From: "Bob Cowdery" <bob@REDACTED>
>> To: erlang-questions@REDACTED
>> Sent: Sunday, March 13, 2011 11:03:57 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
>> Subject: Re: [erlang-questions] Binary manipulation
>>
>> Having received only one private reply which would not really scale up
>> and having looked quite closely at all the API's available I can't see
>> any way to manipulate binary data with any efficiency. I am therefore
>> thinking of doing a NIF library.  However I'm a bit concerned about what
>> the following statement means:
>>
>> "Avoid doing lengthy work in NIF calls as that may degrade the
>> responsiveness of the VM. NIFs are called directly by the same scheduler
>> thread that executed the calling Erlang code. The calling scheduler will
>> thus be blocked from doing any other work until the NIF returns."
>>
>> I presume this just means the calling process cannot do any other work
>> until the NIF returns and not that all processes can suffer degradation.
>>
>> Thanks
>> Bob
>>
>> On 11/03/2011 09:01, Bob Cowdery wrote:
>>> Hi
>>>
>>> I would appreciate any advice/ideas on how to efficiently handle a large
>>> binary.
>>>
>>> The binary in question has a couple of protocol wrappers around it which
>>> are no problem as the bit syntax easily copes with that. What I'm not
>>> sure of is how to handle the data.
>>>
>>> <<I:24, Q:24, M:16 ... repeated 63 times >>
>>>
>>> What I have to do is extract I and Q which are 24 bit le values and end
>>> up with another binary which is:
>>>
>>> <<I/float, Q/float ... repeated 63 times (see below) >>
>>>
>>> Where I and Q are now floating point numbers. I will need M in the
>>> future, but not right now. It's a little more complicated than that
>>> because several of the raw data binaries will go into the processed
>>> binary which will be a multiple of 64 up to 1024.
>>>
>>> I will also need to do the opposite, i.e. take the processed binary and
>>> transform it back to the raw data binary.
>>>
>>> Thanks
>>> Bob
>>>
>>> ________________________________________________________________
>>> erlang-questions (at) erlang.org mailing list.
>>> See http://www.erlang.org/faq.html
>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>



-- 
--Hynek (Pichi) Vychodil

Analyze your data in minutes. Share your insights instantly. Thrill
your boss.  Be a data hero!
Try GoodData now for free: www.gooddata.com