[erlang-questions] Binary manipulation

[Bob Cowdery]

Thanks. Sounds like I should move all the processing down into a
threaded C NIF. So I would pass back, in the case of decoding an empty
binary or a binary which is the multiple of 64 required. This can easily
be done on a separate thread as there are several calls to build one
response. Opposite on the encoding bit, pass back the largest list of
binaries that can be encoded from the multiple of 64 passed in. However,
this amounts to a polling loop from the erlang process waiting for the
result.  That doesn't sound so great. Another alternative is to do all
that in a linked-in driver as it is a big chuck of C code that is
processing these large binaries anyway. Downside is that it breaks good
program structure.

Bob

On 13/03/2011 10:22, Lukas Larsson wrote:
> If you do write a nif call which could potentially take a long time you should use an async thread to do the work in it rather than doing it in the erlang vm thread. If not you could get some strange behaviour out of the vm depending on how unlucky you are with processes scheduling and unrelated work which other processes do. See http://www.erlang.org/cgi-bin/ezmlm-cgi?4:mss:55980:201101:jconogbffcaogeijbdkl for details.
>
> Lukas
>
>
> ----- Original Message -----
> From: "Bob Cowdery" <bob@REDACTED>
> To: erlang-questions@REDACTED
> Sent: Sunday, March 13, 2011 11:03:57 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
> Subject: Re: [erlang-questions] Binary manipulation
>
> Having received only one private reply which would not really scale up
> and having looked quite closely at all the API's available I can't see
> any way to manipulate binary data with any efficiency. I am therefore
> thinking of doing a NIF library.  However I'm a bit concerned about what
> the following statement means:
>
> "Avoid doing lengthy work in NIF calls as that may degrade the
> responsiveness of the VM. NIFs are called directly by the same scheduler
> thread that executed the calling Erlang code. The calling scheduler will
> thus be blocked from doing any other work until the NIF returns."
>
> I presume this just means the calling process cannot do any other work
> until the NIF returns and not that all processes can suffer degradation.
>
> Thanks
> Bob
>
> On 11/03/2011 09:01, Bob Cowdery wrote:
>> Hi
>>
>> I would appreciate any advice/ideas on how to efficiently handle a large
>> binary.
>>
>> The binary in question has a couple of protocol wrappers around it which
>> are no problem as the bit syntax easily copes with that. What I'm not
>> sure of is how to handle the data.
>>
>> <<I:24, Q:24, M:16 ... repeated 63 times >>
>>
>> What I have to do is extract I and Q which are 24 bit le values and end
>> up with another binary which is:
>>
>> <<I/float, Q/float ... repeated 63 times (see below) >>
>>
>> Where I and Q are now floating point numbers. I will need M in the
>> future, but not right now. It's a little more complicated than that
>> because several of the raw data binaries will go into the processed
>> binary which will be a multiple of 64 up to 1024.
>>
>> I will also need to do the opposite, i.e. take the processed binary and
>> transform it back to the raw data binary.
>>
>> Thanks
>> Bob
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>