[erlang-questions] Sending a large block of binary data from c to erlang

[Erik Søe Sørensen]

Five minutes certainly sounds like a perplexingly high delay.
Why do you convert your binaries to lists, though? As opposed to either
building a binary instead, or an iolist?
Den 21/11/2012 19.19 skrev "Richard Evans" <richardprideauxevans@REDACTED>:

> Thanks Joe. This is very helpful.
>
> Now that I have set the packet-size as 4 bytes, this does work. I can
> happily send the 2 meg binary block from erlang to c.
>
> But there is a perplexing slow-down which I do not understand.
>
> I am encoding the binary block like this:
>
> encode({load_instance_from_binary, GameIndex, BinaryBlock}) ->
>     Result = lists:append([[?SERVER_INSTRUCTION_LOAD_INSTANCE_FROM_BINARY
> | convertToFourBytes(GameIndex)],
> convertToFourBytes(byte_size(BinaryBlock)),
> binary:bin_to_list(BinaryBlock)]),
>     Result.
>
> convertToFourBytes(X) ->
>     <<B1,B2,B3,B4>> = <<X:32>>,
>     [B1,B2,B3,B4].
>
> This converts the binary block to list format before appending it to the
> rest of the list. This bin_to_list does not seem to cause any slow-down.
>
> The slow-down seems to happen when sending the 2 meg message from erlang
> to c. In this function, taken from Joe's book:
>
> handle_call(Msg, _From, #state{port=Port}=State) ->
>     EncodedMsg = encode(Msg),
>     validate_message(EncodedMsg),
>     Port ! {self(), {command, EncodedMsg}},
>     receive
>         {Port, {data, Data}} ->
>             DecodedData = decode(Data)
>     end,
>     {reply, DecodedData, State};
>
> I have a breakpoint on the c function that corresponds to the message.
>
> What perplexes me is that there is a 5 minute delay between the message
> being sent to the c-port (Port ! {self(), {command, EncodedMsg}}, and the
> c-function actually getting called!
>
> Any advice much appreciated.
>
> thanks,
>
> Richard
>
>
>
> On Saturday, November 10, 2012 1:45:35 PM UTC, Joe Armstrong wrote:
>>
>> The only answer is "try it and see" - it all depends. The computer I'm
>> typing
>> this mail on has 4GB or RAM - compared to that 1M is "nothing".
>>
>> Are you running on a 24GB monster or 5MB embedded system? Do you have
>> one parallel process, sending one 1 MB message, or a thousand? Do you send
>> your 1 MB messages once every millisecond - or once a year.
>>
>> Just knowing that your message is 1MB tells me nothing about the other
>> parts
>> of your system - when GB memories came I stopped calling MBytes "large"
>> and TB disks means that GB files are not really "large" - these are
>> relative terms.
>> Today Peta bytes is "large" - but we get a 1000x scale change every ten
>> years
>>
>> There no intrinsic reason why it should not work - Just make sure the
>> packet length
>> will fit into 4 bytes and use {packet,4}.
>>
>> I have a file replication system in Erlang - I send all small files in
>> single messages
>> (I think the cut-off is 10 MB) and larger files in chunks (mainly so I
>> can restart them
>> if things go wrong)
>>
>> Oh and a port-driver should be fine for this.
>>
>> Cheers
>>
>> [aside] performance always surprises me - I think reading a few hundred
>> small files
>> will take a long time - but it takes a blink of a second - I guess this
>> is because
>> it would take me a heck of a long time to do this - GHz clock speeds are
>> so
>> fast that just about everything I think would take a long time doesn't.
>>
>>
>>
>> /Joe
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Nov 9, 2012 at 3:31 PM, Richard Evans <richardpri...@REDACTED**com>wrote:
>>
>>> I am using a port to communicate between c and erlang, as described in
>>> Joe Armstrong's excellent book, and in this tutorial: http://www.erlang.
>>> **org/doc/tutorial/c_port.html#**id63121<http://www.erlang.org/doc/tutorial/c_port.html#id63121>
>>>
>>> So far, the traffic between the two has been pretty small: typically a
>>> hundred bytes or so at a time.
>>>
>>> But now I need to (occasionally) send a large block of binary data from
>>> c to erlang: up to 1 meg.
>>>
>>> How should I handle such large blocks of binary data? Is there anything
>>> wrong with increasing the size of the buffer (called buf in the tutorial
>>> mentioned above) to 1 meg?
>>>
>>> Is there a better way of doing this? (NB I am using a port, rather than
>>> a port-driver (in which the c-code is linked directly to the erlang code)).
>>>
>>> thanks,
>>> Richard Evans
>>>
>>> ______________________________**_________________
>>> erlang-questions mailing list
>>> erlang-q...@REDACTED
>>> http://erlang.org/mailman/**listinfo/erlang-questions<http://erlang.org/mailman/listinfo/erlang-questions>
>>>
>>>
>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121122/bcea0446/attachment.htm>