[erlang-questions] Sending a large block of binary data from c to erlang

[Richard Evans]

Thanks Joe. This is very helpful.

Now that I have set the packet-size as 4 bytes, this does work. I can 
happily send the 2 meg binary block from erlang to c.

But there is a perplexing slow-down which I do not understand. 

I am encoding the binary block like this:

encode({load_instance_from_binary, GameIndex, BinaryBlock}) -> 
    Result = lists:append([[?SERVER_INSTRUCTION_LOAD_INSTANCE_FROM_BINARY | 
convertToFourBytes(GameIndex)], convertToFourBytes(byte_size(BinaryBlock)), 
binary:bin_to_list(BinaryBlock)]),
    Result.

convertToFourBytes(X) ->
    <<B1,B2,B3,B4>> = <<X:32>>,
    [B1,B2,B3,B4].

This converts the binary block to list format before appending it to the 
rest of the list. This bin_to_list does not seem to cause any slow-down.

The slow-down seems to happen when sending the 2 meg message from erlang to 
c. In this function, taken from Joe's book:

handle_call(Msg, _From, #state{port=Port}=State) ->
    EncodedMsg = encode(Msg),
    validate_message(EncodedMsg),
    Port ! {self(), {command, EncodedMsg}},
    receive
        {Port, {data, Data}} ->
            DecodedData = decode(Data)
    end,
    {reply, DecodedData, State};

I have a breakpoint on the c function that corresponds to the message. 

What perplexes me is that there is a 5 minute delay between the message 
being sent to the c-port (Port ! {self(), {command, EncodedMsg}}, and the 
c-function actually getting called!

Any advice much appreciated.

thanks,

Richard



On Saturday, November 10, 2012 1:45:35 PM UTC, Joe Armstrong wrote:
>
> The only answer is "try it and see" - it all depends. The computer I'm 
> typing
> this mail on has 4GB or RAM - compared to that 1M is "nothing".
>
> Are you running on a 24GB monster or 5MB embedded system? Do you have
> one parallel process, sending one 1 MB message, or a thousand? Do you send
> your 1 MB messages once every millisecond - or once a year.
>
> Just knowing that your message is 1MB tells me nothing about the other 
> parts
> of your system - when GB memories came I stopped calling MBytes "large"
> and TB disks means that GB files are not really "large" - these are 
> relative terms.
> Today Peta bytes is "large" - but we get a 1000x scale change every ten 
> years
>
> There no intrinsic reason why it should not work - Just make sure the 
> packet length
> will fit into 4 bytes and use {packet,4}.
>
> I have a file replication system in Erlang - I send all small files in 
> single messages
> (I think the cut-off is 10 MB) and larger files in chunks (mainly so I can 
> restart them
> if things go wrong)
>
> Oh and a port-driver should be fine for this.
>
> Cheers
>
> [aside] performance always surprises me - I think reading a few hundred 
> small files
> will take a long time - but it takes a blink of a second - I guess this is 
> because
> it would take me a heck of a long time to do this - GHz clock speeds are so
> fast that just about everything I think would take a long time doesn't.
>
>
>
> /Joe
>
>
>
>
>
>
>
>
>
> On Fri, Nov 9, 2012 at 3:31 PM, Richard Evans <richardpri...@REDACTED<javascript:>
> > wrote:
>
>> I am using a port to communicate between c and erlang, as described in 
>> Joe Armstrong's excellent book, and in this tutorial: 
>> http://www.erlang.org/doc/tutorial/c_port.html#id63121
>>
>> So far, the traffic between the two has been pretty small: typically a 
>> hundred bytes or so at a time. 
>>
>> But now I need to (occasionally) send a large block of binary data from c 
>> to erlang: up to 1 meg. 
>>
>> How should I handle such large blocks of binary data? Is there anything 
>> wrong with increasing the size of the buffer (called buf in the tutorial 
>> mentioned above) to 1 meg? 
>>
>> Is there a better way of doing this? (NB I am using a port, rather than a 
>> port-driver (in which the c-code is linked directly to the erlang code)).
>>
>> thanks,
>> Richard Evans
>>  
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-q...@REDACTED <javascript:>
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121121/d7a03740/attachment.htm>