[erlang-questions] Sending a large block of binary data from c to erlang

[Joe Armstrong]

The only answer is "try it and see" - it all depends. The computer I'm
typing
this mail on has 4GB or RAM - compared to that 1M is "nothing".

Are you running on a 24GB monster or 5MB embedded system? Do you have
one parallel process, sending one 1 MB message, or a thousand? Do you send
your 1 MB messages once every millisecond - or once a year.

Just knowing that your message is 1MB tells me nothing about the other parts
of your system - when GB memories came I stopped calling MBytes "large"
and TB disks means that GB files are not really "large" - these are
relative terms.
Today Peta bytes is "large" - but we get a 1000x scale change every ten
years

There no intrinsic reason why it should not work - Just make sure the
packet length
will fit into 4 bytes and use {packet,4}.

I have a file replication system in Erlang - I send all small files in
single messages
(I think the cut-off is 10 MB) and larger files in chunks (mainly so I can
restart them
if things go wrong)

Oh and a port-driver should be fine for this.

Cheers

[aside] performance always surprises me - I think reading a few hundred
small files
will take a long time - but it takes a blink of a second - I guess this is
because
it would take me a heck of a long time to do this - GHz clock speeds are so
fast that just about everything I think would take a long time doesn't.



/Joe









On Fri, Nov 9, 2012 at 3:31 PM, Richard Evans <
richardprideauxevans@REDACTED> wrote:

> I am using a port to communicate between c and erlang, as described in Joe
> Armstrong's excellent book, and in this tutorial:
> http://www.erlang.org/doc/tutorial/c_port.html#id63121
>
> So far, the traffic between the two has been pretty small: typically a
> hundred bytes or so at a time.
>
> But now I need to (occasionally) send a large block of binary data from c
> to erlang: up to 1 meg.
>
> How should I handle such large blocks of binary data? Is there anything
> wrong with increasing the size of the buffer (called buf in the tutorial
> mentioned above) to 1 meg?
>
> Is there a better way of doing this? (NB I am using a port, rather than a
> port-driver (in which the c-code is linked directly to the erlang code)).
>
> thanks,
> Richard Evans
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121110/f5cfbe18/attachment.htm>