[erlang-questions] node to node message passing

[Ryan Zezeski]

On Sun, Sep 12, 2010 at 6:48 AM, Morten Krogh <mk@REDACTED> wrote:

>
> For communication between node A and node B, there is a process (send
> process) on each node, that coordinates all messages. The send process
> keeps queues of different priorities around, e.g., a high priority, medium
> priority and low priority. Messages are split up into fragments of
> a maximum size. The receiver(node B) send process assembles the fragments
> into the original message and delivers it locally to the
> right process. The fragments ensure that no single transfer will occupy the
> connection for very long.
> There will be a function send_priority where the user can specify a
> priority. The usual send will default to medium, say.
> Net tick will use high priority, of course. Small messages that are needed
> to produce a web application response can have high priority. File transfers
> for backup purposes can have low priority.
> The send process then switches between the queues in some way, that could
> be very similar to context switching priorities.
>
>
What happens to the low priority, large binary msg when the system becomes
loaded with a constant stream of smaller, high priority messages?  It seems
the low priority msg would become neglected and then you'll be complaining
how the msg takes too long to arrive.  I think in all but the most trivial
cases you would have to understand the inner workings of the priority
scheduler, if I may call it that, to understand how your messages will be
delivered.  To me, this sounds like a problem to be handled by the
application.

Another thought, why are you trying to pass such a large amount of data
anyways?  If you are truly constrained by such a slow network then why not
do everything in your power to avoid passing large amounts of data?  I'm
thinking of Hadoop here--move the process to the data, not the other way
around.

-Ryan