[erlang-questions] can nodes fail/recover too fast to be seen?

[Tim Watson]

On 5 Jul 2013, at 21:07, Per Hedeland <per@REDACTED> wrote:
> 
> No, of course they can be dropped - if the destination node goes down,
> it's impossible to know whether a given, sent message was a) received at
> the remote *host*, b) received by the remote Erlang node, c) received by
> the remote Erlang process, d) processed by the remote Erlang process, or
> e) none of the above. But if you monitor/link (e.g. use
> gen_server:call()), you will know that "badness happened", and can take
> corrective action. "Re-sending only messages that need to be re-sent" is
> not possible in general, and this is not specific to Erlang distribution.
> 

Yeah I know about those issues. We were discussing it on another thread earlier today. :)

> See also http://www.erlang.org/faq/academic.html#id58000, which Matthias
> Lang was kind enough to write up in a nice form based on some ramblings
> of mine in the distant past. It could probably use s/link/monitor/, but
> the general principle holds.
> 

I'll take a look, thanks.

>> 
>> Just out of interest, is this enforced by epmd or internally?
> 
> epmd has no role in inter-node communication once the connection has
> been established.

Thats what I thought.

> TCP enforces "cannot be used for ...". The
> VM/net_kernel will not make a new connection until it has decided that
> the old one isn't working any more, and at that point it will generate
> the node_down/'DOWN'/exit messages.

So given that intermediate kit could be proxying the connection, it is /possible/ that the node was reset but the network layer didn't notice? I mention that because if it is the case, people should bear it in mind when designing their infrastructure.

> there is a 25% possibility of the new node instance
> getting the same "creation" value.

I wasn't fully aware of this node generation tracking stuff. We see a lot of stale pids (discarding messages from
Old incarnations of the node) in some deployments, but then we actually store pids in mnesia which makes things a bit more complicated viz getting the timing right between node down messages,  updating the db and actually using the pids.

Cheers
Tim