[erlang-questions] Looking for the wisdon of experience...

[Matthew Dempsky]

On Sat, Apr 12, 2008 at 7:29 PM, Darren New <dnew@REDACTED> wrote:
>  I'm assuming the monitor, the query engine, the thing listening to
>  the radio stations, etc would likely each be one "node", rather than
>  putting them all into one node? So I'd be running 'qe@REDACTED' and
>  'qe@REDACTED' and 'radio@REDACTED' and 'radio@REDACTED', rather
>  than having all the processing on one machine all in the same server,
>  right? That's a better way to go?

It's a matter of personal preference.  We use separate erl nodes for
each service/host because sometimes the easiest way for us to resolve
a problem is to restart Erlang and it's less of a problem when that
only affects one service rather than multiple.

>  Right now, each running process logs a "heartbeat" to the central
>  "ground control" server, so I know if something died or I lost
>  ISP connectivity. I'm assuming the right way to do this in Erlang
>  is to watch for nodeup and nodedown messages. Yes?

Yes.  Be sure to request those messages with erlang:monitor_node or
net_kernel:monitor_nodes though.

>  If this is the
>  case, and connectivity fails and then comes back, does the Erlang
>  runtime try to reestablish connectivity automatically? (Some of
>  our ISPs have been less than wonderful.)

See the documentation for the dist_auto_connect setting in the
kernel(6) man page.

>  Also, I'm wondering whether it would be better to have a
>  specific TCP server communicating between cities (i.e.,
>  a custom gen_tcp server) or whether it would be sufficient
>  to simply use the normal inter-node commuincation stuff?
>  Or would something like Mr Armstrong's "lib_chan" be a
>  better start?  It looks like with even trivial abstraction,
>  this wouldn't really be a problem either way, considering
>  how easy it is to ship terms over TCP.

That's (again) mostly a matter of personal preference.  There are some
technical issues to consider like security (if one site is
compromised, the Erlang distribution layer would let the nodes there
take over other sites).  For the most part though, if you're just
planning on using binary_to_term and term_to_binary then you might as
well use Erlang's distribution layer.

If you'd like to be able to interoperate with other languages without
too much difficulty, HTTP typically makes a decent (if slightly
heavier) solution.

> [Questions about error logging.]

I don't have much experience with Erlang's built-in error logging
stuff, so I can't really answer these questions.  For the most part,
we run erlang nodes in a screen session or using runit so the console
output is logged to disk, and then we have monitoring software that
alerts us to problems that we should consult the logs for.

>  Cron-like functionality: How is this best handled? If I have some Erlang
>  task I want to do once a day, or once a week, or to retry something every
>  couple of hours until it succeeds then sleep for a few days... Do people
>  just set this up as a separate process? Or is there something like "cron"
>  written already inside of the Erlang libraries that I just haven't found?
>  Clearly it wouldn't be difficult to implement. Or do people fire
>  up a separate node via cron to run something like this, then let it
>  shut itself down when it's done?  I saw the "hibernate" BIF, which
>  looks like the sort of thing useful for this.

Most of our code only needs to be run ever N seconds/minutes/hours, so
we just have a gen_server call timer:send_interval in its init/1
function and then add an appropriate handle_info clause.

> [Questions about mnesia.]

We mostly use mnesia only as a distributed hash table, so I don't have
anything original to add to the ideas you've already suggested.