trailing a growing file

[Shawn Pearce]

tail -f just does the same thing you are doing by reading the log,
hitting EOF, and waiting a short while.

Classic UNIX/POSIX doesn't have a way to get new data from a file, or
to find out when a file/directory is modified - applications just have
to poll.

FreeBSD and IRIX (that I know of) have methods where an application
can open a directory or file, and tell the OS it is interested in
being notified when a change is made.  Then you read from it and
carry on.  Under IRIX this is called FAM (File Activity Monitor).  I
don't know what FreeBSD calls it; and to my knowledge Linux does not
have support for this.

If the log writer is another Erlang node I would suggest just sending
the log data to the target node, as well as write it to disk.  If the
log writer is an external process, well, you are stuck.  If it is a
syslogd maintained file, perhaps you could put an Erlang node into the
list of targets syslogd will write the logs to.

If the log writer is anything else, you'll need to either patch the
application (example would be with Apache build or reuse a mod_* that
can send the logs over the network to an Erlang node), or just poll
the log file...

If it was me, I'd be polling from within Erlang using a raw file (so
no extra process is started) rather than using a port to tail -f.


Raphael Bellec <raphael.bellec@REDACTED> wrote:
>  
> Hello,
> I am actually working on a multi-log multi-host monitoring (and
> reacting) system, made in erlang and I need some advice:
>  
> I would like to read logs file as soon as data is written to them. I
> actually have found 2 ways for this purpose:
> - With Io module, pooling (each time I get an eof, I wait a small time
> then try again).
> - With ports, spawning "tail -f -n 1 /path/to/file", which is the method
> I currently use.
>  
> Is there a other way to "receive" news lines as they are written in the
> file without pooling or using an external application? I read few
> messages saying that there was no blocking IO in erlang, but I think
> this was mainly from a VM perspective. (I will have few hundreds logs to
> monitor)
>  
> I currently use the port solution and I sometimes have a "broken pipe"
> error, I think this is because of the "in" parameter in open_port, but I
> hat it sometime without this parameter, anybody have a more detailed
> explanation of what could happen ?
>  
> Regards.
>  
> Raphaël.
>  
> Ps: the subject is directly taken from a point in the erlang PLEAC which
> is still empty.

-- 
Shawn.

  If a subordinate asks you a pertinent question, look at him as if he had
  lost his senses.  When he looks down, paraphrase the question back at him.