[erlang-bugs] file_server processes very large numbers of messages slowly

[Björn-Egil Dahlberg]

Much of the "cooked" file-handling stems from distributed erlang. (Which I
now read Raimo explained so on to the next issue ...)

Transparent distributed access to a file-system or a fast local file-system?

Though a very nifty feature, one can certainly question the reasons for
having a file-server being the default behavior in erlang. I would argue
against this.

A (normal?) three tier solution would probably be better.
Tier 1: Gritty driver/nif interface to files. (prim_file)  unstable api
Tier 2: local file acess (file) stable api
Tier 3: distributed access (not file)

At first glance Tier 1 and 2 could be collapsed to the same tier but that
would not be ideal. I want the prim_* api to be changeable. (Which is the
reason why it is not documented I believe)
For example, changing the #file_info file times to epochs instead of having
*localtime* datetime tuples while the file module still can have its
datetime tuples. (This is a change which probably will be in r15).

It is unlikely we would remove file-server from file and into a another
abstraction layer, but the default behavior should be changed. And file
should be extended with the functionality Raimo described. We should
probably remove as much functionality as possible in the efile_driver and
let higher tier (in erlang) take care of that.

Now if I only could find a time-machine or a pill that lets me work 36h a
day.

// Björn-Egil
-- Concern for the past is hindering our future.




2011/9/27 Alexandru Scvorţov <scvalex@REDACTED>

> Hi Scott,
>
> Thanks a lot for the suggestion.  We needed a few more functions than
> just read_file_info(), including a few from filelib (which used file
> internally).  We ended up writing a few wrappers around the prim_file
> functions.  We also had a bit of problem hitting the open files limit,
> but we already had a system in place to manage that.
>
> Thanks again,
>
> Alex
>
> On Thu, Sep 15, 2011 at 01:24:48PM -0500, Scott Lystig Fritchie wrote:
> > Alexandru, I'm about to hop on a plane, so I don't have much time.  Yes,
> > your analysis appears sound.  There has been discussion in the last week
> > on the mailing list that is related to prim_file and the file_server_2
> > (registered proc name) process.
> >
> > The probable answer (at least for now) is to write your own interface to
> > the prim_file:read_file_info() (name?) function so that you can take
> > advantage of the "+A n" I/O worker Pthread pool ... assuming of course
> > that your workload doesn't overwhelm them to the point where selective
> > receive doesn't bite you again.
> >
> > -Scott
> >
> >
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs@REDACTED
> http://erlang.org/mailman/listinfo/erlang-bugs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20110928/972ba6a8/attachment.htm>