[erlang-questions] Obsolete exported functions file:raw_{read, write}_file_info/2 - why?

[Paul Fisher]

Agreed.  For us, file_server_2 represented a significant bottleneck on our 8 core nodes (e.g. delete/2, rename/2, etc.)  We created a module that used the low-level prim_file interface directly to get around the bottlenecks.  We have a bounded number of processes that perform file operations, so we ended up with the following basic approach in this module:


-spec delete(Name :: file:name()) -> 'ok' | {'error', file:posix()}.

delete(Name) ->
    Port = get_port(),
    prim_file:delete( Port, Name ).

…

get_port() ->
    case get( ?MODULE ) of
        undefined ->
            Port = case prim_file:start() of
                       {ok, Port0} -> Port0;
                       {error, Reason} ->
                           exit( {?MODULE, failed_to_create_prim_file, Reason} )
                   end,
            put( ?MODULE, Port ),
            Port;
        Port -> Port
    end.


There was some discussion a few months ago about someone experimenting/testing an alternative file implementation (NIF-based, I believe), but I forget the source of the comment.

On Sep 14, 2011, at 11:42 AM, Scott Lystig Fritchie wrote:

> Joseph Norton <norton@REDACTED> wrote:
> 
> jn> I'm curious if someone happens to know why the
> jn> file:raw_read_file_info/1 and file:raw_read_file_info/2 methods were
> jn> made obsolete?
> 
> Joe, I stumbled across those functions while putting DTrace probes into
> the efile_drv driver.  I thought, hey, those would be quite useful.
> 
> I recommend reading the file.erl source.  It's quite instructive to see
> how many file I/O functions are redirected to the 'file_server_2'
> process.  For file I/O-intensive applications (e.g. Hibari and Riak I
> know, CouchDB and RabbitMQ I'd guess), having all calls to(*)
> file:read_file_info/1 serialized by the file server process is a source
> of latency that we (DB authors) may desire to live without.
> 
> Having planted bugs in prim_file.erl accidentally, it is also quite
> instructive to see how many different ways the VM's bootstrap process
> can be broken.  So I now understand reluctance to deprecate functions
> that may be necessary at some weird situation when booting.  However,
> there may come a time when some of us submit a patch to expose *more*
> raw file I/O functions in file.erl, not fewer.
> 
> -Scott
> 
> (*) Or file:delete/1 or file:rename/2 or ...
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

--
paul

director, platform services
alertlogic, inc.
713-484-8383 x2314

"When in doubt, use brute force."  -- Ken Thompson