[erlang-questions] Fast directory walker

[Michael Truog]

On 12/10/2016 12:43 AM, Frank Muller wrote:
> Hi Michael,
>
> Better, but still ~2 seconds:
>
> > timer:tc(fun() -> filelib:fold_files("/usr/share", ".*", true, fun(F, N) -> N + 1 end, 0) end).
> {1993074,28953 <tel:1993074,28953>}
>
> If I get it correctly the call matches only on files, not on dirs.
Yes, the regex only matches on filenames, not directory names. Currently, the filelib:fold_files/5 function requires a regex so you are likely paying a penalty for it matching on a wildcard regex on each filename.  So, it would probably be nice to have a filelib:fold_files/4 that doesn't require a regex.

Best Regards,
Michael

>
> /Frank
>
> Le sam. 10 déc. 2016 à 03:12, Michael Truog <mjtruog@REDACTED <mailto:mjtruog@REDACTED>> a écrit :
>
>
>
>     On 12/09/2016 03:15 PM, Frank Muller
>
>     wrote:
>
>
>
>
>>
>>
>>
>>
>>     Hi
>>
>>
>>
>>
>>
>>
>>
>>     I would like to improve the speed of my directory
>>
>>     walker.
>>
>>
>>
>>
>>
>>
>>
>>     walk(Dir) ->
>>
>>
>>         {ok, Files} = prim_file:list_dir(Dir),
>>
>>
>>         walk(Dir, Files).
>>
>>
>>
>>
>>
>>
>>
>>     walk(Dir, [ Basename | Rest ]) ->
>>
>>
>>         Path = filename:join([ Dir, Basename ]),
>>
>>
>>         case filelib:is_dir(Path) of
>>
>>
>>             true  ->
>>
>>
>>                 walk(Path);
>>
>>
>>             false ->
>>
>>
>>                 io:format("~s~n", [Path]),
>>
>>
>>     filelib:file_size(Path)
>>
>>
>>         end,
>>
>>
>>         walk(Dir, Rest);
>>
>>
>>     walk(_, []) ->
>>
>>
>>         ok.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>     Compared
>>
>>     to almost anything i found on the web, it’s still very slow:
>>
>>
>>
>>
>>     > timer:tc(fun() -> dir:walk("/usr/share")
>>
>>     end).
>>
>>
>>
>>
>>     {4662361 <tel:4662361>,ok}
>>
>>
>
>
>
>
>
>     Have you tried filelib:fold_files/5
>
>     (http://erlang.org/doc/man/filelib.html#fold_files-5) ?
>
>
>
>
>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20161210/c1ed75b8/attachment.htm>