[erlang-questions] Turning rebar into a package manager

[José Valim]

I believe one of the issues that would need to be addressed first in rebar
is the one of repeatability.

rebar does not ensure that developers in the same project are going to use
the same snapshot of a dependency. This means developers can depend on the
same git repository but different commits leading to the famous "the tests
pass on my machine".

I am aware this could be fixed by the team enforcing a commit sha when
declaring the dependency but many simply don't do it. Besides it puts the
work on you to manage the commit shas. If you want to update the
dependency, you need to remove the commit sha, update the repo, write the
new commit sha.

This could be easily fixed by having a lock file per project with the
commit shas of the dependencies. "rebar get-deps" should retrieve the shas
in such lock, if one is available, or simply get the latest and write to
the lock. A command like "rebar unlock-deps" could be available when
developers want to update their dependencies, regardless of the current
lock.

Also, regarding Joe's proposal, picking up a version from a git repository
may be tricky. The best approach is to index git tags only, hoping the
publisher won't modify an existing tag. If someone is depending on a branch
though, it should not be considered a version or a part of .erl_packages. I
believe it should still belong to the repo.

*José Valim*
www.plataformatec.com.br
Skype: jv.ptec
Founder and Lead Developer



On Fri, Oct 19, 2012 at 12:36 PM, Joe Armstrong <erlang@REDACTED> wrote:

> Here is a programming exercise ...
>
> I think we can turn rebar into a package manager with 4 simple and one not
> so
> simple tweaks.
>
> This would turn rebar from being something that is very useful to something
> that is very very useful :-)
>
>
> (aside - I'm revising my Erlang book, and I'd like to describe
> how package management works, but there is no package manager
> which is a shame - so I got to thinking about package managers
> and wondered why still don't have a decent package manager.
>
> The one I've seen that I like a lot is npm (node package manager)
> so we could do worse than follow their design)
>
> So this is how I think rebar could become a package manager
>
>
> Tweak 1
> =======
>
>    > rebar get-deps
>
> Fetches the dependencies in rebar.conf and stores them in
> the current directory, in a directory called deps.
>
> After a while you get dependencies scattered all over the place,
> I've seen many examples of this.
>
> This means that there is no central place to go and look if you
> want to find code.
>
> NPM sticks ALL dependencies under ${HOME}/.npm
>
> Suggestion:
>
>    All dependencies should be stored in ${HOME}/.erl_packages
>
>    (Aside - if we followed npm the directory structure would be
>     something like)
>
>    ${HOME}/.erl_packages/cowboy/6.2.1/src
>                                                                /ebin
>                                                      /5.2.3
>                         /stdlib/12.3.1
>                                /14.6.8
>
>    etc. ie a package/Vsn/src ... structure
>
>
> Tweak 2
> =======
>
>    move rebar.config to ${HOME}/.erl_packages/rebar.config
>    and add commands to automatically edit it
>
>    > rebar add_package foo
>
>    might add a line like
>    {foo, ".*", "git://...."}
>
>    to rebar.config
>
>    The point here is to change the content of rebar.config with
>    shell commands rather that editing it by hand
>
> Tweak 3
> =======
>    fix rebar so it just downloads a particular version of a
>    program and not all the .git stuff - most people will only
>    want to use other peoples code, not hack it.
>
> Tweak 4
> =======
>
> Fix the erlang load paths to find all the dependencies
>
> I do this now. Put all my dependencies in
> ${HOME}/nobackup/erlang_imports/deps and put the following
> code in my .erlang startup file
>
>   Home = os:getenv("HOME").
>   Dir = Home ++ "/nobackup/erlang_imports/deps",
>   case file:list_dir(Dir) of
>      {ok, L} ->
>          lists:foreach(fun(I) ->
>                                Path = Dir ++ "/" ++ I ++ "/ebin",
>                                code:add_path(Path)
>                        end, L);
>     _ ->
>         void
>   end.
>
> Tweak 5
> =======
>    This can't be done by rebar
>    Make an index of all erlang apps on github that follow
>    the erlang packaging structure
>
>    Somebody has to write a program to do this.
>
>    The package index should be at a well know URL
>
>    > rebar get_index
>
>
>    This should fetch the  index from the well-know URL and store in
>    ${HOME}/.erl_packages
>
>
>    > rebar search XXX
>
>    would search the fetched index
>
>    > rebar add XXXX
>
>    would take the index entry add it to the config fill fetch the code
>    and compile it
>
> Note the following:
>
> 1) The trust model.
>
>    We trust the supplier of a program not the program.
>
>    So for example on github we might trust programs
>    published by the user joearms (me) or erlang (the OTP release)
>    but not by some_guy_ive_never_hear_of
>
>    It would be quite easy to add a trust section to rebar.config
>
>    {trust, ["git://joearms", git://erlang", ...]}
>
> 2) There is no "publish" step
>
>    We put our code on github (or somewhere) and hope that it gets indexed
>    by the indexer.
>
>    We might mail the indexer if it is published in an obscure place.
>
>   Comments?
>
>    Cheers
>
> /Joe
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121019/7479a6b4/attachment.htm>