[erlang-questions] Turning rebar into a package manager

Joe Armstrong <>
Tue Oct 23 09:46:18 CEST 2012

On Fri, Oct 19, 2012 at 2:33 PM, Fred Hebert <> wrote:
> For anyone interested, there's been an interesting blog post on the topic of
> package managers written at
> http://hyperthunk.wordpress.com/2012/05/28/does-erlangotp-need-a-new-package-management-solution/

That's very interesting. Thanks for the reference

I got to thinking a bit more. ...

The above link has an example command

> git fetch-pack --include-tag -v :hyperthunk/gitfoo.git refs/tags/lager-0.9.4

This appears to fetch a tagged tree without all all other versions -
just what I wanted

At some level of abstraction we need to tag versions of applications.
Using the latest
version is fine for development but not for shipping a product.

If we adopt a common tagging convention then we can do two things.

     1) express dependencies (ie foo-1.4.5 depends upon bar-3.2.6)
     2) make a "release" ie say that a certain set of tagged
applications has been tested and is known
         to work together.

At the moment I guess very few Erlang git projects made by rebar have
vI.J.K style tags.

Two more problems remain:

     A) Weakening the dependencies

        I know what foo-3.4.2 depends upon bar-4.2.5 means - this is
very precise

        Should we allow foo-3.* depends upon bar-4.*

        There must be some language of dependencies, and conventions
(ie odd major version numbers are experimental,
         even stable)

         Does anybody have any pointers to a discussion of this.

         I'd like a few simple rules that cover the common use cases
rather than fancy rules that cover everything

     B) The *one* module name problem

          We can't load modules in foo-1.2.3 and foo-2.1.6 into the
same node so if A depends upon
          foo-1.2.3 and B depends upon foo-2.1.6 we can't run A and B
in the same node

         There are a number of solutions possible:

         i)  name_munging -> convert xx.erl to xx_foo_1.2.3.erl

            It *is* possible - but you have to take care of registered
process names etc. there are a lot of
            edge cases that make this tricky.

        ii) Don't solve the problem at all. Run A and B in different nodes

        My feeling is ii) is correct - we could then define a node as
a release (ie a set of tagged applications)
        that are know to work together.

        I think problems of /consistency of
modules/security/authentication should be solved at the node level
        and not the process/module level

How could one make a release?

     a) All contributors must tag versions of their applications and
follow common packaging conventions
     b) we need an improved manifest

Maybe volunteers could make releases - actually a release is little
more than a set of applications
that are know to.

Seems we also need to standardize a new an improved manifest file - we
need two types
one that describe applications, the other which describes releases.
These should be super-sets of
the existing descriptions.

What should be in the manifest?

How should things be packaged?

I rather like the "put everything in a zip file (with a changed
extension) and add a top-level
manifest that describes what will be in the zip file" approach.

Again design of the manifest should strike a balance between simple
and intuitive and
"can do everything" -
(a zip file with some code that you do apply on :-) would be too
simple and dangerous)



> They mention a few very interesting issues that are likely to be brought up
> again here, and I would recommend reading it as a nice entry point to the
> discussion of what problems common solutions have, and what could possibly
> be done to fix it.
> On 12-10-19 6:36 AM, Joe Armstrong wrote:
>> Here is a programming exercise ...
>> I think we can turn rebar into a package manager with 4 simple and one not
>> so
>> simple tweaks.
>> This would turn rebar from being something that is very useful to
>> something
>> that is very very useful :-)
>> (aside - I'm revising my Erlang book, and I'd like to describe
>> how package management works, but there is no package manager
>> which is a shame - so I got to thinking about package managers
>> and wondered why still don't have a decent package manager.
>> The one I've seen that I like a lot is npm (node package manager)
>> so we could do worse than follow their design)
>> So this is how I think rebar could become a package manager
>> Tweak 1
>> =======
>>     > rebar get-deps
>> Fetches the dependencies in rebar.conf and stores them in
>> the current directory, in a directory called deps.
>> After a while you get dependencies scattered all over the place,
>> I've seen many examples of this.
>> This means that there is no central place to go and look if you
>> want to find code.
>> NPM sticks ALL dependencies under ${HOME}/.npm
>> Suggestion:
>>     All dependencies should be stored in ${HOME}/.erl_packages
>>     (Aside - if we followed npm the directory structure would be
>>      something like)
>>     ${HOME}/.erl_packages/cowboy/6.2.1/src
>>                                                                 /ebin
>>                                                       /5.2.3
>>                          /stdlib/12.3.1
>>                                 /14.6.8
>>     etc. ie a package/Vsn/src ... structure
>> Tweak 2
>> =======
>>     move rebar.config to ${HOME}/.erl_packages/rebar.config
>>     and add commands to automatically edit it
>>     > rebar add_package foo
>>     might add a line like
>>     {foo, ".*", "git://...."}
>>     to rebar.config
>>     The point here is to change the content of rebar.config with
>>     shell commands rather that editing it by hand
>> Tweak 3
>> =======
>>     fix rebar so it just downloads a particular version of a
>>     program and not all the .git stuff - most people will only
>>     want to use other peoples code, not hack it.
>> Tweak 4
>> =======
>> Fix the erlang load paths to find all the dependencies
>> I do this now. Put all my dependencies in
>> ${HOME}/nobackup/erlang_imports/deps and put the following
>> code in my .erlang startup file
>>    Home = os:getenv("HOME").
>>    Dir = Home ++ "/nobackup/erlang_imports/deps",
>>    case file:list_dir(Dir) of
>>       {ok, L} ->
>>           lists:foreach(fun(I) ->
>>                                Path = Dir ++ "/" ++ I ++ "/ebin",
>>                                code:add_path(Path)
>>                        end, L);
>>      _ ->
>>         void
>>    end.
>> Tweak 5
>> =======
>>     This can't be done by rebar
>>     Make an index of all erlang apps on github that follow
>>     the erlang packaging structure
>>     Somebody has to write a program to do this.
>>     The package index should be at a well know URL
>>     > rebar get_index
>>     This should fetch the  index from the well-know URL and store in
>>     ${HOME}/.erl_packages
>>     > rebar search XXX
>>     would search the fetched index
>>     > rebar add XXXX
>>     would take the index entry add it to the config fill fetch the code
>>     and compile it
>> Note the following:
>> 1) The trust model.
>>     We trust the supplier of a program not the program.
>>     So for example on github we might trust programs
>>     published by the user joearms (me) or erlang (the OTP release)
>>     but not by some_guy_ive_never_hear_of
>>     It would be quite easy to add a trust section to rebar.config
>>     {trust, ["git://joearms", git://erlang", ...]}
>> 2) There is no "publish" step
>>     We put our code on github (or somewhere) and hope that it gets indexed
>>     by the indexer.
>>     We might mail the indexer if it is published in an obscure place.
>>    Comments?
>>     Cheers
>> /Joe
>> _______________________________________________
>> erlang-questions mailing list
>> http://erlang.org/mailman/listinfo/erlang-questions

More information about the erlang-questions mailing list