[erlang-questions] Turning rebar into a package manager
Joe Armstrong
erlang@REDACTED
Tue Oct 23 09:46:18 CEST 2012
On Fri, Oct 19, 2012 at 2:33 PM, Fred Hebert <mononcqc@REDACTED> wrote:
> For anyone interested, there's been an interesting blog post on the topic of
> package managers written at
> http://hyperthunk.wordpress.com/2012/05/28/does-erlangotp-need-a-new-package-management-solution/
That's very interesting. Thanks for the reference
I got to thinking a bit more. ...
The above link has an example command
> git fetch-pack --include-tag -v git@REDACTED:hyperthunk/gitfoo.git refs/tags/lager-0.9.4
This appears to fetch a tagged tree without all all other versions -
just what I wanted
At some level of abstraction we need to tag versions of applications.
Using the latest
version is fine for development but not for shipping a product.
If we adopt a common tagging convention then we can do two things.
1) express dependencies (ie foo-1.4.5 depends upon bar-3.2.6)
and
2) make a "release" ie say that a certain set of tagged
applications has been tested and is known
to work together.
At the moment I guess very few Erlang git projects made by rebar have
vI.J.K style tags.
Two more problems remain:
A) Weakening the dependencies
I know what foo-3.4.2 depends upon bar-4.2.5 means - this is
very precise
Should we allow foo-3.* depends upon bar-4.*
There must be some language of dependencies, and conventions
(ie odd major version numbers are experimental,
even stable)
Does anybody have any pointers to a discussion of this.
I'd like a few simple rules that cover the common use cases
rather than fancy rules that cover everything
B) The *one* module name problem
We can't load modules in foo-1.2.3 and foo-2.1.6 into the
same node so if A depends upon
foo-1.2.3 and B depends upon foo-2.1.6 we can't run A and B
in the same node
There are a number of solutions possible:
i) name_munging -> convert xx.erl to xx_foo_1.2.3.erl
It *is* possible - but you have to take care of registered
process names etc. there are a lot of
edge cases that make this tricky.
ii) Don't solve the problem at all. Run A and B in different nodes
My feeling is ii) is correct - we could then define a node as
a release (ie a set of tagged applications)
that are know to work together.
I think problems of /consistency of
modules/security/authentication should be solved at the node level
and not the process/module level
How could one make a release?
a) All contributors must tag versions of their applications and
follow common packaging conventions
b) we need an improved manifest
Maybe volunteers could make releases - actually a release is little
more than a set of applications
that are know to.
Seems we also need to standardize a new an improved manifest file - we
need two types
one that describe applications, the other which describes releases.
These should be super-sets of
the existing descriptions.
What should be in the manifest?
How should things be packaged?
I rather like the "put everything in a zip file (with a changed
extension) and add a top-level
manifest that describes what will be in the zip file" approach.
Again design of the manifest should strike a balance between simple
and intuitive and
"can do everything" -
(a zip file with some code that you do apply on :-) would be too
simple and dangerous)
Cheers
/Joe
>
> They mention a few very interesting issues that are likely to be brought up
> again here, and I would recommend reading it as a nice entry point to the
> discussion of what problems common solutions have, and what could possibly
> be done to fix it.
>
>
> On 12-10-19 6:36 AM, Joe Armstrong wrote:
>>
>> Here is a programming exercise ...
>>
>> I think we can turn rebar into a package manager with 4 simple and one not
>> so
>> simple tweaks.
>>
>> This would turn rebar from being something that is very useful to
>> something
>> that is very very useful :-)
>>
>>
>> (aside - I'm revising my Erlang book, and I'd like to describe
>> how package management works, but there is no package manager
>> which is a shame - so I got to thinking about package managers
>> and wondered why still don't have a decent package manager.
>>
>> The one I've seen that I like a lot is npm (node package manager)
>> so we could do worse than follow their design)
>>
>> So this is how I think rebar could become a package manager
>>
>>
>> Tweak 1
>> =======
>>
>> > rebar get-deps
>>
>> Fetches the dependencies in rebar.conf and stores them in
>> the current directory, in a directory called deps.
>>
>> After a while you get dependencies scattered all over the place,
>> I've seen many examples of this.
>>
>> This means that there is no central place to go and look if you
>> want to find code.
>>
>> NPM sticks ALL dependencies under ${HOME}/.npm
>>
>> Suggestion:
>>
>> All dependencies should be stored in ${HOME}/.erl_packages
>>
>> (Aside - if we followed npm the directory structure would be
>> something like)
>>
>> ${HOME}/.erl_packages/cowboy/6.2.1/src
>> /ebin
>> /5.2.3
>> /stdlib/12.3.1
>> /14.6.8
>>
>> etc. ie a package/Vsn/src ... structure
>>
>>
>> Tweak 2
>> =======
>>
>> move rebar.config to ${HOME}/.erl_packages/rebar.config
>> and add commands to automatically edit it
>>
>> > rebar add_package foo
>>
>> might add a line like
>> {foo, ".*", "git://...."}
>>
>> to rebar.config
>>
>> The point here is to change the content of rebar.config with
>> shell commands rather that editing it by hand
>>
>> Tweak 3
>> =======
>> fix rebar so it just downloads a particular version of a
>> program and not all the .git stuff - most people will only
>> want to use other peoples code, not hack it.
>>
>> Tweak 4
>> =======
>>
>> Fix the erlang load paths to find all the dependencies
>>
>> I do this now. Put all my dependencies in
>> ${HOME}/nobackup/erlang_imports/deps and put the following
>> code in my .erlang startup file
>>
>> Home = os:getenv("HOME").
>> Dir = Home ++ "/nobackup/erlang_imports/deps",
>> case file:list_dir(Dir) of
>> {ok, L} ->
>> lists:foreach(fun(I) ->
>> Path = Dir ++ "/" ++ I ++ "/ebin",
>> code:add_path(Path)
>> end, L);
>> _ ->
>> void
>> end.
>>
>> Tweak 5
>> =======
>> This can't be done by rebar
>> Make an index of all erlang apps on github that follow
>> the erlang packaging structure
>>
>> Somebody has to write a program to do this.
>>
>> The package index should be at a well know URL
>>
>> > rebar get_index
>>
>>
>> This should fetch the index from the well-know URL and store in
>> ${HOME}/.erl_packages
>>
>>
>> > rebar search XXX
>>
>> would search the fetched index
>>
>> > rebar add XXXX
>>
>> would take the index entry add it to the config fill fetch the code
>> and compile it
>>
>> Note the following:
>>
>> 1) The trust model.
>>
>> We trust the supplier of a program not the program.
>>
>> So for example on github we might trust programs
>> published by the user joearms (me) or erlang (the OTP release)
>> but not by some_guy_ive_never_hear_of
>>
>> It would be quite easy to add a trust section to rebar.config
>>
>> {trust, ["git://joearms", git://erlang", ...]}
>>
>> 2) There is no "publish" step
>>
>> We put our code on github (or somewhere) and hope that it gets indexed
>> by the indexer.
>>
>> We might mail the indexer if it is published in an obscure place.
>>
>> Comments?
>>
>> Cheers
>>
>> /Joe
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
More information about the erlang-questions
mailing list