[erlang-questions] Building, Packaging and Installing

[Tim Watson]

Joe this is exactly what we need, and I hope this gains some traction. A few comments inline.

On 7 May 2012, at 08:37, Joe Armstrong <erlang@REDACTED> wrote:

> Hello,
> 
> I've been watching this thread - but not contributing ...
> 
> I have a few questions/thoughts/comments
> 
> 1) NodeJSs NPM looks pretty good to me. I've been toying with the idea
> of making a clone
>    the advantages of cloning seem to be:
>    - it is very well documented
>    - it has a decent set of commands
>    - it produces a reasonable (target) directory structure
> 
> 2) What's wrong today?
> 
>     It seems to me that the single biggest problem is the lack of a decent
>     manifest file.
> 
>     NodeJS uses
>             https://github.com/isaacs/npm/blob/master/package.json
>     For comparison Google chrome apps use
>            http://code.google.com/chrome/extensions/manifest.html
>     We (Erlang) use
>            http://www.erlang.org/doc/design_principles/applications.html#id73480
> 
>      In a sense once you *defined* the manifest file and the file
> system layout, all the rest
>      follows naturally. (Once you've agreed on the data formats -
> which program produces
>      the files is irrelevant)
> 
>      Now the .app files date from a time *before* Erlang was open
> source, so there
>      are no things for licenses, digital signatures and so on.
> 
>      So I think a first step is hammer out the content of the manifest.
> 

+1

>       The manifest could be simple or very complicated - so there are
> a lot of design questions
>       here.
> 
> 3) Tricky problems
> 
>        How do we avoid version nightmares?
> 
>        i.e. if A depends upon X.vsn1 and breaks with X.vsn2
>             and B depends upon X.vsn2 and breaks with X.vsn1
>        how can we run A and B in the same namespace?
> 
>      Three answers:
> 
>         a) You can't
>         b) Declare A to be broken and fix it (assuming X.vsn2 is the
> latest and best version of
>              X)
>         c) do some magic name munging (refactoring)
>              that allows A and B to run in the same namspace with
> X.vsn1 and X.vsn2
>         d) run in separate nodes with message passing
>         e) add some sort of namespaces to erlang
> 
>       Comments:
> 
>               c) is pie in the sky - I don't see this happening realistically
> 
>               d) yes - but the apps need changing
> 
>               e) is research
> 
>               a) is bad
> 
>               b) is the best option - fix A - now why can't the
> refactoring guys do this for us :-)
> 
>            In practice I'd mail the authors of A and hope they could help
> 

I agree with this. I think that it would also be really useful if I could fork the project, change the dependency and re-package it signed under my packaging id. Eric Merrit and I have talking about this as a kind of third party signing.

> 4) Signing
> 
>      No we want digital signatures - bla bla bla .... signed code
> 

Yep 

> 
> 5) Should packages be unpacked on the target machines
> 
>     (I think not - this is abstraction breaking)
> 
>     The way rebar is packaged is nice.
> 

Yes but not all stuff can be done like that. If we fix the little problems with .ez files then they're probably ideal for most cases.

> 6) The way forward
> 
>     - Define a new manifest (say erlang .app files with inspiration from NPM)
>     - Define the layout of the local repository (ie where things land
> locally after
>        a package has been fetched)
>    - Make a clone of NPM (or similar)
>    - Make a converter that sucks code from rebar github repository
>       and repackages, throwing away the history inn (.git)
> 

You also need to support a notion of two or more publishers for apps win the same name and version, in order to properly support the kind of forking we see in many open source packages. I think I this as the 'publisher'

> 7) Binary or source packages?
> 
>    - Ummm - tricky
> 

I think binary is better but adds a lot of complexity. Erts versions, projects containing native code, etc. makin this the publishers responsibility means the tool only has to focus on the packaging problem not the build space. Third party  signing allows you to support sources which gaven't been published by the originator.

On the other hand, being able to only build conforming projects makes edge cases awkward when using source packages and whilst consistency and uniformity are good, the real world Ian's always like that.

> 8) Community buy-in
> 
>    We need one recognised format that everybody uses.
> 
> I'd actually proposed a NPM clone as the subject of a hackathon at the
> next Erlang
> User conference (in may) - now (of course) this is not a particularly difficult
> program to write (famous last words) - the tricky bit is agreeing on the
> manifest and file layout and semantics of updates etc.
> 
> If any of you guys are coming to the Erlang users conference in
> Stockholm I'd be happy to
> discuss this - or even participate in a hackathon session
> 
> 
> Cheers
> 
> /Joe
> 
> 
> 
> 
> 
> 
> 
> 
> On Wed, May 2, 2012 at 8:39 PM, Ciprian Dorin Craciun
> <ciprian.craciun@REDACTED> wrote:
>>    (As no-one replied in 4 days, I'll add my opinion...)
>> 
>> On Sat, Apr 28, 2012 at 4:36 PM, Tristan Sloughter
>> <tristan.sloughter@REDACTED> wrote:
>>> I started with this problem as something to simply discuss with the
>>> maintainers of certain build and package management projects -- sinan and
>>> agner specifically. But it seems to be more broad and cover how all of us
>>> who keep apps on github handle versioning and dependencies.
>>> 
>>> The basic issue is app versions within .app files on branches in github, the
>>> resultant directory from a agner install and the discrepancies this causes.
>>> 
>>> [...]
>>> 
>>> Am I the only one going crazy with this?
>> 
>>    Nop. I've been faced with this problem myself a couple of times.
>> (I've chosen to ignore it.)
>> 
>> 
>>> Does anyone have
>>> suggestions/examples of what they do for production projects on teams to
>>> ease these annoyances?
>> 
>>    For example for my project I have one big Git repository called
>> `myproject-repositories` (replace `myproject` with what you need) with
>> submodules pointing to the dependencies.
>> 
>>    Then inside my own project repositories I have a symlink
>> `.repositories` poiting to the "current" snapshot of dependencies.
>> 
>>    I also don't use any of rebar, agner, etc., mainly because I've
>> hacked something on-top of the ninja build system.
>> 
>> 
>>> I mostly just create my own packages and repos of dependencies, or package
>>> third party deps with the project.
>> 
>>    Yup. +1 :)
>> 
>> 
>>    P.S.: I hate Java from the bottom of my heart, but boy-o-boy is
>> the Java tooling way better than that of Erlang... I mean what Erlang
>> is missing is something similar to NodeJS's NPM, or Java's Maven
>> (minus the XML baggage and all useless complexities), or Python
>> setup-tools (with virtual environment), etc...
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions