[erlang-questions] how do you build your releases with/without rebar?

[Yuri Lukyanov]

On Thu, Apr 17, 2014 at 10:57 AM, Garrett Smith <g@REDACTED> wrote:
> On Wed, Apr 16, 2014 at 2:03 PM, Yuri Lukyanov <snaky@REDACTED> wrote:
>>
>> Well, I can add our story.
>
> Thanks for this!
>
>> In the beginning, we had a single repository for everything. And even
>> without apps dir. It was a single monster with everything inside the
>> src dir. At some point, it became obvious that it couldn't go on like
>> this and we started separating the code into apps. 'apps' directory
>> looked like a good fit.
>>
>> Then we came across Riak and its sources. The way they organised their
>> code seemed so gorgeous. A tiny repository for each application! Super
>> easy to think of as a separate unit of your system. Different teams
>> work on different repositories. Independent testing. Ability to open
>> certain parts of your project to the community. Superb!
>>
>> And we started a 'big move' to that world. At the beginning,
>> everything was smooth. Later, we even created a project template that
>> uses that every-app-is-a-repo structure
>> (https://github.com/EchoTeam/rebar-templates).
>>
>> But what we ended up with is what Björn-Egil Dahlberg said above. The
>> team is divided. Some of us just couldn't stand the growing number of
>> repositories. The tree of dependencies started to be so big that it
>> was a nightmare to change anything inside. Imagine that you need to
>> update, say, lager version. We heavily use lager in all deps. You will
>> need to update the deps list of one application, then the deps list of
>> the application which is dependant on the first one, then 10
>> application which depend on the latter, etc...
>
> This is very interesting to me as I'm experimenting with the
> one-repo-per-app model, having done the apps-under-libdir thing.
>
> Have you experimented with tooling/automation to help manage the
> dependencies across the various applications. E.g. a centralized list
> of of company "standard" apps + versions that the various app/repos
> pulled from?


We extensively use rebar_lock_deps_plugin:
https://github.com/seth/rebar_lock_deps_plugin
The main purpose of it is to fix all repos at exact commits in a
single root rebar.config.lock.
It helps to have repeatable builds at any point in time.
And in fact, you don't even need to have other dep-specific
rebar.config's. Though this would ruin one of the main purpose of
separating your code into different repositories - having independent
working self-sufficient pieces of code.




>
> In a case where I need to standardize on a version of something, as a
> matter of keeping things simple, I will tend to fork that repository
> to a location I can control and then refer to that from each deps
> list.

Yes, that could be one of the options. We are considering this approach too.
But the thing is that you will need to fork the whole dependency tree
of a particular project/library.
Consider forking https://github.com/basho/riak_kv, as an example.
There are 8 deps in it. Each dep, in turn, can have about the same
number of deps, and so forth. You will end up forking several dozens
of applications, and it's only for one nearest dependency that you
wanted to attach to your project! And it's only a beginning. Now you
need to change _all_ dependencies list (i.e. rebar.config's) in all
deps to point to your locations.
Well, you can say that you have to do this only once. But what if you
want regularly update some of you deps? You will need merging from the
original locations in which the authors will also update their code
and rebar.config's. Considering the amount of deps you have now, you
may have tons of conflicts in dependency management.

Not mentioning, that you will still have to use something like
rebar_lock_deps as you want to have a linear history of changes in
your main project.


So, I think, we all need a new tool for dependency management :) Rebar
and erlang.mk, though doing very useful job at some point, do not
scale well. This is a huge subject on its own and people talk about it
from time to time. One of the most thorough words IMO on the subject
is this blog post by hyperthunk:
http://hyperthunk.wordpress.com/2012/05/28/does-erlangotp-need-a-new-package-management-solution/

Having a good dependency manager is not solely about Erlang. Having a
single approach for all languages is long overdue :) They all have
similar problems and all go their own ways. But I probably walked away
from the main topic, sorry :)


> I happen to use git (both in github and private locations) but
> any protocol supported by the build tool would work. When I need to
> "rev" that dependency (bug fixes, etc.) I can just selectively
> pull/merge the applicable commit or tag.
>
> Anyway, I'm not trying to solve your team's problems here :) I'm
> curious though about the specific pain points and am wondering if
> there are any good alternatives to address them, short of
> consolidating everything into a single repo.
>
>> At the moment, we are still having different repos for different apps.
>> But some people are already thinking to put _everything_ in one single
>> repository. Not only all deps, but all projects as well :)
>>
>> There is no one-size-fits-all solution. Right, "In the long run,
>> everything fails".
>> Just don't be too religious about things. If you think that either
>> approach is _the_ best, perhaps you just don't have use cases that can
>> help your approach fail or your project is not big/small enough.
>
> I don't see a religious debate emerging here -- I'm encouraged that
> there are options and support for different org structures and
> workflows.
>
> One of Erlang's most compelling features IMO is the app + release
> abstraction -- this is a remarkably enlightened application design. It
> mirrors what happens at the operating system level. And just as
> operating system ecosystems are supported by the one-repo-per-app
> model -- which works, clearly -- I suspect this model can be made to
> work well for Erlang projects. But I have limited experience with
> complex projects -- so specific data points like yours are very
> helpful.
>
> Thanks!
>
> Garrett