[erlang-questions] erlc speed (or lack thereof), Make and emake

[Paul Davis]

I once wrote a custom erlc frontend for couchdb to see if I could speed up
our compilation process. It made a bit of a difference but on a decent
machine with -J the improvement wasn't enough to justify adding the
complexity.

Script is here:

https://github.com/davisp/couchdb/blob/couch_erlc/bin/couch_erlc

It was working well enough for me to run tests but I didn't spend much time
testing it for general bugs and what not.


On Thu, Jan 24, 2013 at 8:51 AM, Tim Watson <watson.timothy@REDACTED>wrote:

> Take a look at the Rabbit Makefile - it doesn't recompile all the time
> *and* it works with -J and is generally really fast.
>
> http://hg.rabbitmq.com/rabbitmq-server/file/default/
>
> Cheers,
> Tim
>
> On 24 Jan 2013, at 14:48, Loïc Hoguin wrote:
>
> > On 01/24/2013 01:49 PM, Matthias Lang wrote:
> >> Hi,
> >>
> >> I'm looking for tips and tricks to use speed up compiling a bunch of
> >> Erlang modules from within a build system that uses (gnu)Make.
> >>
> >> The system uses Erlang, but most of it is actually C code, which is
> compiled
> >> under four different compilers, plus FPGA code. So ditching 'make'
> >> for something Erlang-centric like 'rebar' doesn't look very attractive.
> >>
> >> Background
> >> ----------
> >>
> >> Doing this on R14B03:
> >>
> >>   erlc a.erl
> >>   erlc b.erl
> >>   erlc c.erl
> >>   ...
> >>
> >> for 70 modules on my machine takes 39s. If I do this instead:
> >>
> >>   erlc a.erl b.erl c.erl ...
> >>
> >> it improves to 7s. If I split the job in two parts which run at the
> >> same time, it improves again to 4s wall time, with both cores
> >> busy.
> >>
> >> So, a 10x improvement in compilation speed seems attainable, but how do
> >> I explain how to do it to 'make'?
> >>
> >>
> >> Ideas
> >> -----
> >>
> >> I had a few simple ideas for fixing this, but none really work. See
> below.
> >>
> >> The more complicated idea I have is to make a shell script replacement
> >> for 'erlc' which delegates the compilation work to an Erlang node
> >> which is already running. Seems workable, and it won't spring
> >> surprises on 'make', e.g. -j 4 will still do the right thing.
> >>
> >> Anyone got a neat idea?
> >>
> >> Rest of this mail is just a list of things that don't seem to work.
> >>
> >> Matt
> >>
> >> ======================================================================
> >>
> >>
> >> Bad Idea #1: straightforward 'make'
> >> -----------
> >>
> >> Here's my minimal starting point for a system of three modules, a b c:
> >>
> >>    beams=(addsuffix .beam, a b c)
> >>    %.beam: %.erl nasty_global_stuff.hrl
> >>         erlc $<
> >>
> >> That causes 'make' to hand one module to erlc at a time, which is slow.
> >>
> >>
> >> Bad Idea #2: try and use just one rule for all .beams
> >> -----------
> >>
> >> Like this:
> >>
> >>   modules=a b c
> >>   beams=$(addsuffix .beam, $(modules))
> >>   erls=$(addsuffix .erl, $(modules))
> >>   all: $(beams)
> >>
> >>   $(beams): $(erls) nasty_global_stuff.hrl
> >>         erlc $(filter %.erl, $@)
> >>
> >> That does almost the right thing, but actually not at all. Change one
> >> .erl and it'll recompile every .beam, because I've told 'make' that
> >> each .beam depends on all .erl files.
> >>
> >>
> >> Bad Idea #3: use erl -make for all Erlang compilations
> >> -----------
> >>
> >>   all:
> >>      erl -make
> >>
> >> That sort-of works, but it only runs one process (Erlang's own 'make'
> >> doesn't seem to attempt running more than one compilation at a time),
> >> passing options is not straightforward (they're not even syntactically
> >> the same as for erlc) and it requires writing 'EMakefiles', so you
> >> sort-of have to understand Erlang in order to be able to fiddle with
> >> the build system.
> >>
> >> It's also fiddly explaining dependencies between Erlang stuff and
> >> other stuff to a combination of ordinary 'make' and erlang's own make.
> >>
> >> --end of bad ideas---
> >
> > When removing rebar dependency in the Cowboy makefile to improve my
> workflow, I ran into the same issues you did. The normal way of doing
> things with make is too damn slow. I'm assuming here that each erlc call
> starts an Erlang VM, which isn't made to be fast.
> >
> > I settled for this:
> >  https://github.com/extend/cowboy/blob/master/Makefile#L33
> >
> > This recompiles everything all the time, but that's not really an issue
> for me. I might steal your bad idea #2 to avoid recompiling deps all the
> time though.
> >
> > --
> > Loïc Hoguin
> > Erlang Cowboy
> > Nine Nines
> > http://ninenines.eu
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130124/a625e03b/attachment.htm>