[erlang-questions] erlc speed (or lack thereof), Make and emake

[Florian Schintke]

Hi,

we use the Erlang module 'make' and an Emakefile to specify what and
how to compile if necessary.
See http://erldocs.com/R14B04/tools/make.html

Our Unix makefile entry looks as follows:

---
compile:
        @$(ERL) -pa ebin -noinput +B -eval 'case make:all() of up_to_date -> halt(0); error -> halt(1) end.'
---

It starts Erlang once and compiles all touched/depending modules,
which is really fast. Parallel make is missing, but not necessary at
that speed.

See scalaris.googlecode.com for a sample Emakefile.

Florian

[Matthias Lang]
> Hi,
> 
> I'm looking for tips and tricks to use speed up compiling a bunch of
> Erlang modules from within a build system that uses (gnu)Make.
> 
> The system uses Erlang, but most of it is actually C code, which is compiled
> under four different compilers, plus FPGA code. So ditching 'make'
> for something Erlang-centric like 'rebar' doesn't look very attractive.
> 
> Background
> ----------
> 
> Doing this on R14B03:
> 
>   erlc a.erl
>   erlc b.erl
>   erlc c.erl
>   ...
> 
> for 70 modules on my machine takes 39s. If I do this instead:
> 
>   erlc a.erl b.erl c.erl ...
> 
> it improves to 7s. If I split the job in two parts which run at the
> same time, it improves again to 4s wall time, with both cores
> busy.
> 
> So, a 10x improvement in compilation speed seems attainable, but how do
> I explain how to do it to 'make'?
> 
> 
> Ideas
> -----
> 
> I had a few simple ideas for fixing this, but none really work. See below.
> 
> The more complicated idea I have is to make a shell script replacement
> for 'erlc' which delegates the compilation work to an Erlang node
> which is already running. Seems workable, and it won't spring
> surprises on 'make', e.g. -j 4 will still do the right thing.
> 
> Anyone got a neat idea?
> 
> Rest of this mail is just a list of things that don't seem to work.
> 
> Matt
> 
> ======================================================================
> 
> 
> Bad Idea #1: straightforward 'make'
> -----------
> 
> Here's my minimal starting point for a system of three modules, a b c:
> 
>    beams=(addsuffix .beam, a b c)
>    %.beam: %.erl nasty_global_stuff.hrl
>    	   erlc $<
> 
> That causes 'make' to hand one module to erlc at a time, which is slow.
> 
> 
> Bad Idea #2: try and use just one rule for all .beams
> -----------
> 
> Like this:
> 
>   modules=a b c
>   beams=$(addsuffix .beam, $(modules))
>   erls=$(addsuffix .erl, $(modules))
>   all: $(beams)
> 
>   $(beams): $(erls) nasty_global_stuff.hrl
>         erlc $(filter %.erl, $@)
> 
> That does almost the right thing, but actually not at all. Change one
> .erl and it'll recompile every .beam, because I've told 'make' that
> each .beam depends on all .erl files.
> 
> 
> Bad Idea #3: use erl -make for all Erlang compilations
> -----------
> 
>   all:
>      erl -make
> 
> That sort-of works, but it only runs one process (Erlang's own 'make'
> doesn't seem to attempt running more than one compilation at a time),
> passing options is not straightforward (they're not even syntactically
> the same as for erlc) and it requires writing 'EMakefiles', so you
> sort-of have to understand Erlang in order to be able to fiddle with
> the build system.
> 
> It's also fiddly explaining dependencies between Erlang stuff and
> other stuff to a combination of ordinary 'make' and erlang's own make.
> 
> --end of bad ideas---