[erlang-questions] erlc speed (or lack thereof), Make and emake

[Erik Søe Sørensen]

One possibility is to let the shell figure out which files need to be
recompiled, and invoke erlc on that batch.
Figuring out what needs recompilation could possibly be done by calling
make in dry-run mode, to see what it thinks needs to be done, and then
processing that output - or, as below, in a more direct manner.
I don't know if all the features necessary to do this are 100% portable,
but sticking with (gmake +) bash, this works:

# Iterating through the source files, determine which need recompilation.
# Then - if any do - invoke the compiler on the batch.
fast-compile:
files=""; for i in src/*.erl ; do j=$${i%.erl}.beam; j=ebin/$${j#src/} ; [
"$$j" -nt "$$i" ] || files="$$files $$i" ; done ; \
[ -z "$$files" ] || erlc -o ebin $$files

/Erik

2013/1/24 Matthias Lang <matthias@REDACTED>

> Hi,
>
> I'm looking for tips and tricks to use speed up compiling a bunch of
> Erlang modules from within a build system that uses (gnu)Make.
>
> The system uses Erlang, but most of it is actually C code, which is
> compiled
> under four different compilers, plus FPGA code. So ditching 'make'
> for something Erlang-centric like 'rebar' doesn't look very attractive.
>
> Background
> ----------
>
> Doing this on R14B03:
>
>   erlc a.erl
>   erlc b.erl
>   erlc c.erl
>   ...
>
> for 70 modules on my machine takes 39s. If I do this instead:
>
>   erlc a.erl b.erl c.erl ...
>
> it improves to 7s. If I split the job in two parts which run at the
> same time, it improves again to 4s wall time, with both cores
> busy.
>
> So, a 10x improvement in compilation speed seems attainable, but how do
> I explain how to do it to 'make'?
>
>
> Ideas
> -----
>
> I had a few simple ideas for fixing this, but none really work. See below.
>
> The more complicated idea I have is to make a shell script replacement
> for 'erlc' which delegates the compilation work to an Erlang node
> which is already running. Seems workable, and it won't spring
> surprises on 'make', e.g. -j 4 will still do the right thing.
>
> Anyone got a neat idea?
>
> Rest of this mail is just a list of things that don't seem to work.
>
> Matt
>
> ======================================================================
>
>
> Bad Idea #1: straightforward 'make'
> -----------
>
> Here's my minimal starting point for a system of three modules, a b c:
>
>    beams=(addsuffix .beam, a b c)
>    %.beam: %.erl nasty_global_stuff.hrl
>            erlc $<
>
> That causes 'make' to hand one module to erlc at a time, which is slow.
>
>
> Bad Idea #2: try and use just one rule for all .beams
> -----------
>
> Like this:
>
>   modules=a b c
>   beams=$(addsuffix .beam, $(modules))
>   erls=$(addsuffix .erl, $(modules))
>   all: $(beams)
>
>   $(beams): $(erls) nasty_global_stuff.hrl
>         erlc $(filter %.erl, $@)
>
> That does almost the right thing, but actually not at all. Change one
> .erl and it'll recompile every .beam, because I've told 'make' that
> each .beam depends on all .erl files.
>
>
> Bad Idea #3: use erl -make for all Erlang compilations
> -----------
>
>   all:
>      erl -make
>
> That sort-of works, but it only runs one process (Erlang's own 'make'
> doesn't seem to attempt running more than one compilation at a time),
> passing options is not straightforward (they're not even syntactically
> the same as for erlc) and it requires writing 'EMakefiles', so you
> sort-of have to understand Erlang in order to be able to fiddle with
> the build system.
>
> It's also fiddly explaining dependencies between Erlang stuff and
> other stuff to a combination of ordinary 'make' and erlang's own make.
>
> --end of bad ideas---
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130125/de296220/attachment.htm>