[erlang-questions] erlc speed (or lack thereof), Make and emake
Thu Jan 24 21:07:25 CET 2013
I once wrote a custom erlc frontend for couchdb to see if I could speed up
our compilation process. It made a bit of a difference but on a decent
machine with -J the improvement wasn't enough to justify adding the
Script is here:
It was working well enough for me to run tests but I didn't spend much time
testing it for general bugs and what not.
On Thu, Jan 24, 2013 at 8:51 AM, Tim Watson <watson.timothy@REDACTED>wrote:
> Take a look at the Rabbit Makefile - it doesn't recompile all the time
> *and* it works with -J and is generally really fast.
> On 24 Jan 2013, at 14:48, Loïc Hoguin wrote:
> > On 01/24/2013 01:49 PM, Matthias Lang wrote:
> >> Hi,
> >> I'm looking for tips and tricks to use speed up compiling a bunch of
> >> Erlang modules from within a build system that uses (gnu)Make.
> >> The system uses Erlang, but most of it is actually C code, which is
> >> under four different compilers, plus FPGA code. So ditching 'make'
> >> for something Erlang-centric like 'rebar' doesn't look very attractive.
> >> Background
> >> ----------
> >> Doing this on R14B03:
> >> erlc a.erl
> >> erlc b.erl
> >> erlc c.erl
> >> ...
> >> for 70 modules on my machine takes 39s. If I do this instead:
> >> erlc a.erl b.erl c.erl ...
> >> it improves to 7s. If I split the job in two parts which run at the
> >> same time, it improves again to 4s wall time, with both cores
> >> busy.
> >> So, a 10x improvement in compilation speed seems attainable, but how do
> >> I explain how to do it to 'make'?
> >> Ideas
> >> -----
> >> I had a few simple ideas for fixing this, but none really work. See
> >> The more complicated idea I have is to make a shell script replacement
> >> for 'erlc' which delegates the compilation work to an Erlang node
> >> which is already running. Seems workable, and it won't spring
> >> surprises on 'make', e.g. -j 4 will still do the right thing.
> >> Anyone got a neat idea?
> >> Rest of this mail is just a list of things that don't seem to work.
> >> Matt
> >> ======================================================================
> >> Bad Idea #1: straightforward 'make'
> >> -----------
> >> Here's my minimal starting point for a system of three modules, a b c:
> >> beams=(addsuffix .beam, a b c)
> >> %.beam: %.erl nasty_global_stuff.hrl
> >> erlc $<
> >> That causes 'make' to hand one module to erlc at a time, which is slow.
> >> Bad Idea #2: try and use just one rule for all .beams
> >> -----------
> >> Like this:
> >> modules=a b c
> >> beams=$(addsuffix .beam, $(modules))
> >> erls=$(addsuffix .erl, $(modules))
> >> all: $(beams)
> >> $(beams): $(erls) nasty_global_stuff.hrl
> >> erlc $(filter %.erl, $@)
> >> That does almost the right thing, but actually not at all. Change one
> >> .erl and it'll recompile every .beam, because I've told 'make' that
> >> each .beam depends on all .erl files.
> >> Bad Idea #3: use erl -make for all Erlang compilations
> >> -----------
> >> all:
> >> erl -make
> >> That sort-of works, but it only runs one process (Erlang's own 'make'
> >> doesn't seem to attempt running more than one compilation at a time),
> >> passing options is not straightforward (they're not even syntactically
> >> the same as for erlc) and it requires writing 'EMakefiles', so you
> >> sort-of have to understand Erlang in order to be able to fiddle with
> >> the build system.
> >> It's also fiddly explaining dependencies between Erlang stuff and
> >> other stuff to a combination of ordinary 'make' and erlang's own make.
> >> --end of bad ideas---
> > When removing rebar dependency in the Cowboy makefile to improve my
> workflow, I ran into the same issues you did. The normal way of doing
> things with make is too damn slow. I'm assuming here that each erlc call
> starts an Erlang VM, which isn't made to be fast.
> > I settled for this:
> > https://github.com/extend/cowboy/blob/master/Makefile#L33
> > This recompiles everything all the time, but that's not really an issue
> for me. I might steal your bad idea #2 to avoid recompiling deps all the
> time though.
> > --
> > Loïc Hoguin
> > Erlang Cowboy
> > Nine Nines
> > http://ninenines.eu
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
> erlang-questions mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions