[erlang-questions] erlc speed (or lack thereof), Make and emake

[Matthias Lang]

On Thursday, January 24, Tim Watson wrote:

> Take a look at the Rabbit Makefile - it doesn't recompile all the
> time *and* it works with -J and is generally really fast.

That Makefile is nice and tidy, but it's compiling the .erl files in
the simplest---and slowest---way possible. It calls 'erlc' about 370
times, i.e. once per file. I.e. exactly as per "Bad Idea #1" in my OP.

Quick measurements on building 3.0.1 from source:

  plain 'make': 220s wall time
   'make -j 2': 112s

I wanted to get an idea of how much of a speedup was possible, so I
broke out all the calls to 'erlc' from the build process and put
them in a shell script. There are about 370.

Running the script so it does one call to 'erlc' at a time takes 183s.

Changing the script so that each call to 'erlc' compiles many files
drops the time to 40s.

Further changing the script to use multiple (unix) processes gets me to 26s.

That's pretty encouraging. It suggests that a 3x or 4x speedup is
possible for the Erlang part of the Rabbit build process.

Matt