[erlang-questions] Dangers of generating a large erlang module

Ivan Uemlianin ivan@REDACTED
Sun Sep 29 11:29:02 CEST 2013

Dear All

I am generating some erlang code from a csv file and using the function 
save_compile_and_load/2 below to load it into the running system.  The 
input csv is essentially a series of rules, so the outpout erlang has 
roughly the same number of lines as the input has lines.

     save_compile_and_load(FilePath, Text) ->
         file:write_file(FilePath, Text),
         {ok, ModName} = compile:file(FilePath),  % <--
         {module, _M} = code:load_file(ModName),

All goes well on small test files, but the files I want to use IRL are 
relatively large --- around 120,000 lines.

Running on a Mac (oldish, probably with 32bit emulator), compiling (the 
line marked in the code below) causes an error and crashes the VM, with 
several repeats of this error message:

     beam.smp(1992,0xb022b000) malloc:
       *** mmap(size=913309696) failed (error code=12)
       *** error: can't allocate region
       *** set a breakpoint in malloc_error_break to debug

Running on linux with (presumably) a 64bit emulator, the compile works, 
but takes about 15 minutes.

I'm not especially bothered that it takes a long time, but I am worried 
that this is something that could crash the whole system.

I'm looking into splitting up the code into several smaller modules, but 
obviously I'd rather avoid crashing the VM.

- is compiling large files on a 64bit emulator safe?  What effects could 
there be on other processes?
- can I wrap the compile:file call in something so that it will bail out 
if memory issues look likely (try/catch is no use)?

With thanks and best wishes


Ivan A. Uemlianin PhD
Speech Technology Research and Development


                         festina lente

More information about the erlang-questions mailing list