[erlang-questions] Dangers of generating a large erlang module

[Ivan Uemlianin]

Dear All

I am generating some erlang code from a csv file and using the function 
save_compile_and_load/2 below to load it into the running system.  The 
input csv is essentially a series of rules, so the outpout erlang has 
roughly the same number of lines as the input has lines.

     save_compile_and_load(FilePath, Text) ->
         file:write_file(FilePath, Text),
         {ok, ModName} = compile:file(FilePath),  % <--
         {module, _M} = code:load_file(ModName),
         code:purge(ModName),
         ok.

All goes well on small test files, but the files I want to use IRL are 
relatively large --- around 120,000 lines.

Running on a Mac (oldish, probably with 32bit emulator), compiling (the 
line marked in the code below) causes an error and crashes the VM, with 
several repeats of this error message:

     beam.smp(1992,0xb022b000) malloc:
       *** mmap(size=913309696) failed (error code=12)
       *** error: can't allocate region
       *** set a breakpoint in malloc_error_break to debug

Running on linux with (presumably) a 64bit emulator, the compile works, 
but takes about 15 minutes.

I'm not especially bothered that it takes a long time, but I am worried 
that this is something that could crash the whole system.

I'm looking into splitting up the code into several smaller modules, but 
obviously I'd rather avoid crashing the VM.

Questions:
- is compiling large files on a 64bit emulator safe?  What effects could 
there be on other processes?
- can I wrap the compile:file call in something so that it will bail out 
if memory issues look likely (try/catch is no use)?

With thanks and best wishes

Ivan


-- 
============================================================
Ivan A. Uemlianin PhD
Llaisdy
Speech Technology Research and Development

                     ivan@REDACTED
                      www.llaisdy.com
                          llaisdy.wordpress.com
               github.com/llaisdy
                      www.linkedin.com/in/ivanuemlianin

                         festina lente
============================================================