[erlang-questions] Dangers of generating a large erlang module

[Ivan Uemlianin]

Thanks for your comment.

On 29/09/2013 11:49, Gustav Simonsson wrote:
> What kind of rules are contained in the CSV files? What information is
> defined in each rule?

See reply to Anthony Ramine.

> It's probably easier parsing the rules to some Erlang data type which
> can be given to a generic function.
> The internal format of the rules could be kept in e.g. ETS or in the
> state of a gen_server.

Agreed.  However, these rules are going to be heavily consulted. 
Wouldn't gen_server and even ETS be more of a bottleneck danger?

Best wishes

Ivan


> Cheers,
> Gustav Simonsson
>
>
> On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin <ivan@REDACTED
> <mailto:ivan@REDACTED>> wrote:
>
>     Dear All
>
>     I am generating some erlang code from a csv file and using the
>     function save_compile_and_load/2 below to load it into the running
>     system.  The input csv is essentially a series of rules, so the
>     outpout erlang has roughly the same number of lines as the input has
>     lines.
>
>          save_compile_and_load(__FilePath, Text) ->
>              file:write_file(FilePath, Text),
>              {ok, ModName} = compile:file(FilePath),  % <--
>              {module, _M} = code:load_file(ModName),
>              code:purge(ModName),
>              ok.
>
>     All goes well on small test files, but the files I want to use IRL
>     are relatively large --- around 120,000 lines.
>
>     Running on a Mac (oldish, probably with 32bit emulator), compiling
>     (the line marked in the code below) causes an error and crashes the
>     VM, with several repeats of this error message:
>
>          beam.smp(1992,0xb022b000) malloc:
>            *** mmap(size=913309696) failed (error code=12)
>            *** error: can't allocate region
>            *** set a breakpoint in malloc_error_break to debug
>
>     Running on linux with (presumably) a 64bit emulator, the compile
>     works, but takes about 15 minutes.
>
>     I'm not especially bothered that it takes a long time, but I am
>     worried that this is something that could crash the whole system.
>
>     I'm looking into splitting up the code into several smaller modules,
>     but obviously I'd rather avoid crashing the VM.
>
>     Questions:
>     - is compiling large files on a 64bit emulator safe?  What effects
>     could there be on other processes?
>     - can I wrap the compile:file call in something so that it will bail
>     out if memory issues look likely (try/catch is no use)?
>
>     With thanks and best wishes
>
>     Ivan
>
>
>     --
>     ==============================__==============================
>     Ivan A. Uemlianin PhD
>     Llaisdy
>     Speech Technology Research and Development
>
>     ivan@REDACTED <mailto:ivan@REDACTED>
>     www.llaisdy.com <http://www.llaisdy.com>
>     llaisdy.wordpress.com <http://llaisdy.wordpress.com>
>     github.com/llaisdy <http://github.com/llaisdy>
>     www.linkedin.com/in/__ivanuemlianin
>     <http://www.linkedin.com/in/ivanuemlianin>
>
>                              festina lente
>     ==============================__==============================
>     _________________________________________________
>     erlang-questions mailing list
>     erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>     http://erlang.org/mailman/__listinfo/erlang-questions
>     <http://erlang.org/mailman/listinfo/erlang-questions>
>
>

-- 
============================================================
Ivan A. Uemlianin PhD
Llaisdy
Speech Technology Research and Development

                     ivan@REDACTED
                      www.llaisdy.com
                          llaisdy.wordpress.com
               github.com/llaisdy
                      www.linkedin.com/in/ivanuemlianin

                         festina lente
============================================================