[erlang-questions] Dangers of generating a large erlang module

[Gustav Simonsson]

What kind of rules are contained in the CSV files? What information is
defined in each rule?

It's probably easier parsing the rules to some Erlang data type which can
be given to a generic function.
The internal format of the rules could be kept in e.g. ETS or in the state
of a gen_server.

Cheers,
Gustav Simonsson


On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin <ivan@REDACTED> wrote:

> Dear All
>
> I am generating some erlang code from a csv file and using the function
> save_compile_and_load/2 below to load it into the running system.  The
> input csv is essentially a series of rules, so the outpout erlang has
> roughly the same number of lines as the input has lines.
>
>     save_compile_and_load(**FilePath, Text) ->
>         file:write_file(FilePath, Text),
>         {ok, ModName} = compile:file(FilePath),  % <--
>         {module, _M} = code:load_file(ModName),
>         code:purge(ModName),
>         ok.
>
> All goes well on small test files, but the files I want to use IRL are
> relatively large --- around 120,000 lines.
>
> Running on a Mac (oldish, probably with 32bit emulator), compiling (the
> line marked in the code below) causes an error and crashes the VM, with
> several repeats of this error message:
>
>     beam.smp(1992,0xb022b000) malloc:
>       *** mmap(size=913309696) failed (error code=12)
>       *** error: can't allocate region
>       *** set a breakpoint in malloc_error_break to debug
>
> Running on linux with (presumably) a 64bit emulator, the compile works,
> but takes about 15 minutes.
>
> I'm not especially bothered that it takes a long time, but I am worried
> that this is something that could crash the whole system.
>
> I'm looking into splitting up the code into several smaller modules, but
> obviously I'd rather avoid crashing the VM.
>
> Questions:
> - is compiling large files on a 64bit emulator safe?  What effects could
> there be on other processes?
> - can I wrap the compile:file call in something so that it will bail out
> if memory issues look likely (try/catch is no use)?
>
> With thanks and best wishes
>
> Ivan
>
>
> --
> ==============================**==============================
> Ivan A. Uemlianin PhD
> Llaisdy
> Speech Technology Research and Development
>
>                     ivan@REDACTED
>                      www.llaisdy.com
>                          llaisdy.wordpress.com
>               github.com/llaisdy
>                      www.linkedin.com/in/**ivanuemlianin<http://www.linkedin.com/in/ivanuemlianin>
>
>                         festina lente
> ==============================**==============================
> ______________________________**_________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/**listinfo/erlang-questions<http://erlang.org/mailman/listinfo/erlang-questions>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130929/02ab3420/attachment.htm>