[erlang-questions] Dangers of generating a large erlang module

[Ivan Uemlianin]

Thanks, that sounds worth a go.

Generating the code into smaller modules was starting to get hacky :P

Best wishes

Ivan


On 29/09/2013 12:16, Sergej Jurecko wrote:
> Try a public ets table with read_concurrency true.
>
>
> Sergej
>
> On Sep 29, 2013, at 12:53 PM, Ivan Uemlianin wrote:
>
>> Thanks for your comment.
>>
>> On 29/09/2013 11:49, Gustav Simonsson wrote:
>>> What kind of rules are contained in the CSV files? What information is
>>> defined in each rule?
>>
>> See reply to Anthony Ramine.
>>
>>> It's probably easier parsing the rules to some Erlang data type which
>>> can be given to a generic function.
>>> The internal format of the rules could be kept in e.g. ETS or in the
>>> state of a gen_server.
>>
>> Agreed.  However, these rules are going to be heavily consulted. Wouldn't gen_server and even ETS be more of a bottleneck danger?
>>
>> Best wishes
>>
>> Ivan
>>
>>
>>> Cheers,
>>> Gustav Simonsson
>>>
>>>
>>> On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin <ivan@REDACTED
>>> <mailto:ivan@REDACTED>> wrote:
>>>
>>>     Dear All
>>>
>>>     I am generating some erlang code from a csv file and using the
>>>     function save_compile_and_load/2 below to load it into the running
>>>     system.  The input csv is essentially a series of rules, so the
>>>     outpout erlang has roughly the same number of lines as the input has
>>>     lines.
>>>
>>>          save_compile_and_load(__FilePath, Text) ->
>>>              file:write_file(FilePath, Text),
>>>              {ok, ModName} = compile:file(FilePath),  % <--
>>>              {module, _M} = code:load_file(ModName),
>>>              code:purge(ModName),
>>>              ok.
>>>
>>>     All goes well on small test files, but the files I want to use IRL
>>>     are relatively large --- around 120,000 lines.
>>>
>>>     Running on a Mac (oldish, probably with 32bit emulator), compiling
>>>     (the line marked in the code below) causes an error and crashes the
>>>     VM, with several repeats of this error message:
>>>
>>>          beam.smp(1992,0xb022b000) malloc:
>>>            *** mmap(size=913309696) failed (error code=12)
>>>            *** error: can't allocate region
>>>            *** set a breakpoint in malloc_error_break to debug
>>>
>>>     Running on linux with (presumably) a 64bit emulator, the compile
>>>     works, but takes about 15 minutes.
>>>
>>>     I'm not especially bothered that it takes a long time, but I am
>>>     worried that this is something that could crash the whole system.
>>>
>>>     I'm looking into splitting up the code into several smaller modules,
>>>     but obviously I'd rather avoid crashing the VM.
>>>
>>>     Questions:
>>>     - is compiling large files on a 64bit emulator safe?  What effects
>>>     could there be on other processes?
>>>     - can I wrap the compile:file call in something so that it will bail
>>>     out if memory issues look likely (try/catch is no use)?
>>>
>>>     With thanks and best wishes
>>>
>>>     Ivan
>>>
>>>
>>>     --
>>>     ==============================__==============================
>>>     Ivan A. Uemlianin PhD
>>>     Llaisdy
>>>     Speech Technology Research and Development
>>>
>>>     ivan@REDACTED <mailto:ivan@REDACTED>
>>>     www.llaisdy.com <http://www.llaisdy.com>
>>>     llaisdy.wordpress.com <http://llaisdy.wordpress.com>
>>>     github.com/llaisdy <http://github.com/llaisdy>
>>>     www.linkedin.com/in/__ivanuemlianin
>>>     <http://www.linkedin.com/in/ivanuemlianin>
>>>
>>>                              festina lente
>>>     ==============================__==============================
>>>     _________________________________________________
>>>     erlang-questions mailing list
>>>     erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>>>     http://erlang.org/mailman/__listinfo/erlang-questions
>>>     <http://erlang.org/mailman/listinfo/erlang-questions>
>>>
>>>
>>
>> --
>> ============================================================
>> Ivan A. Uemlianin PhD
>> Llaisdy
>> Speech Technology Research and Development
>>
>>                     ivan@REDACTED
>>                      www.llaisdy.com
>>                          llaisdy.wordpress.com
>>               github.com/llaisdy
>>                      www.linkedin.com/in/ivanuemlianin
>>
>>                         festina lente
>> ============================================================
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>

-- 
============================================================
Ivan A. Uemlianin PhD
Llaisdy
Speech Technology Research and Development

                     ivan@REDACTED
                      www.llaisdy.com
                          llaisdy.wordpress.com
               github.com/llaisdy
                      www.linkedin.com/in/ivanuemlianin

                         festina lente
============================================================