[erlang-questions] Dangers of generating a large erlang module

[Sergej Jurecko]

Try a public ets table with read_concurrency true. 


Sergej

On Sep 29, 2013, at 12:53 PM, Ivan Uemlianin wrote:

> Thanks for your comment.
> 
> On 29/09/2013 11:49, Gustav Simonsson wrote:
>> What kind of rules are contained in the CSV files? What information is
>> defined in each rule?
> 
> See reply to Anthony Ramine.
> 
>> It's probably easier parsing the rules to some Erlang data type which
>> can be given to a generic function.
>> The internal format of the rules could be kept in e.g. ETS or in the
>> state of a gen_server.
> 
> Agreed.  However, these rules are going to be heavily consulted. Wouldn't gen_server and even ETS be more of a bottleneck danger?
> 
> Best wishes
> 
> Ivan
> 
> 
>> Cheers,
>> Gustav Simonsson
>> 
>> 
>> On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin <ivan@REDACTED
>> <mailto:ivan@REDACTED>> wrote:
>> 
>>    Dear All
>> 
>>    I am generating some erlang code from a csv file and using the
>>    function save_compile_and_load/2 below to load it into the running
>>    system.  The input csv is essentially a series of rules, so the
>>    outpout erlang has roughly the same number of lines as the input has
>>    lines.
>> 
>>         save_compile_and_load(__FilePath, Text) ->
>>             file:write_file(FilePath, Text),
>>             {ok, ModName} = compile:file(FilePath),  % <--
>>             {module, _M} = code:load_file(ModName),
>>             code:purge(ModName),
>>             ok.
>> 
>>    All goes well on small test files, but the files I want to use IRL
>>    are relatively large --- around 120,000 lines.
>> 
>>    Running on a Mac (oldish, probably with 32bit emulator), compiling
>>    (the line marked in the code below) causes an error and crashes the
>>    VM, with several repeats of this error message:
>> 
>>         beam.smp(1992,0xb022b000) malloc:
>>           *** mmap(size=913309696) failed (error code=12)
>>           *** error: can't allocate region
>>           *** set a breakpoint in malloc_error_break to debug
>> 
>>    Running on linux with (presumably) a 64bit emulator, the compile
>>    works, but takes about 15 minutes.
>> 
>>    I'm not especially bothered that it takes a long time, but I am
>>    worried that this is something that could crash the whole system.
>> 
>>    I'm looking into splitting up the code into several smaller modules,
>>    but obviously I'd rather avoid crashing the VM.
>> 
>>    Questions:
>>    - is compiling large files on a 64bit emulator safe?  What effects
>>    could there be on other processes?
>>    - can I wrap the compile:file call in something so that it will bail
>>    out if memory issues look likely (try/catch is no use)?
>> 
>>    With thanks and best wishes
>> 
>>    Ivan
>> 
>> 
>>    --
>>    ==============================__==============================
>>    Ivan A. Uemlianin PhD
>>    Llaisdy
>>    Speech Technology Research and Development
>> 
>>    ivan@REDACTED <mailto:ivan@REDACTED>
>>    www.llaisdy.com <http://www.llaisdy.com>
>>    llaisdy.wordpress.com <http://llaisdy.wordpress.com>
>>    github.com/llaisdy <http://github.com/llaisdy>
>>    www.linkedin.com/in/__ivanuemlianin
>>    <http://www.linkedin.com/in/ivanuemlianin>
>> 
>>                             festina lente
>>    ==============================__==============================
>>    _________________________________________________
>>    erlang-questions mailing list
>>    erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>>    http://erlang.org/mailman/__listinfo/erlang-questions
>>    <http://erlang.org/mailman/listinfo/erlang-questions>
>> 
>> 
> 
> -- 
> ============================================================
> Ivan A. Uemlianin PhD
> Llaisdy
> Speech Technology Research and Development
> 
>                    ivan@REDACTED
>                     www.llaisdy.com
>                         llaisdy.wordpress.com
>              github.com/llaisdy
>                     www.linkedin.com/in/ivanuemlianin
> 
>                        festina lente
> ============================================================
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions