<div dir="ltr"><div>What kind of rules are contained in the CSV files? What information is defined in each rule?<br><br></div><div>It's probably easier parsing the rules to some Erlang data type which can be given to a generic function.<br>
</div><div>The internal format of the rules could be kept in e.g. ETS or in the state of a gen_server.<br><br></div><div>Cheers,<br>Gustav Simonsson<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin <span dir="ltr"><<a href="mailto:ivan@llaisdy.com" target="_blank">ivan@llaisdy.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear All<br>
<br>
I am generating some erlang code from a csv file and using the function save_compile_and_load/2 below to load it into the running system. The input csv is essentially a series of rules, so the outpout erlang has roughly the same number of lines as the input has lines.<br>
<br>
save_compile_and_load(<u></u>FilePath, Text) -><br>
file:write_file(FilePath, Text),<br>
{ok, ModName} = compile:file(FilePath), % <--<br>
{module, _M} = code:load_file(ModName),<br>
code:purge(ModName),<br>
ok.<br>
<br>
All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines.<br>
<br>
Running on a Mac (oldish, probably with 32bit emulator), compiling (the line marked in the code below) causes an error and crashes the VM, with several repeats of this error message:<br>
<br>
beam.smp(1992,0xb022b000) malloc:<br>
*** mmap(size=913309696) failed (error code=12)<br>
*** error: can't allocate region<br>
*** set a breakpoint in malloc_error_break to debug<br>
<br>
Running on linux with (presumably) a 64bit emulator, the compile works, but takes about 15 minutes.<br>
<br>
I'm not especially bothered that it takes a long time, but I am worried that this is something that could crash the whole system.<br>
<br>
I'm looking into splitting up the code into several smaller modules, but obviously I'd rather avoid crashing the VM.<br>
<br>
Questions:<br>
- is compiling large files on a 64bit emulator safe? What effects could there be on other processes?<br>
- can I wrap the compile:file call in something so that it will bail out if memory issues look likely (try/catch is no use)?<br>
<br>
With thanks and best wishes<br>
<br>
Ivan<br>
<br>
<br>
-- <br>
==============================<u></u>==============================<br>
Ivan A. Uemlianin PhD<br>
Llaisdy<br>
Speech Technology Research and Development<br>
<br>
<a href="mailto:ivan@llaisdy.com" target="_blank">ivan@llaisdy.com</a><br>
<a href="http://www.llaisdy.com" target="_blank">www.llaisdy.com</a><br>
<a href="http://llaisdy.wordpress.com" target="_blank">llaisdy.wordpress.com</a><br>
<a href="http://github.com/llaisdy" target="_blank">github.com/llaisdy</a><br>
<a href="http://www.linkedin.com/in/ivanuemlianin" target="_blank">www.linkedin.com/in/<u></u>ivanuemlianin</a><br>
<br>
festina lente<br>
==============================<u></u>==============================<br>
______________________________<u></u>_________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/<u></u>listinfo/erlang-questions</a><br>
</blockquote></div><br></div>