[erlang-questions] Is mnesia/erlang/ETP strong enough?

[Yang Zhang]

After 2 weeks programming and running with erlang program, I got lots
of memory error and erl crashed more than 50 times.

I was trying to run 400 processes + mnesia on one node. The 400
processes was designed to scrape data from web then save them to
mnesia database.
At first, after about 2 or 3 hours running, I found erl crashed(not
program). Then I restart then 2 or 3 hours erl crashed again.
I found the erl process consume about 1.3G memory then it crashed, I
start to wonder if there are some things wrong with too many
processes. So I minus to 200 processes and  split them to 10 nodes,
with run 20 processes and the main one running mnesia database.

Things went well untill 2 days later, I found the main erl node
crashed again. After restart it, after 1 or 2 minutes it crashed
again. Eventually I found it is the table type's problem, so I changed
data type from disc_* to disc_only. Then problem solved.

Then it keeps running for 4 days, everything goes well until I want to
check how many data I have. So I open table viewer:
tv:start().
The see there are 6M records in my table, quite happy. Then
double-click, both node crashed. Sign~~

I got clever. Now I never opened that table. I choose a small one
which only have 120K records. Open it. No crash. Safty. The search for
some words, only 2K. OK. No crash. Then search for another one, the
number is climbing, when it hits 4K. oh My god. Both nodes crash again
only show me the ugly

Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "heap").


damm~~~

I am using C#/SqlServer to develop program. I haven't seen anytime
sqlserver open a table with 6M table will crash itself, I even tried
to open a 50M records table.
Why erl system so volunerable?

Is crash the better way than only ternimate the problem process? Don't know why.

Also I have fell in love with erlang, but its so unstable behavior
make it far away from my commerial program.

Any option?


http://harbork.com/Is-mnesia-erlang-ETP-strong-enough/Is-mnesia-erlang-ETP-strong-enough-4b4vyEV84t2fjviIJHLw.html