<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Thank you for providing an explanation
of the problems and the information about the use of Erlang with
databases. Often these points for (or against) the usage of
Erlang, gets lost within passionate arguments and sales pitches,
but this provides some level-headed information.<br>
<br>
On 01/11/2013 10:12 AM, Aliaksey Kandratsenka wrote:<br>
</div>
<blockquote
cite="mid:CADpJO7xGP_XqCzAh7AbbJtGiJD1b1fywUbKfAA1TX5d4Z7kJ-w@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra" style="">Sorry for replying for digest.
I normally just passively listen here.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">I'm one of the guys directly
involved in final stages of figuring out that bug.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">Here's the story.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">As we approached finalization
of Couchbase Server 2.0 we started seeing <a
moz-do-not-send="true"
href="http://www.couchbase.com/issues/browse/MB-6638">http://www.couchbase.com/issues/browse/MB-6638</a>.
Given that we have a bunch of custom nifs, we weren't sure
until very last minute whether it's erlang vm bug or ours.
And, initially, even reliably reproducing this was hard. And
when we learned how to reproduce it still required few hours
of running our full stack. That's why we never asked help in
erlang MLs.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">Backtraces suggested something
in efile driver related to async io threads, so our folks
tried to disable them and observed that crashes were gone.
They also tried to reproduce this problem in smaller scale,
but they only found some different bug. Which Filipe fixed
recently: <a moz-do-not-send="true"
href="https://github.com/erlang/otp/commit/5ddf4118617d7e5bac5b889025aa0f3903796a49">https://github.com/erlang/otp/commit/5ddf4118617d7e5bac5b889025aa0f3903796a49</a></div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">We had to ship 2.0 without
getting on top of this. So 2.0 _does not have_ async io
threads enabled. This means some heavy disk io (which we do)
can cause unpredictable delays for any erlang process and thus
some end-user badness.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">BTW, Why something as crucial
as async io threads is off by default ? When I was trying to
argue for not disabling async io threads prior to 2.0 and
fighting this issue "to death", I've heard argument: "it's
experimental feature because it's off by default". Is it ?</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">In the end we found that when
process linked to raw file dies, it'll stop linked file
driver. And as part of that underlying os file or gzip stream
(depending on compressed option) will be closed. Without
taking into account any possible in-flight async call for that
file. It's somewhat harmless for plain files to try to
read/write closed fd, but it'll clearly cause crash if some
code tries to read from closed (and freed) gzip stream. And of
course tiny possibility of reading/writing to/from another
file that happened to reuse same fd is not fun either.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">We found that file_sorter is
actually passing compressed option "just case" all the time
and we confirmed that indeed crashes happen because of those
"compressed by not really" raw file ports.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">Couchbase Server 2.0.1 will
ship with workaround that replaces file_sorter from stdlib
with it's tiny fork that cuts compressed option out. I've seen
Filipe produced erlang vm patch for that issue too, but what
I've seen only covers closing compressed files. IMHO right fix
would be to cover both options.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">I'm also seeing some folks in
this thread being unhappy and somewhat angry. Apparently they
seem to interpret Damien's opinion as bashing of Erlang. Which
is IMHO not the case. I think his arguments apply for core
database software.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">And In my humble opinion
candid expressions like that should be encouraged and studied
with cold minds.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">It is true that we have found
that getting performance out of Erlang and in general
understanding what happens inside VM is next to impossible.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">And, personally, even without
knowing all I know about challenges of getting performance out
of Erlang VM I'd still say that doing core database in erlang
(or any other not C-like-low-level language) is just crazy.
IMHO.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">BTW, perhaps, not everybody
here is aware that Damien has "erlanger of the year" 2009
award. I guess for CouchDB. Indeed, it's very much like love
affair that's gone :) But hey, un-loved being is not
necessarily bad, right ?</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">As for plans of using Erlang
in Couchbase (which is former Membase). We indeed plan to
incrementally and gradually rewrite performance sensitive
pieces in C or C++. But there are no concrete plans of getting
rid of Erlang entirely, yet. It works ok for our cluster
management layer. </div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">And IMHO compared to some our
competitors which either do all in low-level language (mongo,
rethinkdb) or high-level (riak) our approach of combining
low-level language for "moving bits around" and high-level
language for cluster management and orchestration seems to
work best. </div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">So even if we switch off
Erlang completely, we'll very likely still use something much
higher level than C for cluster management and other not
performance sensitive but sometimes tricky pieces.</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jan 11, 2013 at 3:00 AM, <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:erlang-questions-request@erlang.org"
target="_blank">erlang-questions-request@erlang.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div id=":19n">Date: Fri, 11 Jan 2013 09:43:36 +0100<br>
From: Henning Diedrich <<a moz-do-not-send="true"
href="mailto:hd2010@eonblast.com">hd2010@eonblast.com</a>><br>
To: Anton Lebedevich <<a moz-do-not-send="true"
href="mailto:mabrek@gmail.com">mabrek@gmail.com</a>><br>
Cc: "<a moz-do-not-send="true"
href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>"
<<a moz-do-not-send="true"
href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>><br>
Subject: Re: [erlang-questions] what is the "race
condition bug in<br>
core Erlang" mentioned by @damienkatz?<br>
Message-ID: <<a moz-do-not-send="true"
href="mailto:C5EB7529-A3DA-4FEC-8437-1B54C2F4754F@eonblast.com">C5EB7529-A3DA-4FEC-8437-1B54C2F4754F@eonblast.com</a>><br>
Content-Type: text/plain; charset=windows-1252<br>
<br>
I love that how languages can be love affairs etc.<br>
<br>
A race condition in core Erlang, I am sure Damien will
share his find.<br>
<br>
In the meantime maybe it's worth looking at the
political circumstances.<br>
<br>
Some might note not only that you fall out of love and
then you're irrationally deeply disappointed. You'll
find all the feeling of understanding was an illusion in
the first place. And sometimes you're even right. But
that CouchDB surfed the Erlang hype, a while ago Damien
was able to close a deal, and for some reason I don't
know anyone quite understood announced that he'll
reprogram it all in C.<br>
<br>
Maybe it was an astounding proposition to program a
transactional, local (!) database in the age of Big Data
in a language that happens to be transactional by nature
but is really made for distribution, and it's not too
surprising when that premise is now abandoned. CouchDB
is great for certain things, I have no doubt about that,
how else could it be so successful.<br>
<br>
But maybe one could ask, with the distribution layer of
Couchbase coming from Membase [1] (which means it would
still be Erlang?) but the local storage being in C
(coming from memcached I believe), was there simply a
necessity in play because C would be a better fit with
the rest of the local part of Membase? Like after
renaming things, the CouchDB principle would be
reprogrammed, to replace or amend the memcached parts in
Membase, to become Coucbase, so it had to be in C? And
dealing only with the local storage parts, for a
database, which was probably the task ? I am not sure
that's a natural for Erlang.<br>
<br>
You wouldn't think someone could be talking himself
publicly into loving his partner in a forced marriage?<br>
<br>
Me for instance, I love C. Erlang always makes me feel
stupid. Who wants that.<br>
<br>
Henning<br>
<br>
<br>
[1] old: <a moz-do-not-send="true"
href="http://blog.couchbase.com/why-membase-uses-erlang"
target="_blank">http://blog.couchbase.com/why-membase-uses-erlang</a><br>
</div>
</blockquote>
</div>
<br>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
</body>
</html>