Erlang, Strings, Mnesia?

[Joe Armstrong]

On Thu, 8 Apr 2004, Jimmie Houchin wrote:

> Hello,
> 
> I am trying to understand whether or not Erlang is right or a good 
> option for a website I am building.
> 
> Currently I am processing 422,000 files with 6.6gb of text to place in a 
> database. I will have to regularly parse and process large volumes of 
> textual data.
> 
> The database will initially have 4-5 million rows/objects.
> 
> This data in turn will be searchable. A prime purpose of the data.
> 
> On this page of the Mnesia User Guide it says:
> #http://www.erlang.se/doc/doc-5.3/lib/mnesia-4.1.4/doc/html/Mnesia_chap1.html#1
> #
> #On the other hand, Mnesia may not be appropriate with the following 
> #types of applications:
> #
> #    * Programs that process plain text or binary data files
> 
> I am used to Squeak and Python and native string support.
> 
> I understand well Erlang's strengths. I don't understand where it is weak.
> 
> An example.
> 
> Would Erlang be suitable for a Google type site?
> An Amazon? An eBay?

  Based on what you have said so far there is no way of answering your
question.

  Before doing *any*  major bit of programming I always  try to do the
following:

	1) Identify the most difficult problem(s) to be solved
	2) Prototype solutions the the problems raised in 1)
	3) Measure performance of prototype

  When this has converged you can go ahead and write the program.

  Premature optimizations and pre-conceived ideas about what is and is
not efficient is a sure fired receipt for disaster.

  I  would suggest you  write a  little parser  and build  a prototype
*before* you make any decisions.

  Personally (now I'm *very* biased here) I'd go for yaws - why?

  Yaws got concurrency right.

  I  know nothing  about your  application but  there is  a heck  of a
difference between  having say 1000 users  who just request  a web page
and then break  the connection and 100,000 users who  are logged in to
chat systems and remain connected for hours at a time.

  Yaws is great for handling large numbers of persistent connections -
outstanding.

  I would also  (counter to what some other people  have said) be very
cautious about  using *different* components in your  system.  If you
use Erlang AND mySQL you have to learn to use two things - not one.

  The mySQL way of thinking (SQL)  is NOT the Erlang way of thinking -
and  you  will get  an  impedance mismatch  between  the  two ways  of
thinking.

  Whenever I see projects that use A and B and C and D ... I judder in
horror. Each new thing causes an explosion in complexity << and yes any
problem no matter how trivial can be made nightmarishly difficult just
by using Java, mySQL, Cobra, ASN.1, SOAP, WSDL, ...>>

  Projects  that I  complete  in a  small  amount of  time are  almost
invariably monolingual. Do *everything* in Erlang or C or whatever.

  IMHO - as soon  as you start mixing things you get  into a mess - as
Wirth  said (paraphrased)  - mastery  of one  language is  better than
incomplete knowledge of several.

  So here's what I'd do:

	- Identify the HARD bits
	- do small prototypes of the hard bits
	- measure
	- if fast enough HOORAY
	  if not fast enough ask this group why

  Cheers

  /Joe







> 
> Any wisdom, advise welcome and appreciated.
> 
> Jimmie Houchin
> 
>