[erlang-questions] The quest for the perfect programming language for massive concurrency.

[Joe Armstrong]

On Thu, Jan 30, 2014 at 5:19 PM, kraythe . <kraythe@REDACTED> wrote:

> Ok right up front, I'm a Java Guru, not a Scala or Erlang one. What that
> means is that I know more than most debs about the core java language, but
> enough to know where the problems are. And certainly java has many issues
> but it also has massive momentum. I think one of the issues with Java can
> be expressed in one little programming puzzle I came across recently:
>
> *In a relevant language, create an array of 1000 numbers. Initialize all
> of the values in the array to zero. Create two threads that run
> concurrently and which increment each element of the array one time.*
>
>
> Interesting? The solution in Scala that I came up with is the following.
>
> def mtmap[A](data : List[A], threads: Int, f: (A) => A) = {
>   import scala.concurrent.{ExecutionContext, Future, Await}
>   implicit val executor =
> java.util.concurrent.Executors.newFixedThreadPool(threads)
>   implicit val context =  ExecutionContext.fromExecutorService(executor)
>   val futures : Future[List[A]]= Future.traverse(data)(x => Future{f(x)})
>   import scala.concurrent.duration._
>   import scala.language.postfixOps
>   val results : List[A] = Await.result(futures, 1 minute)
>   results
> }
>
>
> Just thinking of doing this in Java will bring up some of the big problems
> with Java; I will leave it as a mental exercise for the reader. The problem
> is that Scala inherits some of them from the JVM and that has made me look
> into Erlang. The goal being to select a language for the development of a
> concurrent TCP/IP based application with thousands of users interacting in
> a small space.
>

A benchmark that tells you how concurrent array updates works won't tell
you anything about
well sockets work.

Erlang was explicitly designed for managing thousands of simultaneous
sessions (ie telecomms applications) - (that's why things like WhatsApp is
programmed in Erlang)
try Googling "whatsapp+erlang".

Java was designed for set-top boxes - both languages are good for what they
were designed to be good at. Consequently Erlang is good at handling large
numbers of parallel connections and java is really good for set-top boxes.
Both languages suck when used outside their design domains. So Erlang is
crap
at programming small memory footprint embedded devices and java is not
stunning good at handling
concurrency.


>
> So far I think the contenders I have are Scala with Akka or Erlang. And
> yes, I know there are evangelists to both and I will post this to an Scala
> list to get their feedback as well (or something similar). Now, right up
> front I am not peeing on either language. They both rock or they wouldn't
> be on the list. The question is which should win the prize. There is no
> going back once development is 1000 hours down the road.
>
> *Scala: *
> *Pros:*
>
>    1. Based on Java Ecosystem so easier to staff because good Java devs
>    can convert to Scala.
>    2. Decent tools (which are getting better). Many libraries.
>    3. Static typing with type inference.
>
> *Cons:*
>
>    1. Based on Java Ecosystem and inherit the problems of that ecosystem
>    (i.e. immutable is a function of the design of a class, not of the language
>    so it can't be guaranteed.), Also library code under the hood is not as
>    rigorous as scala code in enforcing immutability so at some point you are
>    rolling dice here.
>    2. Scala is also more heavyweight than Erlang when it comes to
>    spawning thousands of processes. Erlang was built from the ground up to do
>    concurrency. For Scala its an Akka bolt on can carries the Java threading
>    nightmare (shared memory, etc).
>
>
How about millions of processes - "thousands" of processes is in the "tiny"
league table


>
>    1.
>    2. Scala is not as fast. My server will be doing billions of vector
>    math calculations per day and they have to be in the terms of milliseconds
>    of latency. It has to be that if I have a user R in the server at position
>    V where V is a vector, I need to calculate all other actors within 50 units
>    and get that answer in milliseconds so that only the network latency is the
>    bottleneck. Some of this can be helped with algorithms like spatial grids
>    and so on but still we are looking at a couple of hundred vector math calls
>    per second.
>
> I don't get a feeling for what "billions of vector math calculations per
day" means. How long does
one "vector math calculation" take - or is this what you were trying to
measure?

If you're doing really heavy number crunching Julia seems the way to go.
Julia is outperforming Fortran on many benchmarks - and Fortran always set
the gold standard for number crunching.

Something like Erlang for orchestration, Julia for number crunching and 0MQ
for messaging would be what I'd look at.



>    1. Scala is harder to hook up to dozens of nodes and move actors from
>    node x to node y than Erlang, mainly because that was one of the design
>    goals of Erlang.
>
>
> *Erlang:*
> *Pros:*
>
>    1. Built for concurrency. Can handle dozens of hardware nodes, build
>    massive applications of the kind I am trying to deploy. Think of 100k users
>    connecting to the cluster with a TCP connection and interacting over that
>    connection which interacts with any or all of the other 100k actors.
>    2. Built from the ground up with immutability in mind. Immutability is
>    a language feature, not compromisable. Its not JVM based and so is not
>    under the same issues as Java is.
>
> What's immutability got to do with anything? If you want to do fast vector
operations things like
Fortran is king - and that's mutable as hell.

The kind of immutability that Erlang has is for reasons of fault tolerance
- we don't want programs to crash and run wild - fast math needs mutable
structures



>
>    1. There is merit in the thought that static typing is sometimes a
>    hinderance to a language.
>
> *Cons:*
>
>    1. The tools are, well frankly, garbage. Sorry, in 2014 to be pushed
>    back to coding with VIM and makefiles is primitive. Rebar is crytptic and
>    just the pet project of a guy on GIT. Compared to Gradle, Maven and even
>    (though I don't care for it much) SBT, rebar is ... lacking. I want to
>    spend time working on my business logic, not fighting tools. There are
>    plugins for eclipse and intellij but they have minimal functionality and i
>    keep reverting back to vim.
>
> Tools are no substitute for thought. None of the tools you mention help me
think - the thing that takes
time in programming is deciding exactly what problem to solve. Once you
know you can usually knock out the program in a few hours.

The best tool we have is (or course) maths - ie logical reasoning and
proofs.

The average programmer produces about 10 lines of code per day, but they
might write hundreds or
even thousands of lines of code per day that never makes it to the final
product. If they knew what to
do they could just write these ten lines in a few minutes, but they don't
know so they have to write
thousands of lines to help try and figure out what the problem is. It also
creates the illusion of work
and they can get paid for it.

The big problem in programming is understanding the problem. "programming
is understanding"
The tools we have today are no better than they were 40 years ago. My brain
is still far far better at
debugging that any fancy tool. I have no idea my brain works - all I know
is that I often work on
a problem and then give up. I go to sleep and wake up with the solution.
Now that is far far better
than *any* IDE on the planet.

I like to spend my time figuring out *exactly* how things related to my
applications, there would be
no point in figuring out (say) eclipse works - since editing text is not a
problem for me - I'm happy
with emacs - and even happier with a pen and paper (which boots far
quicker, is far more portable,
and far more flexible)

I spend 65% of my programming time fixing stuff that I need that is broken.
Every time a program
does not install or work exactly as advertised I have to pull it apart and
fix it. None of these tools
are of any use when they break. Rebar/Maven/Git etc are great when they
work.

Even emacs is crashing randomly on my mac - the problem is not the tools
but that fact that
broken tools take a long time to fix. (this is why I try to use as little
as possible - emacs, make and bash, do it for me)

In the good old days programmers could talk to each other - everybody who
could program knew how to write makefile and bash scripts. Now a guy who
speaks Maven is floored when sent a rakefile.

And as for eclipse etc. these are unbelievably horrible.


>    1. Much harder to staff than Scala because it is not Java based.
>    2. Fewer general purpose libraries and no major central repositories.
>    I don't want to write code to create JSON, that isnt part of my business
>    scope. I will pick that one of the shelf If i can.
>
> It's often far quicker to write code than a) find a library b) install it
c) fix the bugs in the library.
Creating JSON is trivial:

     io:format("{'firstname':'joe', 'lastname':'armstring'}").

Parsing it is a wee bit messy.


>    1. Records as the only structured data type is ... annoying.
>
>
> The problem I have is I can't find the perfect solution. Erlang is
> compelling but also is Scala.
>
> Opinions?
>

There is no easy path.

    - choose your languages
    - write the benchmarks in all the languages
    - measure
    - don't believe what other people say - base the results on your
personal experience
    - understand your problem - write a detailed English description of the
problem and
      give to to your friends - get them to read it and explain the problem
back to you

There are also tradeoff to consider.

    You want super fast math (this implies mutable data)
    You want safety (this implies non-mutable data)
    You mention JSON (implies inefficient data transfer)
    You want massive communication
etc.

    These pull in different directions.

Cheers

/Joe

>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140130/81b954eb/attachment.htm>