[erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com)

[Robert Virding]

You should have added this answer, it would have led to a very interesting discussion! 

Robert 

----- Original Message -----

> From: "Jesper Louis Andersen" <jesper.louis.andersen@REDACTED>

> I remember reading that discussion on Hacker News as for why Google
> considered but did not pick Erlang. The reason I did not answer was because
> I don't believe it would be fruitful to do so, but on this mailing list I
> can't resist.

> One of the things I've been looking into as of late is the price paid for
> error handling. When you anticipate error and handle it, there is an
> overhead to this handling. It is easy to get extreme Queries per second
> (QPS) rates if you ignore faults and have no way to mitigate them.

> I believe that the stance of many systems are to claim that the system is
> void of any internal error and then run without the internal error
> protection of the system. Outside, you run something like monit in order to
> keep the system up and restarted should it die. In turn, such systems can be
> made so they run much faster than the typical Erlang system easily:

> * There is no need for doing hot code upgrade on the fly. Hence, you can
> native-compile everything into one static binary, inline like mad and so on.
> This speeds up a lot of the computational stuff.
> * You don't need to set a monitor for each call. Since there are no errors
> (by your claim) the code never goes wrong and hence why set it?
> * You have no distribution. So everything happens locally in your own memory
> space. This usually gives complete knowledge and simplifies a lot of
> possible error paths to "non-existent".

> To make Go work in practice, you need to adopt this stance: you deploy small
> statically compiled binaries which are "obviously correct" by being small.
> And if something is amiss, you do not go for a partial failure or a gradual
> degradation of the system: off with its head!

> It is a trade-off. You get more speed, but are ignoring certain failure
> scenarios. But you are also suddenly going to rely 100% on the code being
> correct. In your code and in all libraries you use.

> On Tue, Aug 20, 2013 at 9:26 AM, Loïc Hoguin < essen@REDACTED > wrote:

> > Why do you paste us HN discussions? There's no better waste of time than to
> > read programmers comparing "sizes".
> 

> > On 08/20/2013 07:02 AM, giovanni_re wrote:
> 

> > > https://news.ycombinator.com/ item?id=6234736
> > 
> 
> > > Rewriting a large production system in Go ( matt-welsh.blogspot.com )
> > 
> 
> > > 222 points by qdie 1 day ago | 173 comments
> > 
> 
> > > http://matt-welsh.blogspot. com/2013/08/rewriting-large-
> > > production-system-in-go.html
> > 
> 

> > > MW: PhD U. Cali Forni Berkeley
> > 
> 

> > > ========== https://news.ycombinator.com/ item?id=6234736
> > 
> 
> > > derefr 1 day ago | link You know, every time I see some Googler shocked
> > 
> 
> > > at the effectiveness and various advantages of coding in Go, I wonder
> > 
> 
> > > why Google never adopted Erlang. They could have been getting all these
> > 
> 
> > > same advantages (and then some) a decade ago :)
> > 
> 

> > > reply
> > 
> 

> > > [GR: Would Googl use an Erlang on its' search engine show?]
> > 
> 

> > > zaphar 1 day ago | link (full disclosure: I work at google and also like
> > 
> 
> > > erlang) Erlang has fantastic facilities for robustness and concurrency.
> > 
> 
> > > What it does not have is type safety and it's terrible at handling text
> > 
> 
> > > in a performant fashion. So if you don't care about either of those
> > 
> 
> > > things and only care about robustness and concurrency then Erlang is
> > 
> 
> > > great. There were internal discussions about Erlang here but the upshot
> > 
> 
> > > was. We had already basically duplicated Erlangs supervision model in
> > 
> 
> > > our infrastructure, only we did it for all languages and Erlang didn't
> > 
> 
> > > offer any benefits in performance for us. It's only benefit would have
> > 
> 
> > > been the concurrency model. That's much less benefit than Go gives. Go
> > 
> 
> > > gives you Erlangs concurency model, a similar philosophy of robustness,
> > 
> 
> > > type safety, all batteries included, and performance. Equating the two
> > 
> 
> > > languages works in 1 or 2 dimensions but not on all the dimensions
> > 
> 
> > > google cares about.
> > 
> 

> > > reply
> > 
> 

> > > derefr 1 day ago | link Interesting, thanks for that; it's pretty much
> > 
> 
> > > what I guessed (especially the bit about the supervision tree and
> > 
> 
> > > hot-code-upgrade advantages being mooted by your infrastructure.) On a
> > 
> 
> > > tangent, though: > What it does not have is type safety I've tried to
> > 
> 
> > > work this out before (I'm designing a new language for Erlang's VM), but
> > 
> 
> > > as far as I can tell, type safety is in practice incompatible with
> > 
> 
> > > VM-supported hot code upgrade. If you have two services, A and B, and
> > 
> 
> > > you need to upgrade them both, but you can't "stop the world" to do an
> > 
> 
> > > atomic upgrade of both A and B together (because you're running a
> > 
> 
> > > distributed soft real-time system, after all), then you need to switch
> > 
> 
> > > out A, and then switch out B. So, at some point, on some nodes, A will
> > 
> 
> > > be running a version with an ABI incompatible with B. In a
> > 
> 
> > > strongly-typed system, the VM wouldn't allow A's new code to load, since
> > 
> 
> > > it refers to functions in B with type signatures that don't exist. On
> > 
> 
> > > the other hand, in a system with pattern-matching and a "let it crash"
> > 
> 
> > > philosophy, you just let A's new code start up and repeatedly
> > 
> 
> > > try-and-fail to communicate with B for a while, until B's code gets
> > 
> 
> > > upgraded as well--and now the types are compatible again. It's an
> > 
> 
> > > interesting problem.
> > 
> 

> > > reply
> > 
> 

> > > laureny 1 day ago | link > type safety is in practice incompatible with
> > 
> 
> > > VM-supported hot code upgrade. That's not true. First, it's very easy to
> > 
> 
> > > hot reload changes that have been made to the code that are backward
> > 
> 
> > > compatible. The JVM spec describes in very specific details what that
> > 
> 
> > > means (adding or removing a method is not backward compatible, modifying
> > 
> 
> > > a body is, etc...). This is how Hotswap works, the JVM has been using it
> > 
> 
> > > for years. As for changes that are backward incompatible, you can still
> > 
> 
> > > manage them with application level techniques, such as rolling out
> > 
> 
> > > servers or simply allow two different versions of the class to exist at
> > 
> 
> > > the same time (JRebel does that, as do other a few other products in the
> > 
> 
> > > JVM ecosystem). Erlang doesn't really have any advantages over
> > 
> 
> > > statically typed systems in the hot reload area, and its lack of static
> > 
> 
> > > typing is a deal breaker for pretty much any serious production
> > 
> 
> > > deployment.
> > 
> 

> > > reply
> > 
> 

> > > rdtsc 20 hours ago | link > lack of static typing is a deal breaker for
> > 
> 
> > > pretty much any serious production deployment. Are you talking about
> > 
> 
> > > Google only where they made it a mandate or in general? There are
> > 
> 
> > > serious production deployments on Python, Ruby, Erlang and Javascript. I
> > 
> 
> > > will trade expressiveness and less lines of code with a strong but
> > 
> 
> > > dynamically typed language + tests over more a static typed language
> > 
> 
> > > with more lines of code all being equal. Or put it another way, if
> > 
> 
> > > strong typing is the main thing that protects against lack of faults and
> > 
> 
> > > crashes in production, there is a serious issue that needs to be
> > 
> 
> > > addressed (just my 2 cents).
> > 
> 

> > > reply
> > 
> 

> > > derefr 23 hours ago | link > As for changes that are backward
> > 
> 
> > > incompatible, you can still manage them with application level
> > 
> 
> > > techniques, such as rolling out servers or simply allow two different
> > 
> 
> > > versions of the class to exist at the same time (JRebel does that, as do
> > 
> 
> > > other a few other products in the JVM ecosystem). Neither of these allow
> > 
> 
> > > for the whole reason Erlang has hot code upgrade in the first place:
> > 
> 
> > > allowing to upgrade the code on one side of a TCP connection without
> > 
> 
> > > dropping the connection to the other side. Tell me how to do that with a
> > 
> 
> > > static type system :)
> > 
> 

> > > reply
> > 
> 

> > > pmahoney 10 hours ago | link Tomcat (and other app servers) has support
> > 
> 
> > > for doing hot reloads of Java web apps while not reloading the HTTP
> > 
> 
> > > layer (and not dropping TCP connections).
> > 
> 
> > > http://www.javacodegeeks.com/ 2011/06/zero-downtime- deploymen. .. I have
> > 
> 
> > > implemented a similar system for JRuby apps running inside a Servlet
> > 
> 
> > > container. There are many caveats. I don't actually recommend it because
> > 
> 
> > > for a while you're using nearly twice the memory (and JRuby is
> > 
> 
> > > particularly memory hungry). Also there are many ways to leak the old
> > 
> 
> > > class definitions such that they are not GC'd (e.g. thread locals). But
> > 
> 
> > > it's certainly possible. I suspect that Erlang, Java, and all languages
> > 
> 
> > > are in the same boat: some parts can be upgraded live in the VM while
> > 
> 
> > > other parts require a full restart (maybe coordinating with multiple
> > 
> 
> > > nodes and a load balancer to achieve zero-downtime).
> > 
> 

> > > reply
> > 
> 

> > > lenkite 10 hours ago | link Out of curiosity, where/why would such an
> > 
> 
> > > exotic feature be needed in today's internet architectures where you
> > 
> 
> > > always front a group of servers with a load balancer ?
> > 
> 

> > > reply
> > 
> 

> > > butterflyhug 5 hours ago | link Not all Internet protocols are HTTP. If
> > 
> 
> > > you're running a service where long-lived connections are the norm,
> > 
> 
> > > "simply fronting a bunch of servers with a load balancer" can require a
> > 
> 
> > > pretty smart load balancer. E.g. IMAP connections often last hours or
> > 
> 
> > > even days, and are required to maintain a degree of statefulness.
> > 
> 

> > > reply
> > 
> 

> > > DanWaterworth 1 day ago | link Go gives you Erlangs concurency model
> > 
> 
> > > There are a number of significant differences between Erlang's and Go's
> > 
> 
> > > concurrency models: Asynchronous vs synchronous communication,
> > 
> 
> > > per-thread vs per-process heaps, send to process vs send to channel.
> > 
> 

> > > reply
> > 
> 

> > > dbaupp 1 day ago | link Go doesn't have strong type safety either; I
> > 
> 
> > > remember a recent story about a Go stdlib function "accidentally"
> > 
> 
> > > calling an interface it shouldn't.
> > 
> 

> > > reply
> > 
> 

> > > f2f 22 hours ago | link it wasn't accidental -- it was written on
> > 
> 
> > > purpose by a programmer (a conversion from Writer to WriteCloser). it
> > 
> 
> > > was immediately acknowledged as an error and eventually may be caught by
> > 
> 
> > > the standard code examining tool "vet".
> > 
> 

> > > reply
> > 
> 

> > > pcwalton 22 hours ago | link What would the static analysis that "vet"
> > 
> 
> > > is performing enforce to stop this? No interface-to-interface downcasts?
> > 
> 

> > > reply
> > 
> 

> > > .....
> > 
> 
> > > [It goes on, & on, & on & on,
> > 
> 
> > > when Go & Erlang party together.]
> > 
> 

> > > --- Join the BerkeleyTIP-Global mail list - http://groups.google.com/
> > > group/BerkTIPGlobal . All Freedom SW, HW & Culture.
> > 
> 

> > > ______________________________ _________________
> > 
> 
> > > erlang-questions mailing list
> > 
> 
> > > erlang-questions@REDACTED
> > 
> 
> > > http://erlang.org/mailman/ listinfo/erlang-questions
> > 
> 

> > --
> 
> > Loïc Hoguin
> 
> > Erlang Cowboy
> 
> > Nine Nines
> 
> > http://ninenines.eu
> 

> > ______________________________ _________________
> 
> > erlang-questions mailing list
> 
> > erlang-questions@REDACTED
> 
> > http://erlang.org/mailman/ listinfo/erlang-questions
> 

> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130821/0b0a4ca4/attachment.htm>