[erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com)

Tue Aug 20 15:11:29 CEST 2013

I remember reading that discussion on Hacker News as for why Google
considered but did not pick Erlang. The reason I did not answer was because
I don't believe it would be fruitful to do so, but on this mailing list I
can't resist.

One of the things I've been looking into as of late is the price paid for
error handling. When you anticipate error and handle it, there is an
overhead to this handling. It is easy to get extreme Queries per second
(QPS) rates if you ignore faults and have no way to mitigate them.

I believe that the stance of many systems are to claim that the system is
void of any internal error and then run without the internal error
protection of the system. Outside, you run something like monit in order to
keep the system up and restarted should it die. In turn, such systems can
be made so they run much faster than the typical Erlang system easily:

* There is no need for doing hot code upgrade on the fly. Hence, you can
native-compile everything into one static binary, inline like mad and so
on. This speeds up a lot of the computational stuff.
* You don't need to set a monitor for each call. Since there are no errors
(by your claim) the code never goes wrong and hence why set it?
* You have no distribution. So everything happens locally in your own
memory space. This usually gives complete knowledge and simplifies a lot of
possible error paths to "non-existent".

To make Go work in practice, you need to adopt this stance: you deploy
small statically compiled binaries which are "obviously correct" by being
small. And if something is amiss, you do not go for a partial failure or a
gradual degradation of the system: off with its head!

It is a trade-off. You get more speed, but are ignoring certain failure
scenarios. But you are also suddenly going to rely 100% on the code being
correct. In your code and in all libraries you use.

On Tue, Aug 20, 2013 at 9:26 AM, Loïc Hoguin <essen@REDACTED> wrote:

> Why do you paste us HN discussions? There's no better waste of time than
> to read programmers comparing "sizes".
>
>
> On 08/20/2013 07:02 AM, giovanni_re wrote:
>
>> https://news.ycombinator.com/**item?id=6234736<https://news.ycombinator.com/item?id=6234736>
>> Rewriting a large production system in Go (matt-welsh.blogspot.com)
>> 222 points by qdie 1 day ago | 173 comments
>> http://matt-welsh.blogspot.**com/2013/08/rewriting-large-**
>> production-system-in-go.html<http://matt-welsh.blogspot.com/2013/08/rewriting-large-production-system-in-go.html>
>>
>> MW: PhD U. Cali Forni Berkeley
>>
>>
>>
>> ==========  https://news.ycombinator.com/**item?id=6234736<https://news.ycombinator.com/item?id=6234736>
>> derefr 1 day ago | link You know, every time I see some Googler shocked
>> at the effectiveness and various advantages of coding in Go, I wonder
>> why Google never adopted Erlang. They could have been getting all these
>> same advantages (and then some) a decade ago :)
>>
>> reply
>>
>> [GR:  Would Googl use an Erlang on its' search engine show?]
>>
>>
>>
>>
>> zaphar 1 day ago | link (full disclosure: I work at google and also like
>> erlang) Erlang has fantastic facilities for robustness and concurrency.
>> What it does not have is type safety and it's terrible at handling text
>> in a performant fashion. So if you don't care about either of those
>> things and only care about robustness and concurrency then Erlang is
>> great. There were internal discussions about Erlang here but the upshot
>> was. We had already basically duplicated Erlangs supervision model in
>> our infrastructure, only we did it for all languages and Erlang didn't
>> offer any benefits in performance for us. It's only benefit would have
>> been the concurrency model. That's much less benefit than Go gives. Go
>> gives you Erlangs concurency model, a similar philosophy of robustness,
>> type safety, all batteries included, and performance. Equating the two
>> languages works in 1 or 2 dimensions but not on all the dimensions
>> google cares about.
>>
>> reply
>>
>> derefr 1 day ago | link Interesting, thanks for that; it's pretty much
>> what I guessed (especially the bit about the supervision tree and
>> hot-code-upgrade advantages being mooted by your infrastructure.) On a
>> tangent, though: > What it does not have is type safety I've tried to
>> work this out before (I'm designing a new language for Erlang's VM), but
>> as far as I can tell, type safety is in practice incompatible with
>> VM-supported hot code upgrade. If you have two services, A and B, and
>> you need to upgrade them both, but you can't "stop the world" to do an
>> atomic upgrade of both A and B together (because you're running a
>> distributed soft real-time system, after all), then you need to switch
>> out A, and then switch out B. So, at some point, on some nodes, A will
>> be running a version with an ABI incompatible with B. In a
>> strongly-typed system, the VM wouldn't allow A's new code to load, since
>> it refers to functions in B with type signatures that don't exist. On
>> the other hand, in a system with pattern-matching and a "let it crash"
>> philosophy, you just let A's new code start up and repeatedly
>> try-and-fail to communicate with B for a while, until B's code gets
>> upgraded as well--and now the types are compatible again. It's an
>> interesting problem.
>>
>> reply
>>
>> laureny 1 day ago | link > type safety is in practice incompatible with
>> VM-supported hot code upgrade. That's not true. First, it's very easy to
>> hot reload changes that have been made to the code that are backward
>> compatible. The JVM spec describes in very specific details what that
>> means (adding or removing a method is not backward compatible, modifying
>> a body is, etc...). This is how Hotswap works, the JVM has been using it
>> for years. As for changes that are backward incompatible, you can still
>> manage them with application level techniques, such as rolling out
>> servers or simply allow two different versions of the class to exist at
>> the same time (JRebel does that, as do other a few other products in the
>> JVM ecosystem). Erlang doesn't really have any advantages over
>> statically typed systems in the hot reload area, and its lack of static
>> typing is a deal breaker for pretty much any serious production
>> deployment.
>>
>> reply
>>
>> rdtsc 20 hours ago | link > lack of static typing is a deal breaker for
>> pretty much any serious production deployment. Are you talking about
>> Google only where they made it a mandate or in general? There are
>> serious production deployments on Python, Ruby, Erlang and Javascript. I
>> will trade expressiveness and less lines of code with a strong but
>> dynamically typed language + tests over more a static typed language
>> with more lines of code all being equal. Or put it another way, if
>> strong typing is the main thing that protects against lack of faults and
>> crashes in production, there is a serious issue that needs to be
>> addressed (just my 2 cents).
>>
>> reply
>>
>> derefr 23 hours ago | link > As for changes that are backward
>> incompatible, you can still manage them with application level
>> techniques, such as rolling out servers or simply allow two different
>> versions of the class to exist at the same time (JRebel does that, as do
>> other a few other products in the JVM ecosystem). Neither of these allow
>> for the whole reason Erlang has hot code upgrade in the first place:
>> allowing to upgrade the code on one side of a TCP connection without
>> dropping the connection to the other side. Tell me how to do that with a
>> static type system :)
>>
>> reply
>>
>> pmahoney 10 hours ago | link Tomcat (and other app servers) has support
>> for doing hot reloads of Java web apps while not reloading the HTTP
>> layer (and not dropping TCP connections).
>> http://www.javacodegeeks.com/**2011/06/zero-downtime-**deploymen.<http://www.javacodegeeks.com/2011/06/zero-downtime-deploymen.>..
>> I have
>> implemented a similar system for JRuby apps running inside a Servlet
>> container. There are many caveats. I don't actually recommend it because
>> for a while you're using nearly twice the memory (and JRuby is
>> particularly memory hungry). Also there are many ways to leak the old
>> class definitions such that they are not GC'd (e.g. thread locals). But
>> it's certainly possible. I suspect that Erlang, Java, and all languages
>> are in the same boat: some parts can be upgraded live in the VM while
>> other parts require a full restart (maybe coordinating with multiple
>> nodes and a load balancer to achieve zero-downtime).
>>
>> reply
>>
>> lenkite 10 hours ago | link Out of curiosity, where/why would such an
>> exotic feature be needed in today's internet architectures where you
>> always front a group of servers with a load balancer ?
>>
>> reply
>>
>> butterflyhug 5 hours ago | link Not all Internet protocols are HTTP. If
>> you're running a service where long-lived connections are the norm,
>> "simply fronting a bunch of servers with a load balancer" can require a
>> pretty smart load balancer. E.g. IMAP connections often last hours or
>> even days, and are required to maintain a degree of statefulness.
>>
>> reply
>>
>> DanWaterworth 1 day ago | link Go gives you Erlangs concurency model
>> There are a number of significant differences between Erlang's and Go's
>> concurrency models: Asynchronous vs synchronous communication,
>> per-thread vs per-process heaps, send to process vs send to channel.
>>
>> reply
>>
>> dbaupp 1 day ago | link Go doesn't have strong type safety either; I
>> remember a recent story about a Go stdlib function "accidentally"
>> calling an interface it shouldn't.
>>
>> reply
>>
>> f2f 22 hours ago | link it wasn't accidental -- it was written on
>> purpose by a programmer (a conversion from Writer to WriteCloser). it
>> was immediately acknowledged as an error and eventually may be caught by
>> the standard code examining tool "vet".
>>
>> reply
>>
>> pcwalton 22 hours ago | link What would the static analysis that "vet"
>> is performing enforce to stop this? No interface-to-interface downcasts?
>>
>> reply
>>
>> .....
>> [It goes on, & on, & on & on,
>> when Go & Erlang party together.]
>>
>> ---  Join the BerkeleyTIP-Global mail list -   http://groups.google.com/*
>> *group/BerkTIPGlobal <http://groups.google.com/group/BerkTIPGlobal>. All
>> Freedom SW, HW & Culture.
>>
>> ______________________________**_________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/**listinfo/erlang-questions<http://erlang.org/mailman/listinfo/erlang-questions>
>>
>>
>
> --
> Loïc Hoguin
> Erlang Cowboy
> Nine Nines
> http://ninenines.eu
>
> ______________________________**_________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/**listinfo/erlang-questions<http://erlang.org/mailman/listinfo/erlang-questions>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130820/f08ca242/attachment.htm>