[erlang-questions] State Management Problem

[Robert Virding]

I just want to briefly comment Craig's discussion about some of the reasons
behind why Erlang looks like it does. We were out to solve a rather
specific problem domain: telecom systems. These had some basic properties
which any solution had to tackle: massive concurrency, timing constraints,
and fault tolerance where parts were allowed to fail (but not too often of
course) as long as the system as a whole did not crash. What became Erlang
was our way of the solving the problem.

Everything was very problem oriented and we did not have as goals that
Erlang should be a functional language or that we should implement the
actor model. We knew nothing of the actor model until later when we heard
that Erlang implements it. :-) The design of the language evolved together
with our ideas on how language features would be used to build systems with
these properties, which was the base of what became OTP. Processes were a
good solution to concurrency. Asynchronous communication was a natural as
it fitted both the problem and was a natural for building non-blocking
systems. How else would you do it? The error handling primitives made it
easy to take down groups of cooperating processes as well as monitoring
them so you could clean-up after them or restart them when necessary. We
also had the goal to stick to the basics and to keep the language simple.
Simple in the sense that there should be a small number of basic
principles, if these are right then the language will be powerful but easy
to comprehend and use. Small is good. This we managed to do.

Again our goal was to solve the problem, not design a language with a
predefined set of primitives.

That Erlang/OTP is so successful for many other types of systems is that
these also have similar requirements to what we were looking at. We weren't
thinking of this at the start. IMAO one reason that Erlang/OTP has taken a
long time to become adopted is that in many cases it took designers of
these other systems a long to realise that they actually did have these
requirements and that Erlang was a good way of meeting them.

Simple is difficult, complex is easy.

Robert


On 21 December 2015 at 14:56, zxq9 <zxq9@REDACTED> wrote:

> On 2015年12月21日 月曜日 13:11:59 you wrote:
> > I think I see your point. I will definitely think about it using all the
> > knowledge that I have. Databases do solve the problem but I think that
> for
> > real time applications, going to a database would be slow. The part to
> > think about for me, would be to come up with a useful API. Thanks for
> your
> > inputs.
>
> By the way, I've been down the path you are contemplating. I think
> everyone needs to do this once to really understand how tools fit problems.
> That is to say, I will probably never trust a system architect on a
> multi-faceted project unless they've gone through this once themselves.
>
> Now I tend to write an Erlang service as an application service ("Oh no!
> Sooooo old fashioned!"). Clients perceive that service as the mother of all
> data operations, but they are high-level operations -- operations where the
> verb parts are relevant to the high-level business problem and the noun
> parts are themselves business-problem-level entities.
>
> Lower level operations (permanent storage, specialized indexing,
> specialized queries, etc.) still happen in database type systems behind the
> scenes. Very often that's Postgres as a canonical store, but the Erlang
> application service has a "live" cache of whatever data is active and often
> a request for data never makes it to the database because it already is
> live in the application server. Sometimes a graphing database is involved
> for super-fast queries over otherwise difficult queries -- but you might be
> surprised at how much you can lean on Postgres itself without taking a
> performance hit (and when you do have slow queries Postgres has a plethora
> of great tools available to figure things out and tune). Sometimes a
> separate copy of the data is in a document, text, image, geometric, or
> graph db (or separate denormalized tables or even specialized tablespaces)
> because certain searches are just plain hard to optimize for in the general
> case. When you really need this sort of extra help, though, it is usually
> painfully obvious -- so never start out that way, instead get the
> business-level logic right first.
>
> Typically clients deal with a *very* small fraction of the total data in
> the storage backend at any given moment, and they keep hitting that tiny
> fraction over and over. This tends to be true whether its game or business
> data, actually. (Surprising how similar they are.) Old data tends to not be
> of any interest, so there is this small percentage of request churn over
> whatever the going issue of the day happens to be (new contracts, open
> estimates, current projects, new hr data, some specific financial report,
> this month's foo, the latest dungeon content, highest-ranking player
> states, etc.).
>
> There are major advantages to warehousing data in a DBA-approved
> relational sort of way. It makes warehousing issues *much* easier to deal
> with (denormalizing a copy of the data for some specific purpose, for
> example) than trying to take everything out of a gigantic K-V store or
> super-super performant but split personality "webscale" db and normalizing
> it after the fact (protip: you will have all sorts of random loose ends
> that are insanely hard to figure out, no matter what sort of "semantic
> tagging" system you *think* you've invented -- sometimes this is so bad
> that whatever analytic results your client is trying to discover turn out
> totally wrong).
>
> So it still comes down to different tools for different uses. That active,
> high-interest data is a *perfect* fit for ETS tables and/or process state
> (depending on the case -- both are crazy fast, but you don't want every
> item in a game with 500M active items to be a process). But this data
> should always be rebuildable from your backend database state that comes
> from a stodgy old, DBA-endorsed, safety-first sort of database where data
> security is well understood and large community of experts exists. When you
> start your application service up, the first requests are what populate
> your ETS tables and cause processes to spawn. When resources are unneeded
> those caches shrink (processes exit, tables shrink).
>
> Having clients talk to the application service instead of an ORM or the
> data backends directly lets you forget that there is a MASSIVE PROBLEM with
> ORM frameworks (for anything more interesting than, say, a blog website
> framework), because you will still have to write a translation between your
> application representation and the database representation. This is
> probably the most tedious part of writing the whole system -- but if you
> don't do this you will wind up re-inventing every bit of the transactional,
> relational, navigation-capable, document and object database paradigms all
> mishmashed together into an incoherent, buggy, half-baked API without
> realizing it (until its too late to change your mind, that is... *then*
> you'll certainly realize it).
>
> DON'T REINVENT THE DATABASE.
>
> But you'll do this once, no matter what I say. Anyone who architects and
> then implements two huge systems does this once (either the first or the
> second time). And then realize that an application-service-as-a-datasource
> can be a very fast, nice thing, and that databases are magical tools that
> you're really, really glad someone else went to the trouble to write.
>
> Also -- none of this *solves* the distributed state problem. But there is
> hope! As you write systems you'll start feeling out places where it is OK
> to partition the problem, where temporary inconsistency in the cached data
> is OK (and where not in the backend datastore), how much write lag is OK
> between the application service and the backend, what sort of queries are
> pull-your-hair-out hard or slow or slow-and-hard without a specialized
> database (hint: text search, image search and graph queries), and other
> such issues. Also -- what data is just OK to disappear POOF! when something
> goes wrong (ephemeral chat messages, for example).
>
> Its a lot to think through, and whatever tradeoffs you decide fit in
> specific spots are going to depend entirely on the problem you are trying
> to solve in *that* part of the system and the user-facing context. This is
> true in any language and any environment. Anyone who says differently has
> only ever dealt with trivial data or is a big fat liar trying to get some
> magical consultancy cash from you.
>
> None of this is to intimidate or discourage you. Have fun. (Seriously!)
> These data conundrums are some of the most delicate and interesting
> problems you'll ever encounter -- and unlike algorithmic solutions to
> procedural problems that translate readily to arithmetic, data problems are
> *never* fully "solved". (Which also means this is a rabbit hole you can
> lose your entire career/mind in... forever!)
>
> -Craig
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20151223/0f455b2f/attachment.htm>