[erlang-questions] What is best way to maintain state in a web-based multiplayer game?

[Knut Nesheim]

Hey,

I've had the great pleasure to build a few such systems and help out
other teams with their systems. Some of my colleagues and me have
presented about this exact topic over the last few years at the Erlang
Factory in London, San Francisco, Berlin and the Erlang User
Conference in Stockholm. Here's one such presentation:
http://www.erlang-factory.com/upload/presentations/469/wooga-euc2011-v2-111103095416-phpapp01.pdf

If each user can only affect state owned by that user, you have your
parallelism and state management right there. Simplest case is to
store all state in a database (relational or not) and on every
request, load up all or parts, mutate it and write it back.

If you're expecting intensive (1 request every few seconds) and last
long (minutes to hours) it makes sense to try to be smarter. You could
for example load up the user state at session start in an
(unsupervised) Erlang process, mutate it in memory (using purely
functional code and plain Erlang data structures) and when the session
ends write the full state back and stop the process. You could do this
with any of the database options: relational, document or plain
key-value. If you want to make life easy for yourself, going for a
service provided by let's say Amazon like S3 would be a great choice.
Scaling the application servers are then mostly about coordinating
sessions (static sharding or distributed coordination, we eventually
ended up writing a distributed lock: https://github.com/wooga/locker,
presentation here:
http://www.erlang-factory.com/conference/SFBay2013/speakers/KnutNesheim)
Pro tip: if using plain key-value, use a serialization format easy to
digest by other tools, such a json or protobuffers or whatever floats
your boat.

If your game is a multiplayer game where each user can affect the
state of other users (or a shared world), the problem is much more
difficult. I've had the pleasure to work on one such game. Each user
was separate, but users could join the world of their friends and mess
around. This is a much easier problem than the
one-single-huge-persistent world problem. We solved this by having one
Erlang process for the world of a user, one process for each user and
a client that understood how to roll back state updates (needed for
conflict resolution). As the world is single-threaded and processes
events in the order they arrive, that's the order that is used to
resolve conflicting updates. Ie, if we both pick up an apple from the
ground at the same time we will see the UI of the game updating, but
only the user that reaches the server first (ie faster network) will
actually receive the apple and when the second user reaches the server
there's no apple on the ground any longer. The client needs to figure
out a graceful way of rolling back the state (to avoid a full reload)
in a nice way (to avoid pissing off users). We also presented this
system: see https://www.youtube.com/watch?v=AUeFutXPojU and
http://techmeshconf.com/techmesh-london-2012/presentation/You%20are%20not%20Alone%20-%20Scaling%20Multiplayer%20Games%20for%20the%20Web

In the case of MMO, RTS or FPS games they use the same concept as
outlined above, but with some important variations. The worlds are
always sharded, sometimes this is visible to the user (you need to
join a particular "instance" of a world, you pick a map to play, the
world is restricted to only one round, there's a limit on players, the
game gets slow when many players join, etc) or it's not explicitly
visible, but smoothed over as "you can't see or affect users really
really far away" and when you move there, you're actually moving
shards. Conflict resolution is very important, you want clients to
concurrently proceed and only later resolve the conflict, if any.
Conflict in this case might be player death in a FPS game, in which
case you need consider the effect of latency on player states. For
example, if I'm moving in a figure 8 shape users looking at me will
see me at different places at exactly the same wallclock time
depending on the latency of their connection. If a user shoots me, it
might be that in their version of the world the game engine considers
it a hit, but in my version of the world it was a miss. One way to
solve this is that the server keeps track of the game state of each
client and runs the game logic with the state the shooting client
should have seen.

It's a very interesting topic! Good luck with your game!

Knut


On Tue, Jul 15, 2014 at 8:42 PM,  <lloyd@REDACTED> wrote:
> Hi Felix,
>
>
>
> Thanks for helping me think this through.
>
>
>
> - Players do not interact. Each player is playing against the "real world,"
> e.g. results of their actions. Think of it as project planning with many
> many contingencies. I'd say "expert system," but that sounds rather hoity
> toity. I say "game" only in the sense that there are reward points for
> successful action.
>
>
>
> Loop is basically:
>
>
>
> - gather facts
>
> - make plan
>
> - take action
>
> - evaluate results
>
>
>
> - There are universal resources accessed by player state but these, as a
> whole, are static text, links, and images; wild-eyed estimate: 10 megabytes
>
> - I estimate that each player will store less than three megabytes of
> private data.
>
>
>
> - We're not dealing with elaborate graphics or animations. Only latency is a
> logic tree or inference engine evaluating facts; e.g. (Looking at Erlang
> eres/seres; but I may be able get away with something more simple minded).
>
>
>
> Latency, in other words, is a function of how many private and shared
> database lookups are necessary to evaluate a set of condition -> action
> rules. Effective state management should help me keep rule sets fairly
> minimal.
>
>
>
> Prospects for this game are in the 200K range, but I'm successful with low
> hundreds of players.
>
>
>
> Ad hoc querying is not necessary.
>
>
>
> Thanks again for your help.
>
>
>
> Lloyd
>
>
>
>
>
>
>
> -----Original Message-----
> From: "Felix Gallo" <felixgallo@REDACTED>
> Sent: Tuesday, July 15, 2014 1:50pm
> To: "Lloyd R. Prentice" <lloyd@REDACTED>
> Cc: "Erlang-Questions Questions" <erlang-questions@REDACTED>
> Subject: Re: [erlang-questions] What is best way to maintain state in a
> web-based multiplayer game?
>
> I think you have some more marble to cut away before the shape of the
> problem begins to reveal itself.
> * do the players interact with each other directly?  If so, what state do
> they share and how much?  Is there a separation between 'state for everyone'
> (e.g. world of warcraft: the entire world geometry and all 3d models are
> gigabytes of static data that are stored on the local filesystem and not
> kept 'live' in memory) and 'player state'?  What is the size and nature of
> each type of state?
> * what are the constraints around latency -- does it look more like a real
> time game (e.g. street fighter, counterstrike, league of legends) where,
> e.g., 250 ms of latency is unacceptable to the experience, or more like a
> turn based game (e.g. chess, poker, backgammon) where latency of 3s is
> tolerable?  This drives certain considerations; e.g., you may be able to use
> a slightly lower performance but more reliable database system for the
> latter case.
>
> * what is the shape and nature of your cost envelope?  If you are running
> server infrastructure for a game, then you may need to take special measures
> to ensure that your costs remain beneath your gross profit.
> Since, as you say, managing game state is such a giant pain in the ass, my
> general best practice when I do exactly this thing as part of my day job is
> to first write down all of the different kinds of state, their approximate
> quantity in bytes per player/world/universe/shard/whatever, and then do some
> spreadsheet math to figure out what I can get away with.
> Example: Game is Facebook-style web game like Farmville.  State can be
> boiled down to a binary data structure that is 4 kilobytes in size per
> player.  Expected profit per user is on the order of pennies.  Tens of
> millions of users expected.  Solution: save state in the user's own browser
> via encrypted cookie, use tiny database (sqlite, mysql, etc.) for
> leaderboards and sending seeds between players.
> Example: Game is World of Warcraft style MMO but you're trying to do it in
> the browser so can't download gigabytes.  So you must shard each zone into,
> say, a 50 megabyte downloadable size.  Zone shards are static so can be
> files.  Private player data (e.g. inventory, raid history, message log) is
> maybe 1 megabyte; public player data (e.g., x/y/z coordinates, avatar skin)
> is maybe 30K and shared across the entire world (but usually limited to a
> shard), needs fast access though as the world ticks at 30 frames per second
> and you need to do distance calculations, etc.  Store the public player data
> in memory, dets (per shard) or redis depending on whether you want raw
> speed, resiliency, or tooling.  Store the private data in memory, dets,
> redis, or mysql, depending on speed, resiliency, tooling, cost and analytics
> requirements.
> Generally I use ets for fast-access random-lookup state that has to get
> shared between processes, redis for fast-enough (still plenty fast)
> random-lookup state that has to be able to get large and/or survive system
> crashes or writeoffs (via slave replication and aof, rather than
> clustering), mysql for mass market mediocre state storage where the client
> has pre-existing infrastructure and ad hoc querying is important, postgres
> for green fields state storage where ad hoc querying is important, and
> thankfully so far I have been able to avoid having to use inconsistent data
> stores in production.  I wouldn't ever consider dets or mnesia because
> equivalently-functional tools exist that are radically more popular and have
> better tooling, capability, characteristics or safety guarantees.
> F.
>
>
> On Tue, Jul 15, 2014 at 10:03 AM, Lloyd R. Prentice <lloyd@REDACTED>
> wrote:
>>
>>
>> I'm thinking through the design of a web-based multiple-player game. Each
>> player maintains a private database--- most likely dets. Each player moves
>> through nested finite state machines. Each player may be logging in or out
>> at any state of play.
>>
>> At this point my grasp of Erlang architecture breaks down. I don't
>> understand how best to:
>>
>> - parse game structure across players, processes, and directories
>> - maintain player state between sessions
>>
>> At this point in my Erlang education I feel like I know the words but
>> can't play the music.
>>
>> I'd much grateful for any and all ideas and suggestions.
>>
>> Many thanks,
>>
>> LRP
>>
>> Sent from my iPad
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>