[erlang-questions] What is best way to maintain state in a web-based multiplayer game?

[Lloyd R. Prentice]

Hi Knut,

Your experience is truly impressive! I just reviewed the Wooga site and many of the slides on the backend. An education in itself.

I misspoke, I think, when I called my project a multiplayer game. It's really a solitary game played, I hope, by many. I say game, but it's really serious business from the perspective of the player. Thus, the simple case that you noted would apply.

I doubt that I will have the latency issues confronted by many of the Wooga games. Player state data, on the other hand, will be precious to the player and must be protected like Crown Jewels at my end. I'll also be providing a set of tools that must work together seamlessly. 

Knut, I much appreciate your thoughtful response. I hope I can call on you if I paint myself into a corner.

All the best,

Lloyd

Sent from my iPad

> On Jul 15, 2014, at 4:38 PM, Knut Nesheim <knutin@REDACTED> wrote:
> 
> Hey,
> 
> I've had the great pleasure to build a few such systems and help out
> other teams with their systems. Some of my colleagues and me have
> presented about this exact topic over the last few years at the Erlang
> Factory in London, San Francisco, Berlin and the Erlang User
> Conference in Stockholm. Here's one such presentation:
> http://www.erlang-factory.com/upload/presentations/469/wooga-euc2011-v2-111103095416-phpapp01.pdf
> 
> If each user can only affect state owned by that user, you have your
> parallelism and state management right there. Simplest case is to
> store all state in a database (relational or not) and on every
> request, load up all or parts, mutate it and write it back.
> 
> If you're expecting intensive (1 request every few seconds) and last
> long (minutes to hours) it makes sense to try to be smarter. You could
> for example load up the user state at session start in an
> (unsupervised) Erlang process, mutate it in memory (using purely
> functional code and plain Erlang data structures) and when the session
> ends write the full state back and stop the process. You could do this
> with any of the database options: relational, document or plain
> key-value. If you want to make life easy for yourself, going for a
> service provided by let's say Amazon like S3 would be a great choice.
> Scaling the application servers are then mostly about coordinating
> sessions (static sharding or distributed coordination, we eventually
> ended up writing a distributed lock: https://github.com/wooga/locker,
> presentation here:
> http://www.erlang-factory.com/conference/SFBay2013/speakers/KnutNesheim)
> Pro tip: if using plain key-value, use a serialization format easy to
> digest by other tools, such a json or protobuffers or whatever floats
> your boat.
> 
> If your game is a multiplayer game where each user can affect the
> state of other users (or a shared world), the problem is much more
> difficult. I've had the pleasure to work on one such game. Each user
> was separate, but users could join the world of their friends and mess
> around. This is a much easier problem than the
> one-single-huge-persistent world problem. We solved this by having one
> Erlang process for the world of a user, one process for each user and
> a client that understood how to roll back state updates (needed for
> conflict resolution). As the world is single-threaded and processes
> events in the order they arrive, that's the order that is used to
> resolve conflicting updates. Ie, if we both pick up an apple from the
> ground at the same time we will see the UI of the game updating, but
> only the user that reaches the server first (ie faster network) will
> actually receive the apple and when the second user reaches the server
> there's no apple on the ground any longer. The client needs to figure
> out a graceful way of rolling back the state (to avoid a full reload)
> in a nice way (to avoid pissing off users). We also presented this
> system: see https://www.youtube.com/watch?v=AUeFutXPojU and
> http://techmeshconf.com/techmesh-london-2012/presentation/You%20are%20not%20Alone%20-%20Scaling%20Multiplayer%20Games%20for%20the%20Web
> 
> In the case of MMO, RTS or FPS games they use the same concept as
> outlined above, but with some important variations. The worlds are
> always sharded, sometimes this is visible to the user (you need to
> join a particular "instance" of a world, you pick a map to play, the
> world is restricted to only one round, there's a limit on players, the
> game gets slow when many players join, etc) or it's not explicitly
> visible, but smoothed over as "you can't see or affect users really
> really far away" and when you move there, you're actually moving
> shards. Conflict resolution is very important, you want clients to
> concurrently proceed and only later resolve the conflict, if any.
> Conflict in this case might be player death in a FPS game, in which
> case you need consider the effect of latency on player states. For
> example, if I'm moving in a figure 8 shape users looking at me will
> see me at different places at exactly the same wallclock time
> depending on the latency of their connection. If a user shoots me, it
> might be that in their version of the world the game engine considers
> it a hit, but in my version of the world it was a miss. One way to
> solve this is that the server keeps track of the game state of each
> client and runs the game logic with the state the shooting client
> should have seen.
> 
> It's a very interesting topic! Good luck with your game!
> 
> Knut
> 
> 
>> On Tue, Jul 15, 2014 at 8:42 PM,  <lloyd@REDACTED> wrote:
>> Hi Felix,
>> 
>> 
>> 
>> Thanks for helping me think this through.
>> 
>> 
>> 
>> - Players do not interact. Each player is playing against the "real world,"
>> e.g. results of their actions. Think of it as project planning with many
>> many contingencies. I'd say "expert system," but that sounds rather hoity
>> toity. I say "game" only in the sense that there are reward points for
>> successful action.
>> 
>> 
>> 
>> Loop is basically:
>> 
>> 
>> 
>> - gather facts
>> 
>> - make plan
>> 
>> - take action
>> 
>> - evaluate results
>> 
>> 
>> 
>> - There are universal resources accessed by player state but these, as a
>> whole, are static text, links, and images; wild-eyed estimate: 10 megabytes
>> 
>> - I estimate that each player will store less than three megabytes of
>> private data.
>> 
>> 
>> 
>> - We're not dealing with elaborate graphics or animations. Only latency is a
>> logic tree or inference engine evaluating facts; e.g. (Looking at Erlang
>> eres/seres; but I may be able get away with something more simple minded).
>> 
>> 
>> 
>> Latency, in other words, is a function of how many private and shared
>> database lookups are necessary to evaluate a set of condition -> action
>> rules. Effective state management should help me keep rule sets fairly
>> minimal.
>> 
>> 
>> 
>> Prospects for this game are in the 200K range, but I'm successful with low
>> hundreds of players.
>> 
>> 
>> 
>> Ad hoc querying is not necessary.
>> 
>> 
>> 
>> Thanks again for your help.
>> 
>> 
>> 
>> Lloyd
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> -----Original Message-----
>> From: "Felix Gallo" <felixgallo@REDACTED>
>> Sent: Tuesday, July 15, 2014 1:50pm
>> To: "Lloyd R. Prentice" <lloyd@REDACTED>
>> Cc: "Erlang-Questions Questions" <erlang-questions@REDACTED>
>> Subject: Re: [erlang-questions] What is best way to maintain state in a
>> web-based multiplayer game?
>> 
>> I think you have some more marble to cut away before the shape of the
>> problem begins to reveal itself.
>> * do the players interact with each other directly?  If so, what state do
>> they share and how much?  Is there a separation between 'state for everyone'
>> (e.g. world of warcraft: the entire world geometry and all 3d models are
>> gigabytes of static data that are stored on the local filesystem and not
>> kept 'live' in memory) and 'player state'?  What is the size and nature of
>> each type of state?
>> * what are the constraints around latency -- does it look more like a real
>> time game (e.g. street fighter, counterstrike, league of legends) where,
>> e.g., 250 ms of latency is unacceptable to the experience, or more like a
>> turn based game (e.g. chess, poker, backgammon) where latency of 3s is
>> tolerable?  This drives certain considerations; e.g., you may be able to use
>> a slightly lower performance but more reliable database system for the
>> latter case.
>> 
>> * what is the shape and nature of your cost envelope?  If you are running
>> server infrastructure for a game, then you may need to take special measures
>> to ensure that your costs remain beneath your gross profit.
>> Since, as you say, managing game state is such a giant pain in the ass, my
>> general best practice when I do exactly this thing as part of my day job is
>> to first write down all of the different kinds of state, their approximate
>> quantity in bytes per player/world/universe/shard/whatever, and then do some
>> spreadsheet math to figure out what I can get away with.
>> Example: Game is Facebook-style web game like Farmville.  State can be
>> boiled down to a binary data structure that is 4 kilobytes in size per
>> player.  Expected profit per user is on the order of pennies.  Tens of
>> millions of users expected.  Solution: save state in the user's own browser
>> via encrypted cookie, use tiny database (sqlite, mysql, etc.) for
>> leaderboards and sending seeds between players.
>> Example: Game is World of Warcraft style MMO but you're trying to do it in
>> the browser so can't download gigabytes.  So you must shard each zone into,
>> say, a 50 megabyte downloadable size.  Zone shards are static so can be
>> files.  Private player data (e.g. inventory, raid history, message log) is
>> maybe 1 megabyte; public player data (e.g., x/y/z coordinates, avatar skin)
>> is maybe 30K and shared across the entire world (but usually limited to a
>> shard), needs fast access though as the world ticks at 30 frames per second
>> and you need to do distance calculations, etc.  Store the public player data
>> in memory, dets (per shard) or redis depending on whether you want raw
>> speed, resiliency, or tooling.  Store the private data in memory, dets,
>> redis, or mysql, depending on speed, resiliency, tooling, cost and analytics
>> requirements.
>> Generally I use ets for fast-access random-lookup state that has to get
>> shared between processes, redis for fast-enough (still plenty fast)
>> random-lookup state that has to be able to get large and/or survive system
>> crashes or writeoffs (via slave replication and aof, rather than
>> clustering), mysql for mass market mediocre state storage where the client
>> has pre-existing infrastructure and ad hoc querying is important, postgres
>> for green fields state storage where ad hoc querying is important, and
>> thankfully so far I have been able to avoid having to use inconsistent data
>> stores in production.  I wouldn't ever consider dets or mnesia because
>> equivalently-functional tools exist that are radically more popular and have
>> better tooling, capability, characteristics or safety guarantees.
>> F.
>> 
>> 
>> On Tue, Jul 15, 2014 at 10:03 AM, Lloyd R. Prentice <lloyd@REDACTED>
>> wrote:
>>> 
>>> 
>>> I'm thinking through the design of a web-based multiple-player game. Each
>>> player maintains a private database--- most likely dets. Each player moves
>>> through nested finite state machines. Each player may be logging in or out
>>> at any state of play.
>>> 
>>> At this point my grasp of Erlang architecture breaks down. I don't
>>> understand how best to:
>>> 
>>> - parse game structure across players, processes, and directories
>>> - maintain player state between sessions
>>> 
>>> At this point in my Erlang education I feel like I know the words but
>>> can't play the music.
>>> 
>>> I'd much grateful for any and all ideas and suggestions.
>>> 
>>> Many thanks,
>>> 
>>> LRP
>>> 
>>> Sent from my iPad
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>> 
>> 
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>