[erlang-questions] Help creating distributed server cache

[David Fox]

Hi Garret, thanks for the response.

I have not yet finished implementing the API; I'm still in the design 
phase figuring out how everything should be hooked up.

Totally agree on not limiting yourself to options this early on, I'm 
just asking for some help and opinions on some problems I saw in 
potential implementations :)

David Fox
m: 630 930 9219
Chicago

On 12/6/2012 16:34, Garrett Smith wrote:
> Hi David,
>
> On Thu, Dec 6, 2012 at 9:43 PM, David Fox <david@REDACTED> wrote:
>> I'm currently developing a gaming server which stores player information
>> that can be accessed from any of our games via a REST API.
>>
>> So far I've thought of two ways to structure and cache player data:
>>
>> 1. When a client requests data on a player, spawn 1 player process. This
>> process handles: all subsequent requests from clients for this player,
>> retrieving the player data from the DB when created and periodically
>> updating the DB with any new data from clients. If the player is not
>> requested by another client within... say 30 minutes, the player process
>> will terminate.
>>
>> 2. Just keep previously requested data in a distributed LRU cache (e.g.,
>> memcached, redis, mnesia)
>>
>> Out of the two, I prefer #1 since it would allow me to separate the
>> functionality of different "data types" (e.g., player data, game data).
>>
>> There are just 2 problems with doing it this way that I'd like your thoughts
>> and help with:
>> I. I would have to implement some sort of "LRU process cache" so I could
>> terminate processes to free memory for new ones.
>> II. If a load balancer connects a client to node #1, but the process for the
>> requested player is on node #2, how can the player process on node #2 send
>> the data to the socket opened for the client on node #1. Is it possible to
>> somehow send an socket across nodes? The reason I ask, is that I'd like to
>> prevent sending big messages across nodes.
> It's tough to answer high level "approach" style questions, especially
> without some hands on work (tinkering) to help you understand the
> problem.
>
> Limiting yourself to either/or options at this stage might also be premature.
>
> Do you have a first pass at the public API for this service?
>
> If you have an idea of the functions that could define the interface,
> you can ask, for each unimplemented function:
>
> - Can I make this side-effect free -- i.e. calling the function
> doesn't change state or otherwise tamper with the universe?
>
> - Does the function read from or write to long running state?
>
> Side effect free functions are easy, which is why you try to solve
> problems using them exclusively whenever possible.
>
> For long running state, you can use a simple gen_server to implement
> state initialization and mutation. If you have questions about what I
> mean here, you'll need to bone up on gen_server, or alternatively look
> at e2 services (see http://e2project.org) as they're simpler to write.
>
> Once you have something very basic working, see if you're done! It
> might just work for you as is, at least for the short term. If it
> doesn't work, address the specific problem. E.g. if your problem is "I
> loose my state when the VM crahses," you'll need to implement
> persistence in some fashion.
>
> Questions at that level are much easier to answer :)
>
> Garrett