[erlang-questions] Granularity of process (and module) identification. Was: Supervision strategies to automatically restart dynamically added children

Edmond Begumisa <>
Wed Mar 9 13:03:43 CET 2011


Forgot the all important new_game...

On Wed, 09 Mar 2011 22:03:07 +1100, Edmond Begumisa  
<> wrote:

> On Wed, 09 Mar 2011 09:56:02 +1100, Lukas Larsson  
> <> wrote:
>
>> You can still do a logical separation between a player and a game, it  
>> does not have to be physical. By creating an opaque datatype for each  
>> player which you only can access through the player module you put a  
>> separation in between the two concepts but still keep the same  
>> concurrent activities as there are in the world. i.e.
>>
>> -module(player).
>> %% Private record
>> -record(player, {name}).
>>
>> create() ->
>>   #player{}.
>>
>> set_name(Name, Player) ->
>>   Player#player{ name = Name }.
>>
>> get_name(Player) ->
>>   Player#player.name.
>>
>> etc.
>>
>> This makes it (atleast for me) possible to enforce a logical model upon  
>> my physical restrictions which makes my code easier to read and also to  
>> maintain when upgrading.
>>
>> Lukas
>> ----- Original Message -----
>> From: "Dhananjay Nene" <>
>> To: 
>> Sent: Tuesday, March 8, 2011 10:18:49 PM GMT +01:00 Amsterdam / Berlin  
>> / Bern / Rome / Stockholm / Vienna
>> Subject: [erlang-questions] Granularity of process (and module)  
>> identification. Was: Supervision strategies to automatically restart  
>> dynamically added children
>>
>> On Tue, Mar 8, 2011 at 4:13 PM, Edmond Begumisa
>> <> wrote:
>>> On Tue, 08 Mar 2011 17:52:29 +1100, Dhananjay Nene
>>> <> wrote:
>>>
>>>> On Mon, Mar 7, 2011 at 5:08 AM, Edmond Begumisa
>>>> <> wrote:
>>>>>
>>>>> Hi Dhananjay,
>>>>>
>>>>> I too struggled with this exact question for quite some time so I'll
>>>>> chime
>>>>> in here on the two techniques I used to solve it...
>>>>> On Thu, 03 Mar 2011 05:02:06 +1100, Dhananjay Nene
>>>>> <> wrote:
>>>>>
>>>>>>
>>>>>> Question in short : If I have a supervisor which has a number of
>>>>>> dynamic children, how do I set up a mechanism where in case of a
>>>>>> complete system crash, all the dynamic children restart at the point
>>>>>> they were when the system (including the supervisor) crashed.
>>>>>>
>>>>>> Question in long :
>>>>>> =============
>>>>>>
>>>>>> Sample Context : A bowling game
>>>>>> -------------------------------------------------
>>>>>>
>>>>>> Lets say I am writing the software to implement the software  
>>>>>> necessary
>>>>>> to track various games at a bowling alley. I've set up the following
>>>>>> processes :
>>>>>>
>>>>>> a. Lanes : If there are 10 lanes, there are 10 processes, one for  
>>>>>> each
>>>>>> lane. These stay fixed for the entire duration of the program
>>>>>> b. Games : A group of players might get together to start a game on  
>>>>>> a
>>>>>> free lane. A new game will get created to track the game through its
>>>>>> completion. When the game is over, this process shall terminate
>>>>>> c. Players : Each game has a number of players. One process
>>>>>> "player_game" is started per player. Sample state of a player game
>>>>>> would include current score for the player and if the last two rolls
>>>>>> were strike or a spare. For the purpose of brevity, the remainder of
>>>>>> this mail only refers to this process and ignores the others
>>>>>>
>>>>>
>>>>> You could reduce complexity by having each lane process maintain it's
>>>>> current game (players and scores) as part of it's state. The game and
>>>>> player_game processes appear unnecessarily confusing to me.
>>>>>
>>>>
>>>> Interesting point. The lanes are the only static aspects of the game.
>>>> I tried to consider whether it would make any difference from a client
>>>> API perspective, but I imagine for a client, there is no particular
>>>> reason to believe a lane is a better or worse abstraction than a game
>>>> (or a player_game).
>>>>
>>>>>> Objective :
>>>>>> ---------------
>>>>>>
>>>>>> Assuming this is a single node implementation, if the machine were  
>>>>>> to
>>>>>> crash, upon machine / node restart, all the player_games should be
>>>>>> restarted and should be at the point where the player_games were  
>>>>>> when
>>>>>> the machine crashed.
>>>>>>
>>>>>> Possible supervision strategy :
>>>>>> --------------------------------------
>>>>>>
>>>>>> 1. Create a simple_one_for_one supervisor player_game_sup which upon
>>>>>> starting up for the first time would have no children associated  
>>>>>> with
>>>>>> them. Use supervisor:start_child to start each process
>>>>>> 2. The supervisor creates an entry in a database (say mnesia) every
>>>>>> time it launches a new process
>>>>>> 3. Each player_game updates the entry every time the score gets
>>>>>> modified. Upon termination that entry gets deleted
>>>>>> 4. Post crash, the supervisor is started again (say after an
>>>>>> application restart or via another supervisor)
>>>>>> 5. (Here's the difference). By default the supervisor will not  
>>>>>> restart
>>>>>> the dynamically added children (all the player_games). However we
>>>>>> modify the init code to inspect the database and launch a  
>>>>>> player_game
>>>>>> for each record it finds.
>>>>>
>>>>> How? I don't think you can instruct a simple_one_for_one supervisor  
>>>>> to
>>>>> create children from it's init/1 callback. From the documentation...
>>>>>
>>>>> http://www.erlang.org/doc/man/supervisor.html#Module:init-1
>>>>>
>>>>> "...No child process is then started during the initialization  
>>>>> phase, but
>>>>> all children are assumed to be started dynamically using
>>>>> supervisor:start_child/2..."
>>>>
>>>> Fair point. Wasn't something that struck me as an issue then, but yes,
>>>> supervisor starting dynamic children inside init doesn't quite rock.
>>>>
>>>>> AFIAK, creating dynamic children (calling supervisor:start_child/2)  
>>>>> has
>>>>> to
>>>>> be done after the supervisor has initialised by a process other than  
>>>>> the
>>>>> supervisor process.
>>>>
>>>> Certainly. And your separate modeling of a lane_ldr (later down this
>>>> mail) helps that.
>>>>
>>>>> This is normally not a problem if you are calling start_child/2  
>>>>> during
>>>>> the
>>>>> "normal" operation of the application because the supervisor in  
>>>>> question
>>>>> is
>>>>> likely to already be up. But here, you want to call start_child/2 at
>>>>> *startup*. From my experience with this precise matter, this requires
>>>>> some
>>>>> process coordination.
>>>>>
>>>>>> The player_game initialises itself to the
>>>>>> current state as in the database and the game(s) can continue where
>>>>>> it/they left off.
>>>>>>
>>>>>> My questions :
>>>>>> --------------------
>>>>>> a. Does it make sense to move the responsibility to the supervisor  
>>>>>> to
>>>>>> update the database each time a new player game is started or
>>>>>> completed ?
>>>>>
>>>>> I personally don't see the advantage of doing this. Besides (as per  
>>>>> my
>>>>> understanding of OTP design principles), a supervisor's job should be
>>>>> just
>>>>> that -- supervising workers and not doing work itself.
>>>>>
>>>>> Doing this from the your worker gen_servers make more sense to me and
>>>>> seems
>>>>> more natural. i.e Reading the scores from the DB the during
>>>>> player_game:init
>>>>> and writing them every time a score gets bumped or something similar.
>>>>>
>>>>
>>>> I agree
>>>>
>>>>
>>>>> Possible supervision strategy 2a: (Loader version)
>>>>> --------------------------------------------------
>>>>>
>>>>> Rather than separate dynamic children for players and games as in
>>>>> Strategy
>>>>> 1, instead, each lane stores, as part of it's state, info on the  
>>>>> current
>>>>> game (the players playing on the lane and their state/scores). The
>>>>> supervision tree might look like this...
>>>>>
>>>>>          alley_sup
>>>>>         /         \
>>>>>  lane_ldr  ___lanes_sup_____
>>>>>          /       |     :   \
>>>>>       lane(1)  lane(2) .. lane(N)
>>>>>
>>>>> * Application has a startup configuration parameter no_of_lanes which
>>>>> comes
>>>>> from a conf file or the .app file and loaded by the alley_sup...
>>>>>
>>>>
>>>> This is a suggestion thats really had me thinking. I suspect there's a
>>>> bit of the traditional OO modeling experience which is grumbling about
>>>> not being able to model a game or a player game.
>>>
>>> It's not that you can't model them, it's that you don't need to.
>>>
>>> One mantra in Erlang literature (e.g. Casarini & Thompson, pg110), is  
>>> to
>>> create a process for every concurrent *activity* you observe in the  
>>> real
>>> world and not every *task* you observe. So you don't necessarily need  
>>> to use
>>> a process for every "object" you see in the real world.
>>>
>>> With this in mind, my immediate interpretation of your application was  
>>> in
>>> two ways:
>>>
>>> A)
>>>
>>> * You have a bowling alley which has lanes.
>>> * Different _lanes_ can be *concurrently* used at the same time: map  
>>> these
>>> to processes.
>>> * Only 1 player can use a _lane_ at a time: no need for player  
>>> processes.
>>> * Only 1 game can take place on a _lane_ at a time: no need for game
>>> processes.
>>> * It follows that players and their game are just the state of each
>>> concurrently used _lane_.
>>>
>>> So you only need processes for lanes.
>>>
>>> Alternatively, B)
>>>
>>> * You have a bowling alley where people play games.
>>> * Several _games_ can be *concurrently* played at the same time: map  
>>> these
>>> to processes.
>>> * Only 1 player can make a _game_ play at a time: no need for player
>>> processes.
>>> * Only 1 lane can be used per _game_: no need for lane processes.
>>> * It follows that players and their lane are just the state of each
>>> concurrently played game.
>>>
>>> So you only need processes for games.
>>>
>>> A) *might* be easier to implement than B) when you have to interact  
>>> with
>>> hardware that manages the lane machinery, which is why I suggested it.  
>>> But
>>> either way, you only need *one* class of processes. IMO, introducing  
>>> more
>>> just complicates matters unnecessarily.
>>
>> Understood. Here's a dump of my thoughts.
>>
>> Yes, we need to have upto only as many processes as the number of
>> lanes (either the lanes themselves or maximum one game per lane). Lets
>> assume we model a game as a process.
>>
>> There's a lot of state thats maintained at a player level, and very
>> little at a game level. As a goal, separation of game and player (and
>> the states) seems desirable to support these separation of concerns.
>> Perhaps we could still model these using separate modules. Yet, given
>> how state is carried over from one handle_call/handle_cast into the
>> next, not modeling a player as a process forces the internal state of
>> the player to be completely accessible to the game - most of it, it
>> simply does not need access to.
>
> Not necessarily. To reinforce what Lukas said above, it could treat the  
> player and game elements of it's state as something it doesn't  
> understand but keeps hold of. It then takes actions on these elements by  
> calling other modules player.erl and game.erl that do understand them.  
> In this case, those modules would not need to create their own processes  
> to have some seperation.
>
> I'll expand on Lukas's example coz I've found it can be hard to  
> visualise if you've come from OO...
>
> === lane.erl ===
> -behaviour(gen_server).
> ..
> init(Id) ->
>      ..
>     % We don't know what we're creating, we just know it represents a  
> game
>     {ok, game:create(Id)}.
>

hanlde_cast(new_game, OldGame) ->
      {noreply, game:reset(OldGame)};

> hanlde_cast({add_player, PlayerName}, Game0) ->
>      {noreply, game:add_player(Game0)};
> handle_cast({game_play, PlayerName, PinsDown}, Game0) ->
>      % game:play must succeed otherwise lane/game state will be incorrect
>      Game1 = game:play(Game0, PlayerName, PinsDown),
>      ok = reset_pins(),
>      {noreply, Game1}.
>
> handle_call({get_score, PlayerName}, Game) ->
>     case Reply = game:get_score(Game, PlayerName) of
>        {ok, Score} ->
>           {reply, Reply, Game}.
>        Error ->
>           {reply, Reply, Game}  % Same but just making it clear that  
> game:get_score is allowed to fail
>     end.
>
> reset_pins() ->
>    .. maybe talk to some hardware driver ..
>
> == game.erl===
> -export(create/1, add_player/2, play/3).
>
> create(LaneId) ->
>      .. create path from LaneId ..
>      AllPlayers = try read_game(PersistPath)
>                   catch
>                     _:Why ->
>                      ..
>                      []
>                   end
>      {Id,PersistPath,AllPlayers}. % Understood only by game.erl
>
reset({LaneId,PersistPath,_}) ->
       ok = write_game(PersistPath, []),
       {LaneId,PersistPath,[]).

> add_player({LaneId,PersistPath,AllPlayers0}, PlayerName) ->
>      {LaneId, PersistPath, lists:keystore(PlayerName, 1, AllPlayers0,
>                                           player:create(PlayerName)).
>
> %% Part of Jasper's error kernel -- assert happy-case
> play({LaneId,PersistPath,AllPlayers0}, PlayerName, PinsDown) ->
>      Player0 = proplist:get_value(PlayerName, AllPlayers0),
>      false = undefined == Player0,
>      AllPlayers1 = lists:keyreplace(PlayerName, 1, AllPlayers0,
>                                      player:bump_score(Player0,  
> PinsDown))},
>      ok = write_game(PersistPath, AllPlayers1),
>      {LaneId,PersistPath,AllPlayers1}.
>
> get_score({_,_,AllPlayers), PlayerName) ->
>      case proplist:get_value(PlayerName, AllPlayers) of
>          undefined ->
>              {badarg, PlayerName};
>          Player ->
>              {ok, player:get_score(Player)}
>      end.
>
> %%% Internal functions
> read_game() ->
>     ..
> write_game() ->
>     ..
>
> == player.erl ===
>
> -export(create, bump_score, get_score)
>
> %% Jasper's Error Kernel: This state and operations against it must be  
> correct.
> -record(state, {frame = 0,
>                  shot = 1,
>                  bonus_shot = false,
>                  last_shot = normal,
>                  prior_to_last_shot = normal,
>                  max_pins = 10,
>                  score = 0}).
> create() ->
>      #state{}. % Only understood by player.erl
>
> bump_score(#state{} = Player, PinsDown) ->
>     ..
> get_score(#state{} = Player) ->
>     ..
>
>
>> This stems from the fact that in
>> erlang, state is maintained at a process level and not at a module
>> level.
>>
>> I would imagine, this is a dilemma which is not entirely uncommon.
>> Modeling a low level intricate module as a process, allows the finer
>> details of the state to be contained in the implementation
>> modules/processes, and can allow the policy to be maintained in a
>> higher level module/process. This can also allow for easier way to
>> change implementations (eg. hypothetically, in this case the precise
>> semantics of scoring - one could go as far as defining a player to be
>> behaviour). I am fully aware these are thoughts which stem from a
>> classical OO experience.
>>
>> That begs the question -> are there situations where experienced
>> erlang programmers choose to model processes not because they run
>> concurrently, but because they have different independently
>> encapsulated states, and in addition could also be helpful for
>> separating out implementation specific behaviour.
>
> [Keeping in mind that I'm not an "experienced Erlang programmer"]
>
> State machines come to mind (gen_fsm).
>
> You could certainly go your original route. Process creation is fast.  
> Message passing is fast. Just be careful not to inadvertently create a  
> N-to-1-to-N routing for messages between processes. This can become a  
> bottleneck when a program grows and/or becomes very busy. Tracing and  
> debugging gets a little tricker too.
>
> - Edmond -
>
> It can also make things a little hard to follow in terms of tracing and  
> debugging.
>
>> Or is there another
>> programming feature / trick that I am not aware of which could help
>> resolve these conflicting objectives?
>>
>> Dhananjay
>>
>
>


-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/


More information about the erlang-questions mailing list