[erlang-questions] "actor database" - architectural strategy question

[Miles Fidelman]

Michael Truog wrote:
> On 02/17/2014 03:22 PM, Miles Fidelman wrote:
>> Any thoughts on how to do this - perhaps in combination with 
>> extending the hibernate BIF? 
> Most people use gen_server with hibernate in the return value from the 
> callback (e.g., 
> http://www.erlang.org/doc/man/gen_server.html#Module:handle_call-3). 
> If you need something that is more complex, CloudI has a hibernate 
> service configuration option that allows you to hibernate based on the 
> incoming service request rate, so when the service request rate drops 
> too low, the process hibernates whenever possible 
> (http://cloudi.org/api.html#2_services_add_config_opts_hibernate).

Ok... see that.  Thanks!

And thanks to Motiejus Jakštys for the existence proof, sizing info, 
(and the checkpointing scheme - not quoted):
>
> One portability request is one process. At any time we could have had 
> up to 1M processes (practically it was much lower, but we used this 
> number when designing the system). A "portability process" is a finite 
> state machine with communication in SOAP between two or three parties 
> with many internal state changes.
>
> A single process could last from a few hours up to few months (median 
> ~3-4 days), each state up to 10-100KB of uncompressed text (mean ~15KB 
> uncompressed).
>

The question about extending the hibernate BIF, and maybe playing with 
the scheduler, focuses on scaling further - to Joe Armstrong's comments 
that:
> Some kind of "getting things out of memory and onto disk when not 
> needed" layer is needed for your problem,
>
and
> Yes - I guess the real solution would be to change the scheduler to 
> swap processes to disk after they had waited for more than (say) 10 
> minutes for a message, and resurrect them when they are sent a message.

It strikes me that hibernate addresses a big piece of the problem - 
reducing footprint of inactive processes, while keeping them "alive" 
(i.e., able to receive messages, wake up, and all that) - and when 
called via proc_lib:hibernate, keeps exception handling and logging working.

For scaling, the next step would be to swap out to disk - if that 
happens as a hibernate to disk, all the other functionality would stick 
around (as opposed to an application level write-state-to-disk).  Hence 
the question re. internals of the hibernate BIF and the scheduler, and 
how one might think about wiring in a hibernate-to-disk function.

Anyway... the overall approach is starting to look feasible!

Thanks!

Miles






-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra