[erlang-questions] "actor database" - architectural strategy question

[Miles Fidelman]

Just spent some time reading Fred's blog post (thanks Fred), and 
googling "Erlang garbage collection" - interesting stuff, and now I much 
better understand the concerns you raise!

A follow-up if I might:

Mahesh Paolini-Subramanya wrote:
> “How to handle older largely inactive processes”
>  What we do (did?) was to basically flush/hibernate the process.  To 
> generalize, if the process hasn’t “done” anything in a while, save its 
> state somewhere, and then hibernate (which has the added benefit of 
> dealing with GC issues :-)
> Mind you, there is a wealth of info buried in the phrase “save its 
> state somewhere”.  This really, /really/ depends  on how big, how 
> scalable, how fault-tolerant, how geographic, how….. you intend on 
> getting. In short, it can range from “write out a text file” to “pay 
> Riak gobs-o-money and have nodes worldwide”. YMMV

Can you say more about this?  Context in which you are 
flushing/hibernating processes - and how your're saving state?

Cheers,

Miles

>
> “Heroku gang” <— My apologies, I meant “those fine fine folks at 
> Heroku who happen to be doing erlang”
>
> cheers
>
> *
> *Mahesh Paolini-Subramanya 
> <http://www.gravatar.com/avatar/204a87f81a0d9764c1f3364f53e8facf.png>*
> That tall bald Indian guy..
> *
> *
> Google+ <https://plus.google.com/u/0/108074935470209044442/posts>  | 
> Blog <http://dieswaytoofast.blogspot.com/>   | Twitter 
> <https://twitter.com/dieswaytoofast>  | LinkedIn 
> <http://www.linkedin.com/in/dieswaytoofast>
> *
>
> On February 17, 2014 at 4:40:22 PM, Miles Fidelman 
> (mfidelman@REDACTED <mailto://mfidelman@meetinghouse.net>) wrote:
>
>> Mahesh,
>>
>> Mahesh Paolini-Subramanya wrote:
>> > “Large number of processes with very long persistence”
>> >
>> > You *will* run into GC issues here, and of all kinds
>> > - design artifacts (“hmm, the number of lists that I manipulate
>> > increases relentlessly…”)
>> > - misunderstanding (“But I passed the binary on, without
>> > manipulating it at all!”)
>> > - Bugs (Fred has a great writeup on this somewhere)
>>
>> Very good points - though to a degree they sound more like dependency
>> hell than traditional garbage collection to reclaim memory.
>>
>> Given the document-oriented view, I'm viewing garbage collection more in
>> the sense of filing and archiving - the same way that paper documents
>> migrate to filerooms then to archives; or email and computer files
>> simply get buried deeper and deeper in one's file system; sometimes you
>> buy a larger drive; sometimes stuff migrates to off-site backup - but
>> you generally don't throw stuff away (though when working on
>> multi-author documents, one always comes back to how many intermediate
>> copies to retain "for the record" after the final version goes to print).
>>
>> In one sense, this ends up looking a lot like managing a git repository
>> - more and more versions and branches accumulate, and so forth. And
>> once starts thinking about storing only change logs.
>>
>> This is also what motivates my question about how to handle older,
>> largely inactive processes. It's one thing to bury a file deeper and
>> deeper in a file system - and still be able to find and access it (and
>> these days, search for it). It's another to think about migrating an
>> actor from RAM to disk, in a way that retains its ability to respond to
>> the infrequent message.
>>
>> The other area I worry about is exponential growth in network traffic
>> and cpu cycles - assuming that a lot of documents will never completely
>> "die" - maybe an update will come in once week, or once a month, or
>> they'll get searched every once in a while - as the number of processes
>> increases, the amount of traffic will as well.
>>
>> > Just keep in mind that in the end, you will almost certainly end up
>> > doing some form of manual GC activities. Again, the Heroku gang can
>> > probably provide a whole bunch of pointers on this…
>> >
>>
>> Can you say a bit more about what it is about Heroku that I should be
>> looking at? At first glance, it seems like a very different environment
>> than what we're talking about here (or are you thinking about manual
>> housekeeping for the virtual environment?).
>>
>> And.. re. "Bugs (Fred has a great writeup on this somewhere)" - Fred
>> who? (Maybe I can find it by googling!)
>>
>> Thanks Very Much,
>>
>> Miles
>>
>> -- 
>> In theory, there is no difference between theory and practice.
>> In practice, there is. .... Yogi Berra
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions


-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra