[erlang-questions] "actor database" - architectural strategy question

[Miles Fidelman]

Mahesh,

Mahesh Paolini-Subramanya wrote:
> “Large number of processes with very long persistence”
>
> You *will* run into GC issues here, and of all kinds
>    - design artifacts (“hmm, the number of lists that I manipulate 
> increases relentlessly…”)
>    - misunderstanding (“But I passed the binary on, without 
> manipulating it at all!”)
>    - Bugs (Fred has a great writeup on this somewhere)

Very good points - though to a degree they sound more like dependency 
hell than traditional garbage collection to reclaim memory.

Given the document-oriented view, I'm viewing garbage collection more in 
the sense of filing and archiving - the same way that paper documents 
migrate to filerooms then to archives; or email and computer files 
simply get buried deeper and deeper in one's file system; sometimes you 
buy a larger drive; sometimes stuff migrates to off-site backup - but 
you generally don't throw stuff away (though when working on 
multi-author documents, one always comes back to how many intermediate 
copies to retain "for the record" after the final version goes to print).

In one sense, this ends up looking a lot like managing a git repository 
- more and more versions and branches accumulate, and so forth.  And 
once starts thinking about storing only change logs.

This is also what motivates my question about how to handle older, 
largely inactive processes.  It's one thing to bury a file deeper and 
deeper in a file system - and still be able to find and access it (and 
these days, search for it).  It's another to think about migrating an 
actor from RAM to disk, in a way that retains its ability to respond to 
the infrequent message.

The other area I worry about is exponential growth in network traffic 
and cpu cycles - assuming that a lot of documents will never completely 
"die" - maybe an update will come in once week, or once a month, or 
they'll get searched every once in a while - as the number of processes 
increases, the amount of traffic will as well.

> Just keep in mind that in the end, you will almost certainly end up 
> doing some form of manual GC activities.  Again, the Heroku gang can 
> probably provide a whole bunch of pointers on this…
>

Can you say a bit more about what it is about Heroku that I should be 
looking at?  At first glance, it seems like a very different environment 
than what we're talking about here (or are you thinking about manual 
housekeeping for the virtual environment?).

And.. re. "Bugs (Fred has a great writeup on this somewhere)" - Fred 
who?  (Maybe I can find it by googling!)

Thanks Very Much,

Miles

-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra