[erlang-questions] "actor database" - architectural strategy question
Mon Feb 17 22:40:07 CET 2014
Mahesh Paolini-Subramanya wrote:
> “Large number of processes with very long persistence”
> You *will* run into GC issues here, and of all kinds
> - design artifacts (“hmm, the number of lists that I manipulate
> increases relentlessly…”)
> - misunderstanding (“But I passed the binary on, without
> manipulating it at all!”)
> - Bugs (Fred has a great writeup on this somewhere)
Very good points - though to a degree they sound more like dependency
hell than traditional garbage collection to reclaim memory.
Given the document-oriented view, I'm viewing garbage collection more in
the sense of filing and archiving - the same way that paper documents
migrate to filerooms then to archives; or email and computer files
simply get buried deeper and deeper in one's file system; sometimes you
buy a larger drive; sometimes stuff migrates to off-site backup - but
you generally don't throw stuff away (though when working on
multi-author documents, one always comes back to how many intermediate
copies to retain "for the record" after the final version goes to print).
In one sense, this ends up looking a lot like managing a git repository
- more and more versions and branches accumulate, and so forth. And
once starts thinking about storing only change logs.
This is also what motivates my question about how to handle older,
largely inactive processes. It's one thing to bury a file deeper and
deeper in a file system - and still be able to find and access it (and
these days, search for it). It's another to think about migrating an
actor from RAM to disk, in a way that retains its ability to respond to
the infrequent message.
The other area I worry about is exponential growth in network traffic
and cpu cycles - assuming that a lot of documents will never completely
"die" - maybe an update will come in once week, or once a month, or
they'll get searched every once in a while - as the number of processes
increases, the amount of traffic will as well.
> Just keep in mind that in the end, you will almost certainly end up
> doing some form of manual GC activities. Again, the Heroku gang can
> probably provide a whole bunch of pointers on this…
Can you say a bit more about what it is about Heroku that I should be
looking at? At first glance, it seems like a very different environment
than what we're talking about here (or are you thinking about manual
housekeeping for the virtual environment?).
And.. re. "Bugs (Fred has a great writeup on this somewhere)" - Fred
who? (Maybe I can find it by googling!)
Thanks Very Much,
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
More information about the erlang-questions