Billion-triple store

[Leif Johansson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Leif Johansson wrote:
> 
>>> Well as the lead maintainer of kowari, I would be very happy to discuss
>>> any requirements you might have, and see if we can't help you.
> 
> My application is a sink datastore for an enterprise message-bus. My
> message-bus is distributed and collects event-based data from a variety
> of sources including syslog (from 1000s of clients), hr-systems and
> student administration systems (for identity-management), LDAP
> directories, computer-telephony integration servers and network-
> management servers. I want to collect all messages in a datastore
> normalized as RDF. The datastore can quite possibly be distributed
> and must be able to support a sustained rate of 1000s of insertions
> of small RDFs per second.

I have done a couple of experiments with a totally brain-dead (probably)
schema on a single node with {n_fragments, 1000}. I can insert ~250k
statements at a sustained rate of ~2300 statements per second using
disk-copies of everything and a transaction per statement. The resulting
mnesia disk db is ~55M from a 25M n3 file turned into erlang statements
using perl :-)

Stay tuned...

	Cheers Leif
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEWOh48Jx8FtbMZncRAtKyAJ4g2U7qecZxe5GjWf2iRrkqlLVSuQCghmX1
XMDO8YxW6mlnZ9EPm8GkAHk=
=mA6g
-----END PGP SIGNATURE-----