Billion-triple store

[Leif Johansson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> 
> Well as the lead maintainer of kowari, I would be very happy to discuss
> any requirements you might have, and see if we can't help you.

My application is a sink datastore for an enterprise message-bus. My
message-bus is distributed and collects event-based data from a variety
of sources including syslog (from 1000s of clients), hr-systems and
student administration systems (for identity-management), LDAP
directories, computer-telephony integration servers and network-
management servers. I want to collect all messages in a datastore
normalized as RDF. The datastore can quite possibly be distributed
and must be able to support a sustained rate of 1000s of insertions
of small RDFs per second.

How do you like them requirements :-)

> 
> Currently the largest scalability test I am aware of for kowari was
> 500million, but those results indicated that we hadn't reached our limit
> yet.  One of the store-layers designers did some calculations that
> indicate that we should be able to scale to 1-2billion without
> difficulty; although as one of the primary developers of the query layer
> I am aware of some bottle necks that are likely to interfere with any
> queries requiring extremely large intermediate results (~1e6 tuples).

I looked at http://esw.w3.org/topic/TripleStoreScalability where the
recorded claims are a bit lower... but maybe that site is out-of-date.

> 
> At the same time, there are plans to address these issues, and to break
> the scalability bottle necks that are preventing us reaching 1e10 and
> 1e11 at the moment, these include promising prototypes of a new store
> design to improve locality and throughput that should result in us
> scaling comfortably to 1e10.

I am quite interested in update scalability too...

> 
> As far as interfacing with erlang is concerned, we currently support rmi
> and soap, as well as in-process java funcalls.  I am currently working
> on xmlrpc support, and I am aware of plans to introduce a rest interface
> as well.
> 
> Please let me know if there is anything I can do to help.
> 

	Cheers Leif
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEV8+H8Jx8FtbMZncRAq1oAKC0TUQ9STU7ep0n1s4KAVnhkuiOQgCfQlAt
qKI/0ammzDTFzbPHVkq+yz8=
=nWkk
-----END PGP SIGNATURE-----