Billion-triple store

Leif Johansson <>
Tue May 2 23:30:47 CEST 2006

Hash: SHA1

> Well as the lead maintainer of kowari, I would be very happy to discuss
> any requirements you might have, and see if we can't help you.

My application is a sink datastore for an enterprise message-bus. My
message-bus is distributed and collects event-based data from a variety
of sources including syslog (from 1000s of clients), hr-systems and
student administration systems (for identity-management), LDAP
directories, computer-telephony integration servers and network-
management servers. I want to collect all messages in a datastore
normalized as RDF. The datastore can quite possibly be distributed
and must be able to support a sustained rate of 1000s of insertions
of small RDFs per second.

How do you like them requirements :-)

> Currently the largest scalability test I am aware of for kowari was
> 500million, but those results indicated that we hadn't reached our limit
> yet.  One of the store-layers designers did some calculations that
> indicate that we should be able to scale to 1-2billion without
> difficulty; although as one of the primary developers of the query layer
> I am aware of some bottle necks that are likely to interfere with any
> queries requiring extremely large intermediate results (~1e6 tuples).

I looked at where the
recorded claims are a bit lower... but maybe that site is out-of-date.

> At the same time, there are plans to address these issues, and to break
> the scalability bottle necks that are preventing us reaching 1e10 and
> 1e11 at the moment, these include promising prototypes of a new store
> design to improve locality and throughput that should result in us
> scaling comfortably to 1e10.

I am quite interested in update scalability too...

> As far as interfacing with erlang is concerned, we currently support rmi
> and soap, as well as in-process java funcalls.  I am currently working
> on xmlrpc support, and I am aware of plans to introduce a rest interface
> as well.
> Please let me know if there is anything I can do to help.

	Cheers Leif
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla -


More information about the erlang-questions mailing list