We have released version 2 of the Berlin SPARQL Benchmark (BSBM) today and reported the results of running the benchmark against:
- three RDF stores (Virtuoso Version 5.0.8, Sesame Version 2.2, Jena TDB Version 0.53) and
- two relational database-to-RDF wrappers (D2R Server Version 0.4 and Virtuoso - RDF Views Version 5.0.8).
for RDF datasets ranging from 250,000 triples to 100,000,000 triples.
In order to set the SPARQL query performance into context we also report the results of running the SQL version of the benchmark against two relational database management systems (MySQL 5.1.26 and Virtuoso - RDBMS Version 5.0.8).
Compared with version 1, we extended the benchmark to a multi-client scenario, fine-tuned the benchmark dataset
and the query mix, and implemented a SQL version of the benchmark in order to be able to compare SPARQL stores with classical SQL stores.
It is interesting to see:
- that relational database to RDF wrappers generally outperform RDF stores for larger dataset sizes.
- that no store outperforms the others for all queries and dataset sizes.
- that the query throughput still varies widely within the multi-client scenario.
- that the fastest RDF store is still 7 times slower than a relational database.
Thanks a lot to
- Eli Lilly and Company and especially Susie Stephens for making this work possible through a research grant.
- Orri Erling (OpenLink Software, UK), Andy Seaborne (HP Labs, UK), Arjohn Kampman (Adunak, UK), Michael Schmidt (Universität Freiburg, Germany), Richard Cyganiak (DERI, Ireland), Ivan Mikhailov (OpenLink Software, UK), Patrick van Kleef (OpenLink Software, UK), and Christian Becker (Freie Universität Berlin) for their feedback on the benchmark design and their help with configuring the stores and running the benchmark experiment.