1. On correctness in RDF stream processor
benchmarking
Daniele Dell’Aglio, Jean-Paul Calbimonte, Marco Balduini,
Oscar Corcho and Emanuele Della Valle

2. The correctness problem (1)
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen
S
S1
S2
S3
S4
1
3
6
9
t
Executio 1° answer 2° answer
Where are Alice and
n
Bob, when they are
1
:hall [6]
:kitchen [11]
together?
2
:hall [5]
:kitchen [10]
 Let’s consider a
3
:hall [6]
:kitchen [11]
tumbling window
4
- [7]
- [12]
slide
Which is the correct answer?
W(ω=β=5)
All of them!
 Let’s execute the
width
experiment 4 times processor benchmarking
2/16
On correctness in RDF stream
ISWC, 24 October 2013


3. The correctness problem (2)
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen
S
S1
S2
S3
S4
1
3
6
9
t
System 1
System 2
Executio
n
1° answer
2° answer
Executio
n
1° answer
2° answer
1
:hall [6]
:kitchen [11]
1
:hall [3]
:kitchen [9]
2
:hall [5]
:kitchen [10]
2
3
:hall [6]
:kitchen [11]
3
4
- [7]
- [12]
3/16
No answers
:hall [3]
:kitchen [9]
4
No answers
Which system behaves in the
Both!
correct way?
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

4. Different results for the same query? Yes!
Given the same data and
the same query, we notice
that a system can provide
multiple correct
answers



The operational semantics
of the systems often do not
explain this behaviour

Two systems can provide
different correct
answers

Even if the system
implements similar
operational semantics, they
behave in different ways
(that the model does not
explain)
Why is it important to understand those
behaviours?


4/16
To assess the correct implementation of the systems
To improve the comprehension of the benchmarking
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

5. Our contribution


A common model for the RDF stream processor
operational semantics
We propose CSR-bench, an extension of the
SRbench benchmark that focuses on correctness



An oracle (an automatic correctness validator)
A test suite
We considered the window-based RDF stream
processors



5/16
CQELS
C-SPARQL
SPARQLstream
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

6. The CSR Model (by CQL)
Input stream

S2R
operator


R2R
operator


R2S
operator

Converts the infinite stream of RDF elements in a
finite set of mappings
The window operators: time-based, tuple-based,
…
Transform a set of mappings in another set of
mappings
SPARQL 1.0/1.1 queries
Each set of mapping produced by the R2R
operator is transformed and appended to the
output stream
Operators: RStream, DStream, IStream
Output stream
6/16
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

7. The window operator (through SECRET)
REPORT: When is the window content
made available to the R2R operator?
Non-empty content, Content-change,
Window-close, Periodic
t0: When does the
window start?
(internal window
param)
R2R operator
ω
β
TICK: When are
data stream
elements added to
the window?
Triple-based vs
graph-based
S
W(ω,β)
S1
S3
S2
S4
S6
S5
S7
S9
S11
S8
S10
S12
t
7/16
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

8. Classification of existing systems


The report and the tick policies are related to the
RSPs implementation and on how they implement
the window operators
Analysing the RSPs and their related documentation
(papers, technical reports, etc.), we classified the
systems:
CQELS
Report Content-change
Tick Tuple-driven
Empty relation No
notification
8/16
C-SPARQL
SPARQLstream
Window-close
Non-empty
content
Window-close
Non-empty
content
Tuple-driven
Tuple-driven
Yes
No
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

9. RSP output correctness
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen
S
S1
S2
S3
S4
1
3
6
9
t
t0=0
t0=1
t0=2
Execution
1° answer
2° answer
1
:hall [6]
:kitchen [11]
2
:hall [5]
:kitchen
[10]
3
:hall [6]
:kitchen [11]
4
- [7]
- [12]
9/16
Window
1° answer
2° answer
t0=0
t0=1
t0=2
:hall [5]
:kitchen [10]
:hall [6]
:kitchen [11]
- [7]
- [12]
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

10. RSP output correctness
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen
S
S1
S2
S3
S4
1
3
6
9
t
System 1
System 2
Executio
n
1° answer
2° answer
Executio
n
1° answer
2° answer
1
:hall [6]
:kitchen [11]
1
:hall [3]
:kitchen [9]
2
:hall [5]
:kitchen [10]
2
3
:hall [6]
:kitchen [11]
3
4
- [7]
- [12]
4
Window-close vs
content-change
report policy
10/16
No answers
:hall [3]
:kitchen [9]
No answers
Empty relation
notification
(yes|no)
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

11. The oracle
Online
RSP
q
S
Stream
importer
Query
transformer
Query
executor
Result
matcher
M
Correctness
assessment
Offline
Available at: https://github.com/dellaglio/csrbench-ora
(Apache 2.0 licence)
11/16
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

12. Design of the tests

As data set, we consider the LinkedSensorData data
set


We designed the query set taking into account





Data stream describing blizzards and hurricanes in the
US
Window size and slide parameters
Presence of aggregation operators
Joins of timestamped triples
We collected a set of seven parametrized queries
The list of the queries and their explanation is
available at: http://www.w3.org/wiki/CSRBench
12/16
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

13. Results





Initialization
Slide parameter
Window contents
timestamps of the triples

13/16
Internal timestamp management
On correctness in RDF stream processor benchmarking
SPARQLstrea
the window operator
C-SPARQL

CQELS

All the three systems that we
considered in our experiments
showed wrong behaviours
The defects we identified are related
to:
Q1
Q2
Q3
Q4
Q5
Q6
Q7
ISWC, 24 October 2013

14. (Removable) constraints
Reasonin
g
From single to
multi stream
S2R
S2R
R2R
R2S
S2R
From single
to multi
window
q2
14/16
Multiple
queries
On correctness in RDF stream processor benchmarking
Static
knowledge
ISWC, 24 October 2013

15. Conclusions

A model that describes in a more accurate way the
RSPs’ operational semantics helps their
improvement:



A common and shared test environment helps
developers of both existing and upcoming RSPs


Better design of the system
Prediction of the expected behaviours
It becomes easier to set up experiments to detect defects
and to correct them
Possible improvements of the existing
benchmarks (e.g., SRbench and LSBench)


Design of new tests
Better interpretation of the experiment results
15/16
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013

16. Thank you! Questions?
On correctness in RDF stream processor
benchmarking
Daniele Dell’Aglio (DEIB, Politecnico di Milano)
Jean-Paul Calbimonte (OEG, Universidad Politécnica de Madrid)
Marco Balduini (DEIB, Politecnico di Milano)
Oscar Corcho (OEG, Universidad Politécnica de Madrid)
Emanuele Della Valle (DEIB, Politecnico di Milano)
wiki: http://www.w3.org/wiki/CSRBench
software: https://github.com/dellaglio/csrbench-oracle
16/16
On correctness in RDF stream processor benchmarking
ISWC, 24 October 2013