A large part of Linked Data generation entails processing the raw data. However, this process is only documented in human-readable form or as a software repository. This inhibits reproducibility and comparability, as current documentation solutions do not provide detailed metadata and rely on the availability of specific software environments. We propose an automatic capturing mechanism for interchangeable and implementation independent metadata and provenance that includes data processing. Using declarative mapping documents to describe the computational experiment allows automatic capturing of term-level provenance for both schema and data transformations, and for both the used software tools as the input-output pairs of the data processing executions. This approach is applied to mapping documents described using RML and FnO, and implemented in the RMLMapper. The captured metadata can be used to more easily share, reproduce, and compare the dataset generation process, across software environments.

1. Detailed
Provenance Capture
of Data Processing
Ben De Meester, Anastasia Dimou,
Ruben Verborgh, and Erik Mannens
Ghent University – imec – IDLab, Belgium

2. *

4. Outline
Linked Data Generation
Problem
Solution
Results

5. Outline
Linked Data Generation
Problem
Solution
Results

6. Linked Data comes from
data
Unstructured data
Semi-structured data
Structured data
…

7. Linked Data comes from
processed data
Unstructured data Parse
Semi-structured data Extract
Structured data Add schema annotations
… …

8. Going from data to linked data
Data
Schema transformations
Data transformations
Linked Data

9. Linked Data generation =
schema + data transformations
dbr:
Barney_G
umble

10. Linked Data generation =
schema + data transformations
dbr:
Barney_G
umble
dbo:birthDate
dbp:voiceactor
dbp:gender
dbp:name
…

11. Linked Data generation =
schema + data transformations
dbr:
Hawaii
dbr:
Barney_G
umble
dbo:birthDate
dbp:voiceactor
dbp:gender
dbp:name
“1954-4-20"
dbr:
Dan_Caste
llaneta
“Male”
“Barney Gumble”@en
… …

12. Problem: there’s always a drunk Barney
Data
Schema processing
Data processing
Linked Data

13. Data
Data
Processing

14. Knowing where the data
comes from is as important as
the data itself
Oh Yeah?

15. Outline
Linked Data Generation
Problem
Solution
Results

16. dbr:
Hawaii
dbr:
Barney_G
umble
dbo:birthDate
dbp:voiceactor
dbp:gender
dbp:name
“1954-4-20"
dbr:
Dan_Caste
llaneta
“Male”
“Barney Gumble”@en
… …
Linked Data re-generation?
Provenance of those transformations

17. How it’s done for data processing
Provenance log:
Person A used Software B, on System C

18. Problem: how to reproduce?
Provenance log:
Person A used Software B, on System C
Software B offline?
System C not booting?

19. What do we need?
Fine-grained provenance
for schema transformations
for data transformations
Independent of the implementation

20. How can we tell where the
data comes from, without
depending on the system?

21. Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Results

22. What do we want?
Term-level,
implementation-independent provenance
for schema transformations
for data transformations

23. What do we want?
Term-level,
implementation-independent provenance
for schema transformations
for data transformations
Generated automatically

24. What do we want?
Term-level,
implementation-independent provenance
for schema transformations
for data transformations
Generated automatically
Declarative generation process

25. Steps
Align schema and data transformations
in a declarative document
Generate provenance based on
declarative schema transformations
Generate provenance based on
declarative data transformations

26. Declarative generation process?
Align schema and data transformations in a declarative
document

27. Declarative generation process? Solved!
Align schema and data transformations in a declarative
document
RML + FnO

28. Declarative generation process? Solved!
Align schema and data transformations in a declarative
document
RML + FnO for DBpedia EF
Declarative data transformations for Linked Data generation: the case of DBpedia
De Meester, B., Maroy, W., Dimou, A., Verborgh, R., and Mannens, E.
Sustainable Linked Data Generation: The Case of DBpedia
Maroy, W., Dimou, A., Kontokostas, D., De Meester, B., Verborgh, R., Lehmann, J., Mannens, E. and Hellmann, S.

29. Schema transformations provenance?
Generate provenance based on declarative mapping
document

30. Schema transformations provenance?
Solved!
Generate provenance based on declarative mapping
document
RML + PROV
Automated metadata generation for Linked Data generation and publishing workflows
Dimou, A., De Nies, T., Verborgh, R., Mannens, E., and Van de Walle, R.

31. Data transformations provenance?
Generate provenance based on declarative data
transformations

32. Data transformations provenance?

33. Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Declarative generation
FnO and PROV
Results

34. FnO: Function
expects
output
inputString
predicate
outputString
predicate
DBpedia_
date_parser
fno:Function

35. FnO: Execution
DBpedia_
date_parser
Function
“April 20th 1954”
parseExecution
fno:Execution
“1954-04-20”
outputString
executesinputString

36. FnO: General Execution
Function
Data Transformation Output
Input

37. Aligning FnO and PROV
Output
prov:Entity
Tool
prov:Agent
wasGeneratedBy
Data Transformation
prov:Activity
Function
prov:Entity
used
Input
prov:Entity
used wasAssociatedWith
wasAttributedTo

38. Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Declarative generation
FnO and PROV
Results

39. Uncool thing:
It’s big
When including provenance generation,
for every processed term,
you add 10 triples

40. Cool thing #1:
System details complementary
Output
prov:Entity
Tool
prov:Agent
wasGeneratedBy
Data Transformation
prov:Activity
Function
prov:Entity
used
Input
prov:Entity
used wasAssociatedWith
wasAttributedTo

41. Cool thing #2:
Aligning with RML complementary
wasInformedBy
Output
prov:Entity
Tool
prov:Agent
wasGeneratedBy
Data Transformation
prov:Activity
Function
prov:Entity
used
Input
prov:Entity
Schema
Transformation
prov:Activity
used wasAssociatedWith
wasAttributedTo

42. Cool thing #3:
It actually works
RMLMapper
https://github.com/RMLio/RML-Mapper
FunctionProcessor
https://github.com/FnOio/function-processor-java
DBpedia Extraction Sample
https://fno.io/prov/dbpedia/

43. How can we find a drunk Barney?
Query for long-lasting processes
Query all outputs of a certain function/tool
Query all input-output pairs

44. What to do with a drunk Barney?
Performance evaluation
Qualitative comparison
Iterative improvement
(only changing what is needed!)

45. Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Declarative generation
FnO and PROV
Results

46. Detailed
Provenance Capture
of Data Processing
Ben De Meester, Anastasia Dimou,
Ruben Verborgh, and Erik Mannens
Ghent University – imec – IDLab, Belgium