1. INDEXING AND SEARCHING
RDF DATASETS
Improving Performance of Semantic Web Applications with
Lucene, SIREn and RDF
Mike Hugo
Entagen, LLC

2. slides and sample code can be found at
https://github.com/mjhugo/rdf-lucene-siren-presentation

4. ENTAGEN
ACCELERATING INSIGHT

5. 17

8. AGENDA

10. SPARQL

11. SPARQL
LUCENE

12. SPARQL
LUCENE
SIREN

13. SPARQL
LUCENE
SIREN
TripleMap.com

15. LINKING OPEN DATA

16. LIFE SCIENCE LINKED DATA

17. WHAT’S A TRIPLE?
Subject
Predicate
Object

18. WHAT’S A TRIPLE?
<Mike>
<name>
“Mike Hugo”

19. WHAT’S A TRIPLE?
“Minneapolis”
<lives_in_city>
<Mike>
<name>
“Mike Hugo”

20. WHAT’S A TRIPLE?
“Minneapolis”
<Mike> <lives_in_city>
<daughter> <name>
“Mike Hugo”
<Lydia>

21. WHAT’S A TRIPLE?
“Minneapolis”
<Mike> <lives_in_city>
<name>
<daughter>
“Mike Hugo”
<Lydia>
<name> “Lydia Hugo”

26. Ready, GO!

27. select id, label
from targets
where label = ‘${queryValue}’

28. select id, label
from targets
where label
ilike ‘%${queryValue}%’

29. SELECT ?uri ?type ?label WHERE {
?uri rdfs:label ?label .
?uri rdf:type ?type .
FILTER (?label = '${params.query}')
} LIMIT 10

30. SELECT ?uri ?type ?label WHERE {
?uri rdfs:label ?label .
?uri rdf:type ?type .
FILTER regex(?label,
'Q${params.query}E', 'i')
} LIMIT 10

31. SELECT ?uri ?type ?label WHERE {
?uri rdfs:label ?label .
?uri rdf:type ?type .
FILTER regex(?label,
'Q${params.query}E', 'i')
} LIMIT 10
case insensitive
query as literal value

32. DEMO
Baseline SPARQL Query Performance

33. FASTER!

35. Java API
Indexing and Searching Text

36. `
http://wiki.apache.org/lucene-java/PoweredBy

37. indexing storage

38. Document

39. Document
field value
ID 2
name “Mike Hugo”
company “Entagen”
“lorem ipsum
bio
dolor sum etc...”

40. Index
field
field value
value
field
field value
value
field
field value
name
name field “mike value
value
hugo”
“mike hugo”
name
name “mike hugo”
“mike hugo”
name
nameid “mike hugo”
“mike 2hugo”
company
company “Entagen”
“Entagen”
company
company
name “Entagen”
“Entagen”
“Mike Hugo”
company
company “Entagen”
“Entagen”
“Entagen” Indexed
company “lorem ipsum
“lorem ipsum
bio
bio “lorem ipsum
“lorem etc...”
bio
bio “loremipsum
dolorsum ipsum
dolor“loremetc...” not
bio
bio sum ipsum
dolorsum etc...””
dolor sum etc... ”
bio dolor sum ipsum”
“lorem etc...
dolor sum etc... Stored
dolor sum etc...”

41. Query: name: mike

42. Query: name: mike
Matching
Documents: field value
idfield 2
value
idfield 2
value
idfield 2
value
id 2

43. field value
id 2

44. field value
id 2

45. field value
id 2
field value
ID 2
name “Mike Hugo”
company “Entagen”
“lorem ipsum
bio
dolor sum etc...”

46. Simplest Solution

47. Lucene index of rdfs:label

48. Build the Index

49. String queryLabels = """
SELECT ?uri ?label
WHERE {
?uri rdfs:label ?label .
} Build a SPARQL
""" query to find all the
rdfs:label properties
sparqlQueryService.executeForEach(repo
def doc = new Document()
String uri = it.uri.stringValue()
String label = it.label.stringValu
doc.add(new Field(SUBJECT_URI_FIEL

50. sparqlQueryService.executeForEach
(repository, queryLabels) {
String uri = it.uri.stringValue()
String label = it.label.stringValu
Execute the
def doc = new Document()
SPARQL query
doc.add(new Field(SUBJECT_URI_FIEL
Field.Store.YES, Field.Ind
doc.add(new Field(LABEL_FIELD, lab
Field.Store.NO, Field.Inde
writer.addDocument(doc)
}

51. arqlQueryService.executeForEach(reposito
String uri = it.uri.stringValue()
String label = it.label.stringValue()
Document doc = new Document()
doc.add(new Field(SUBJECT_URI_FIELD,
uri, Instantiate a
new Lucene
Field.Store.YES,
Document
Field.Index.ANALYZED))
doc.add(new Field(LABEL_FIELD,
label,
Field.Store.NO,
Field.Index.ANALYZED))
writer.addDocument(doc)

52. key
Document doc = new Document()
doc.add(new Field(SUBJECT_URI_FIELD,
value uri,
Field.Store.YES,
Field.Index.ANALYZED))
doc.add(new Field(LABEL_FIELD,
label, Add the Subject
Field.Store.NO, URI to the
Document
Field.Index.ANALYZED))
writer.addDocument(doc)
lly {

53. Field.Store.YES,
Field.Index.ANALYZED))
doc.add(new Field(LABEL_FIELD, key
value label,
Field.Store.NO,
Field.Index.ANALYZED))
Add the Label field
writer.addDocument(doc) document
to the
(but don’t store it)
lly {
iter.close() // Close index

54. doc.add(new Field(LABEL_FIELD, labe
Field.Store.NO,
Field.Index.ANALYZED))
writer.addDocument(doc)
}
inally {
writer.close() // Closethe document
Add index
to the Index

55. Query the Index

56. f query = {
Query query = new QueryParser(
Version.LUCENE_CURRENT,
LABEL_FIELD, query this field
new StandardAnalyzer())
.parse(params.query);
for this value
def s Create a Lucene
= new Date().time
Query from user
List results = executeQuery(query)
input
def e = new Date().time
render(view: 'index', model: [results:

57. IndexSearcher searcher = luceneSearche
ScoreDoc[] scoreDocs =
searcher.search(query, 10).scoreDo
List results = [] Search the index
(limit 10) for
def connection = repository.connection
scoreDocs.each { matching
documents
Document doc = searcher.doc(it.doc
String uri = doc[SUBJECT_URI_FIELD
Map labelAndType = sparqlQueryServ
results << [uri: uri, type: labelA
}
connection.close()
return results

58. List results = []
def connection = repository.connection
scoreDocs.each {
Document doc = searcher.doc(it.doc)
String uri = doc[SUBJECT_URI_FIELD]
Map labelAndType =
For each matching
sparqlQueryService.
document, get the
getLabelAndType(uri, connection)
doc and extract the
results.add([
Subject URI
uri: uri,
type: labelAndType.type,
label: labelAndType.label])
}
connection.close()
return results

59. List results = []
def connection = repository.connection
scoreDocs.each {
Document doc = searcher.doc(it.doc)
String uri = doc[SUBJECT_URI_FIELD]
Map labelAndType =
sparqlQueryService.
getLabelAndType(uri, connection)
results.add([
uri: uri, Using the Subject
URI, load properties
type: labelAndType.type,
from the triplestore
label: labelAndType.label])
}
connection.close()
return results

60. List results = []
def connection = repository.connection
scoreDocs.each {
Document doc = searcher.doc(it.doc)
return results
containing Subject
String uri = doc[SUBJECT_URI_FIELD]
Map labelAndType Type, and Label
URI, =
sparqlQueryService.
getLabelAndType(uri, connection)
results.add([
uri: uri,
type: labelAndType.type,
label: labelAndType.label])
}
connection.close()
return results

61. DEMO
Lucene Index of Searchable Labels

62. WHAT ABOUT ENTITY
RELATIONSHIPS?

63. WHAT ABOUT OTHER
PROPERTIES?

64. Lucene Extension
Indexing and Searching
Semi-Structured Data

65. Document

66. Document
field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .

67. Build the Index

68. Connection connection = repository.conn
y {
String subjectUris = """
SELECT distinct ?uri
WHERE {
?uri ?p ?o .
}
"""
sparqlQueryService.executeForEach(rep
def doc = new Select all Subject
Document()
URIs from the
triplestore
String subjectUri = it.uri.string
doc.add(new Field(SUBJECT_URI_FIE
subjectUri,

69. """
sparqlQueryService.executeForEach(
repository, subjectUris) {
def doc = new Document()
String subjectUri = it.uri.stringV
doc.add(new Field(SUBJECT_URI_FIEL
subjectUri,
Field.Store.YES,
Execute the Sparql Query
Field.Index.ANALYZED))
For each URI, create a
new Document
StringWriter triplesStringWriter =
NTriplesWriter nTriplesWriter =
new NTriplesWriter(triplesStri

70. epository, subjectUris) {
def doc = new Document()
String subjectUri = it.uri.stringValue
doc.add(new Field(SUBJECT_URI_FIELD,
subjectUri,
Field.Store.YES,
Field.Index.ANALYZED))
StringWriter triplesStringWriter = new
NTriplesWriter nTriplesWriter =URI
Add the Subject
to the Document
new NTriplesWriter(triplesStringWr
connection.exportStatements(
new URIImpl(subjectUri),
null, null, false,

71. Field.Index.ANALYZED))
StringWriter triplesStringWriter = new
NTriplesWriter nTriplesWriter =
new NTriplesWriter(triplesStringWr
connection.exportStatements(
new URIImpl(subjectUri),
null, null, false,
nTriplesWriter)
Get an NTriples
doc.add(new Field(TRIPLES_FIELD,
string from the
triplesStringWriter.toString()
Field.Store.NO, triplestore
Field.Index.ANALYZED))

72. new URIImpl(subjectUri),
null, null, false,
nTriplesWriter)
doc.add(new Field(TRIPLES_FIELD,
triplesStringWriter.toString()
Field.Store.NO,
Field.Index.ANALYZED))
Add the NTriples
writer.addDocument(doc)
string to the
document

73. doc.add(new Field(TRIPLES_FIELD,
triplesStringWriter.toString()
Field.Store.NO,
Field.Index.ANALYZED))
writer.addDocument(doc)
Add the document
to the index

74. Query the Index

75. SirenCellQuery predicate =
new SirenCellQuery(
new SirenTermQuery(
new Term(TRIPLES_FIELD,
RDFS.LABEL.stringValue())));
predicate.constraint = PREDICATE_CELL
SirenCellQuery object =
query the Triples
new SirenCellQuery(
new SirenTermQuery( field
new Term(TRIPLES_FIELD,
params.query.toLowerCase()))
object.constraint = OBJECT_CELL

76. SirenCellQuery predicate =
new SirenCellQuery(
new SirenTermQuery(
new Term(TRIPLES_FIELD,
RDFS.LABEL.stringValue())));
predicate.constraint = PREDICATE_CELL
SirenCellQuery object =
new SirenCellQuery( a predicate
for
new SirenTermQuery(
new Term(TRIPLES_FIELD,
params.query.toLowerCase()))
object.constraint = OBJECT_CELL

77. SirenCellQuery predicate =
new SirenCellQuery(
new SirenTermQuery(
new Term(TRIPLES_FIELD,
RDFS.LABEL.stringValue())));
predicate.constraint = PREDICATE_CELL
of rdfs:label *
SirenCellQuery object =
new SirenCellQuery(
new SirenTermQuery(
new Term(TRIPLES_FIELD,
params.query.toLowerCase()))
* note: could be any predicate!
object.constraint = OBJECT_CELL

78. SirenCellQuery object =
new SirenCellQuery(
new SirenTermQuery(
new Term(TRIPLES_FIELD,
params.query.toLowerCase())
object.constraint = OBJECT_CELL
Query query = new SirenTupleQuery()
query the Triples
query.add(predicate,
field
SirenTupleClause.Occur.MUST)
query.add(object,
SirenTupleClause.Occur.MUST)

79. SirenCellQuery object =
new SirenCellQuery(
new SirenTermQuery(
new Term(TRIPLES_FIELD,
params.query.toLowerCase())
object.constraint = OBJECT_CELL
Query query = new SirenTupleQuery()
query.add(predicate,
for an object
SirenTupleClause.Occur.MUST)
query.add(object,
SirenTupleClause.Occur.MUST)

80. SirenCellQuery object =
new SirenCellQuery(
new SirenTermQuery(
new Term(TRIPLES_FIELD,
params.query.toLowerCase())
object.constraint = OBJECT_CELL
Query query = new SirenTupleQuery()
query.add(predicate, matching the
user input
SirenTupleClause.Occur.MUST)
query.add(object,
SirenTupleClause.Occur.MUST)

81. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query: “imatinib”

82. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
triples field

83. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
predicate = rdfs:label

84. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
predicate = rdfs:label
object = “imatinib”

85. List executeQuery(Query query) {
IndexSearcher searcher = sirenSearcherM
ScoreDoc[] scoreDocs =
searcher.search(query, 10).scoreDocs
List results = []
def connection = repository.connection
Search the index
scoreDocs.each { (limit 10) for
matching
Document doc = searcher.doc(it.doc)
documents
String uri = doc[SUBJECT_URI_FIELD]
Map labelAndType = sparqlQueryServi
getLabelAndType(uri, connectio
results.add([
uri: uri,
type: labelAndType.type,

86. List results = []
def connection = repository.connection
scoreDocs.each {
Document doc = searcher.doc(it.doc)
String uri = doc[SUBJECT_URI_FIELD]
Map labelAndType = sparqlQueryServic
For each matching
getLabelAndType(uri, connection
document, get the
results.add([
doc and extract the
uri: uri,
Subject URI
type: labelAndType.type,
label: labelAndType.label])
}
connection.close()
return results

87. connection = repository.connection
reDocs.each {
Document doc = searcher.doc(it.doc)
String uri = doc[SUBJECT_URI_FIELD]
Map labelAndType = sparqlQueryService.
getLabelAndType(uri, connection)
results.add([
uri: uri, Using the Subject
type: labelAndType.type,
URI, load properties
label: labelAndType.label])
from the triplestore
nection.close()
urn results

88. String uri = doc[SUBJECT_URI_FIELD]
Map labelAndType = sparqlQueryService.
getLabelAndType(uri, connection)
results.add([
uri: uri,
type: labelAndType.type,
label: labelAndType.label])
nection.close() return results
urn results containing Subject
URI, Type, and Label

89. DEMO
SIREn Index of RDF Entities

90. FLEXIBILITY

91. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
predicate = rdfs:label
object = “imatinib”

92. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
object = “imatinib”

93. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
object = “imatinib”
OR
object = “gleevec”

94. MORE THAN LITERALS

95. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
predicate = brandName

96. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
predicate = target

97. RELATIONSHIPS

102. field value
URI <DB00619>
<DB00619> rdfs:label "Imatinib" .
<DB00619> rdf:type <drugbank:drugs> .
triples
<DB00619> drugbank:brandName "Gleevec" .
<DB00619> drugbank:target <targets/1588> .
Query:
object = <targets/1588>

103. DEMO
Searching SIREn Index for Relationships

104. Distributed
Indexing and Searching
Semi-Structured Data

106. Replication

111. 400 Million Documents
> 12 Billion Triples

112. Query Parser

113. Query Parser
subject
predicate object

114. DEMO
SIREn in action on TripleMap.com

115. DEMO
SIREn in action on TripleMap.com

116. SPARQL
LUCENE
SIREN
TripleMap.com

117. QUESTIONS?
mike@entagen.com / twitter: @piragua
TripleMap
http://www.entagen.com http://www.triplemap.com