LOD2 Webinar: SIREn

Creating Knowledge out of Interlinked
Data
LOD2 Webinar . 24.06.2014 . Page 1 http://lod2.eu

Data
http://lod2.eu
LOD2 is a large-scale integrating project co-funded by the European
Commission within the FP7 Information and Communication Technologies
Work Programme. This 4-year project comprises leading Linked Open
Data technology researchers, companies, and service providers. Coming
from across 12 countries the partners are coordinated by the Agile
Knowledge Engineering and Semantic Web Research Group at the
University of Leipzig, Germany.
LOD2 will integrate and syndicate Linked Data with existing large-scale
applications. The project shows the benefits in the scenarios of Media and
Publishing, Corporate Data intranets and eGovernment.

Data
http://lod2.eu
Once per month the LOD2 webinar series offer a free webinar about
tools and services along the Linked Open Data Life Cycle.
Stay with us and learn more about acquisition, editing, composing,
connected applications – and finally publishing Linked Open Data.

Data
Agenda
• Nested Data Model
• SIREn Overview
• Getting Started with the SIREn Elasticsearch Plugin
• Demo

Data
Schema-Less Nested Data Model
• Model becoming prevalent: JSON, XML, Avro, …
– Can be arbitrarily nested and large
– No strict schema / structure enforced
• Schema-less brings
– Flexibility
– Ease of development
• Developers do not have to invest significant modelling
effort upfront

Data
Introducing SIREn
• Lucene, Solr and Elasticsearch plugin for indexing and
searching JSON
• Rich data model (JSON)
– Nested objects, nested arrays, datatypes
– Generic architecture compatible with various nested data models: JSON,
JSON-LD, XML, Avro, ...
• Schema-agnostic
– SIREn does not require any schema definition to index and search data
– Schema definition can change across records
• Designed from the ground up for high performance and
scalability

Data
Introducing SIREn

Data
Elasticsearch - Overview
• Document-oriented search and analytics engine
– JSON Document
– Based on Apache Lucene
• Distributed, Replication
– High Performance and Availability
• REST API

Data
Elasticsearch – Basic Concepts
• Index = Collection of Documents
– Can have multiple shards and replicas
• Type = A set of documents sharing the same schema
– Similar to a DB table
• Document = JSON object
– Uniquely identified (index/type/id)
– Similar to a DB record

Data
SIREn – Basic Concepts
• JSON object = Tree
– Different mapping available
• Node = An element of the tree
– Can have a parent and one or more children
– Contains data: text, numeric, boolean

Data
SIREn – Basic Concepts
JSON Object Tree
{
{
"name" : "Elasticsearch",
"category_code" : "search",
"funding_rounds" : [
{
{
"round_code" : "a",
"raised_amount" : 10000000,
"funded_year" : 2012,
"investments" : [
"round_code" : "a",
"investments" : [
{
{
"name" : "Data Collective",
"type" : "financial-org"
},
…
},
…
]
]
},
…
},
…
]
]
}
}
name : Elasticsearch
category_code : search
funding_rounds
round_code : a
raised_amount : 10000000
funded_year : 2012
investments
name : Data Collective
type : financial-org
investments
name : ...

Data
Getting Started with Elasticsearch & SIREn
http://sirendb.com/downloads/
(Elasticsearch Distribution Coming Soon)

Data
SIREn Elasticsearch Distribution
├── dist
│ ├── siren-core-1.3.jar
│ └── siren-qparser-1.3.jar
├── docs
│ ├── apidoc
│ └── siren-1.3-javadoc.jar
├── example
├── CHANGES.txt
├── LICENSE.md
├── README.md
└── THIRD-PARTY.txt
├── dist
│ ├── siren-core-1.3.jar
│ └── siren-qparser-1.3.jar
├── docs
│ ├── apidoc
│ └── siren-1.3-javadoc.jar
├── example
├── CHANGES.txt
├── LICENSE.md
├── README.md
└── THIRD-PARTY.txt
SIREn
Libraries
SIREn
Libraries
SIREn
Javadoc
SIREn
Javadoc
Elasticsearch
Distribution
Elasticsearch
Distribution

Data
SIREn Elasticsearch
Distribution
├── bin
│ ├── elasticsearch
│ └── ...
├── config
│ ├── elasticsearch.yml
│ └── logging.yml
├── lib
│ ├── elasticsearch-1.2.0.jar
│ └── ...
├── plugins
│ └── siren-plugin
│ ├── siren-elasticsearch-1.3.jar
│ └── ...
├── datasets
├── NOTICE.txt
└── README.md
├── bin
│ ├── elasticsearch
│ └── ...
├── config
│ ├── elasticsearch.yml
│ └── logging.yml
├── lib
│ ├── elasticsearch-1.2.0.jar
│ └── ...
├── plugins
│ └── siren-plugin
│ ├── siren-elasticsearch-1.3.jar
│ └── ...
├── datasets
├── NOTICE.txt
└── README.md
EExxeeccuutatabbleless
CCoonnfifgiguuraratitoionn
SSIRIREEnn P Plulugginin

Data
Starting Elasticsearch
$ ./bin/elasticsearch
$ ./bin/elasticsearch
[INFO ][node ] [Valerie Cooper] version[1.2.0], pid[12842], …
[INFO ][node ] [Valerie Cooper] initializing ...
[INFO ][plugins] [Valerie Cooper] loaded [siren-plugin], sites []
[INFO ][node ] [Valerie Cooper] initialized
[INFO ][node ] [Valerie Cooper] starting ...
…
[INFO ][node ] [Valerie Cooper] started
[INFO ][node ] [Valerie Cooper] version[1.2.0], pid[12842], …
[INFO ][node ] [Valerie Cooper] initializing ...
[INFO ][plugins] [Valerie Cooper] loaded [siren-plugin], sites []
[INFO ][node ] [Valerie Cooper] initialized
[INFO ][node ] [Valerie Cooper] starting ...
…
[INFO ][node ] [Valerie Cooper] started

Data
Create an Index and Map
SIREn’s Field
$ curl -XPUT 'http://localhost:9200/test/' –d '
{
$ curl -XPUT 'http://localhost:9200/test/' –d '
{
"mappings" : {
"mappings" : {
"companies" : {
"companies" : {
"properties" : {
"properties" : {
SIREn’s Field
"_siren_source" : {
"index" : "analyzed",
"analyzer" : "hybrid-json",
"postings_format" : "Siren10AFor",
"store" : "no",
"type" : "string"
}
},
"_siren" : {}
"_siren_source" : {
"index" : "analyzed",
"analyzer" : "hybrid-json",
"postings_format" : "Siren10AFor",
"store" : "no",
"type" : "string"
}
},
"_siren" : {}
}
}
}
}
}'
}'
SIREn’s Analyzer
SIREn’s
Posting Format

Data
Indexing a Document
$ curl -XPUT 'http://localhost:9200/test/companies/1' –d '
{
$ curl -XPUT 'http://localhost:9200/test/companies/1' –d '
{
{
{
"round_code" : "a",
"investments" : [
"round_code" : "a",
"investments" : [
{
{
}
}
]
]
}
}
]
]
}'
}'

Data
Customising Datatype Analysis
• SIREn distinguishes 5 types of values in the JSON model:
– Field, String, Long, Double, Boolean
{
{
{
{
"round_code" : "a",
"investments" : [
"round_code" : "a",
"investments" : [
{
{
json:field
}
}
]
]
}
}
]
]
}
}
xsd:long
xsd:string

Data
Customising Datatype Analysis
# config/elasticsearch.yml
# config/elasticsearch.yml
siren.analysis.datatype:
siren.analysis.datatype:
http://json.org/field:
http://json.org/field:
index_analyzer: keyword
index_analyzer: keyword
http://www.w3.org/2001/XMLSchema#string:
http://www.w3.org/2001/XMLSchema#string:
index_analyzer: simple
search_analyzer: simpleFolding
index_analyzer: simple
search_analyzer: simpleFolding
http://www.w3.org/2001/XMLSchema#long:
http://www.w3.org/2001/XMLSchema#long:
index_analyzer: long
index_analyzer: long
index.analysis.analyzer:
index.analysis.analyzer:
simpleFolding:
simpleFolding:
type: custom
tokenizer: letter
filter: [lowercase, asciifolding]
type: custom
tokenizer: letter
filter: [lowercase, asciifolding]

Data
Searching: Node Query
$ curl -XGET 'localhost:9200/test/companies/_search' –d '
{
"query" : {
{
"query" : {
"hybrid-json" : {
"hybrid-json" : {
"node" : {
"node" : {
"query" : "search",
"query" : "search",
}
}
}
}
}'
}'
funding_rounds
round_code : a
funded_year : 2012
investments
investments
name : ...
SIREn’s Query Type
SIREn’s Query
Full-Text Query:
•Boolean, Phrase, Range, Fuzzy,
Regexp, Proximity, ...

Data
{
"query" : {
{
"query" : {
"hybrid-json" : {
"hybrid-json" : {
"node" : {
"node" : {
"query" : "search",
"query" : "search",
}
}
}
}
}'
}'
funding_rounds
round_code : a
funded_year : 2012
investments
investments
name : ...

Data
{
"query" : {
{
"query" : {
"hybrid-json" : {
"hybrid-json" : {
"node" : {
"node" : {
"query" : "search",
"attribute" : "category_code",
"query" : "search",
}
}
}
}
}'
}'
funding_rounds
round_code : a
funded_year : 2012
investments
investments
name : ...

Data
{
"query" : {
{
"query" : {
"hybrid-json" : {
"hybrid-json" : {
"node" : {
"node" : {
"query" : "search",
"query" : "search",
}
}
}
}
}'
}'
funding_rounds
round_code : a
funded_year : 2012
investments
investments
name : ...

Data
Searching: Twig Query
• Query operators for Ancestor-Descendant and Parent-
Child relationships

Data
Child relationships
• Consists of a root query
Boolean

Data
Child relationships
• Consists of a root query and one or more child
Boolean
Phrase
MUST

Data
Child relationships
• Consists of a root query and one or more child and
descendant queries
Boolean
Phrase
MUST
Boolean
SHOULD

Data
Child relationships
• Consists of a root query and one or more child and
descendant queries
• Can be nested to form complex tree structure
Boolean
Phrase
MUST
Twig
NOT
Range
MUST
Boolean
SHOULD

Data
{
"query" : {
{
"query" : {
"hybrid-json" : {
"hybrid-json" : {
"twig" : {
"twig" : {
Twig Query
Operator Root Query
"root" : "funding_rounds"
"child" : [{
"child" : [{
"occur" : "must",
"node" : {
"occur" : "must",
"node" : {
"query" : "2012",
"attribute" : "funded_year"
"query" : "2012",
}
}
}]
}]
}
}
}
}
}'
}'
funding_rounds
round_code : a
funded_year : 2012
investments
investments
name : ...
Child Query

Data
{
"query" : {
{
"query" : {
"hybrid-json" : {
"hybrid-json" : {
"twig" : {
"twig" : {
"child" : [{
"child" : [{
"occur" : "must",
"node" : {
"occur" : "must",
"node" : {
"query" : "2012",
"query" : "2012",
}
}
}]
}]
}
}
}
}
}'
}'
funding_rounds
round_code : a
funded_year : 2012
investments
investments
name : ...

Data
Searching: Proximity
Query
$ curl -XGET 'localhost:9200/test/companies/_search' –d '{
"query" : { "hybrid-json" : {
"boolean" : {
"boolean" : {
"inOrder" : true,
"slop" : 0,
"clause" : [{
"inOrder" : true,
"slop" : 0,
"clause" : [{
"twig" : {
"twig" : {
"root" : "funding_rounds",
"descendant" : [{
Proximity Constraint
"descendant" : [{
Order Constraint
"level" : 2,
"node" : { "query" : "Data AND Collective" }
"level" : 2,
}]
}]
}
}
},{
},{
"twig" : {
"twig" : {
"descendant" : [{
"descendant" : [{
"level" : 2,
"node" : { "query" : "Index AND Ventures" }
"level" : 2,
}]
}]
}
}
}]
}]
}}}}'
}}}}'
funding_rounds
investments
funding_rounds
investments
name : Index Ventures
11
22
33
44
Boolean Clauses

Data
Query
"boolean" : {
"boolean" : {
"inOrder" : true,
"slop" : 0,
"clause" : [{
"inOrder" : true,
"slop" : 0,
"clause" : [{
"twig" : {
"twig" : {
"descendant" : [{
"descendant" : [{
"level" : 2,
"level" : 2,
}]
}]
}
}
},{
},{
"twig" : {
"twig" : {
"descendant" : [{
"descendant" : [{
"level" : 2,
"level" : 2,
}]
}]
}
}
}]
}]
}}}}'
}}}}'
funding_rounds
investments
funding_rounds
investments
11
22
33
44

Data
Demo

Data
Conclusion
• SIREn’s Key Feature
– Dynamic/Schema-less Data Management
– Nested Data
– High performance and scalability
– Powerful search operators
– Elasticsearch/Solr integration
• Contact
– http://sirendb.com
– SindiceTech
– info@sindicetech.com

Data
Credits
Jingle R.E.M., Martin Kaltenböck, Florian Kondert
Coordination Thomas Thurner
Martin Kaltenböck
Moderation Martin Kaltenböck
Presented by Renaud Delbru, Harish Kumar

Data
http://lod2.eu
Hope you enjoyed staying with us – if you need more detailed
information, visit us at www.lod2.eu and let us know how we can
improve to meet your expectations!
Don’t forget to register for our next webinar
20.12. 2011 - Virtuoso (Open Link Software)
24.01. 2012 - OntoWiki (University of Leipzig, Germany)
Have a great day and don’t forget ...

Data
http://lod2.eu

LOD2 Webinar: SIREn

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie LOD2 Webinar: SIREn

Ähnlich wie LOD2 Webinar: SIREn (20)

Mehr von LOD2 Creating Knowledge out of Interlinked Data

Mehr von LOD2 Creating Knowledge out of Interlinked Data (19)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

LOD2 Webinar: SIREn

Hinweis der Redaktion