Presentation of HOBBIT Joint Event Post-EDF 2016. Eindhoven, Netherlands (This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688227.)

1. HOBBIT
in a Nutshell
Axel Ngonga
Horizon 2020
GA No 688227
01/12/2016–30/11/2018
Joint Event Post-EDF 2016
Eindhoven, Netherlands
July 1st, 2016
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 1 / 17

2. A Lot of Data
1
1http://www.ibmbigdatahub.com/infographic/four-vs-big-data
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 2 / 17

3. A Lot of Tools
2
2https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.png
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 3 / 17

4. Core Questions
Developers: How good is my tool?
Vendors: Who is my tool good for?
Users: Which tool(s) should I use for
my application?
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 4 / 17

5. Many Questions
Where are the current bottlenecks?
Which steps of the data lifecycle are
critical?
Which solutions are available?
Which key performance indicators
are relevant?
How well do or should tools
perform?
How do existing solutions perform
w.r.t. relevant indicators?
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 5 / 17

6. GERBIL
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 6 / 17

7. GERBIL
Evaluation platform for NER/NEL
9 reference annotation systems
11 reference datasets
Benchmarking 10× faster
Archiving of results
Citeable URIs
Additional analysis
Open-source project
Local deployment
Normalized implementation of KPIs
Online instance at
http://gerbil.aksw.org/
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 6 / 17

8. GERBIL
Annotator Tasks
NIF-based Annotators 2519
Babelfy 958
DBpedia Spotlight 922
TagMe 2 811
WAT 787
Kea 763
Wikipedia Miner 714
NERD-ML 639
Dexter 587
AGDISTIS 443
Entityclassifier.eu NER 410
FOX 352
Cetus 1
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 7 / 17

9. HOBBIT
Rationale
A community-driven benchmarking framework for the community
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 8 / 17

10. HOBBIT
Rationale
A community-driven benchmarking framework for the community
Focus on Big Linked Data
Cover all steps of the Linked Data lifecycle
Used by a growing number of companies
Mature and maturing technologies
Open benchmarks based on industrial data
and use cases
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 8 / 17

11. HOBBIT
36-month project
Project begin: Dec. 1st, 2015
Project volume: ca. 4 million Euros
10 partners
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 9 / 17

12. Aims
1 Gather real requirements
Performance indicators
Performance thresholds
2 Provide universal benchmarking platform
Standardized hardware
Comparable results
3 Develop benchmarks based on real data
4 Periodic benchmarking challenges
5 Periodic reporting
6 Found independent Hobbit association
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 10 / 17

13. Overview
Data Collection
Industry
data
Measure Collection
Benchmark Creation
Benchmark 1
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
Benchmark 2
Benchmark n
HOBBIT
Platform
Solution 1
Solution k
Solution 2
Challenges
Reports
Participants/Community
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 11 / 17

14. We offer a benchmarking platform
Controller
Data
Generator
Task
Generator
Data
Generator
Data
Generator
Task
Generator
Task
Generator
FrontendSystem Adapter
System
data flow
creates component
Store
SPARQL
Endpoint
Analysis
Benchmark
Evaluator
Module
Eval. Store
Message Bus
Node
Observer
Logging
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 12 / 17

15. We offer a benchmarking platform
Addresses all steps of the Linked
Data Lifecycle
Benchmarks derived from industry
use cases
Real data under the benchmarks
Scalable size of benchmarks
Open-source implementation
Local instance on server cluster
Uses established deployment
technologies
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 13 / 17

16. We offer benchmarks
Streaming and static deterministic benchmarks
Realistic benchmarks
Controlled volume and velocity
Generation and Acquisition
Conversion of XML into RDF
Entity recognition and linking
Relation extraction
Analysis and Processing
Link Discovery
Machine Learning
Supervised and unsupervised
Storage and Curation
Triple stores
Versioning
Incl. updates
Visualization and Services
Question Answering
Faceted Browsing
Usage-based benchmarks
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 14 / 17

17. We offer datasets
Twitter7 dataset
ca. 476 million tweets
ca. 17 million users
ClueWeb12
ca. 733 million websites
1+ billion annotations
Printing Machinery
ca. 6.5 trillion events
1500 printing machines
LIVED
ca. 2.5 billion measurements
6 households, two years
Injection molding industry
ca. 120 million measurements
Traffic data archive
ca. 15 trillion speed measurements
100+ million road segments
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 15 / 17

18. We need ...
Your use cases
Participate in the survey
Join the HOBBIT community
Provide KPIs
Provide datasets
Join the platform development
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 16 / 17

19. Thank You
http://project-hobbit.eu/get-involved/
http://goo.gl/forms/1iRIoG4Xpb
https://twitter.com/hobbit_project
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 17 / 17