1. © 2015 black swan
blackswan
1
Analytic SEARCH
…to find better answers faster.
Klaus Kater
klaus.kater@blaswa.com
II-SDV 2015, Nice

2. © 2015 black swan
blackswan
2
Surface Web
Deep Web
Corporate
Resources
Analytic SEARCH
• Targeted Crawling: Surface Web, Deep Web and Corporate Data
• Analyzing/Aggregating: Link content from different data sources
• Scraping: Extract structured data from unstructured information

3. © 2015 black swan
blackswan
3
SEARCHCORPUS®
Surface Web
Deep Web
Corporate
Resources
in some Scientific Publication
Match: some scientific term
Search Hit:
Referenced document
or database entry.
• Indexing crawled documents and extracted data
• Content AND context uniquely identify a document
• Documents annotated with context build a SEARCHCORPUS®

4. © 2015 black swan
blackswan
4
P3 – Use Cases
• Pull: Viewers used as SEARCH interface in any application
• Pimp: Extend existing data with crawled information
• Push: Profile driven notification of events (Email, …)
pull
push
pimp

5. © 2015 black swan
blackswan
5
Analytic SEARCH
Features:
Modeling / Documentation
•Graphical filter chain designer
•Filter chain template collection
•Rich text documentation of project
•Explanation component for SERACHCORPUS®
•Multidimensional SEARCHCORPUS®
•Logging module for job execution
Analysis / Data Quality
•Full text search
(Fuzzy, regular expressions, proximity, Levensthein, boolean
operators)
•Content detection (remove advertising, etc.)
•Text extractors for common formats
•Archive extractors
•White list / black list include / exclude rules
•Web crawler
(per seed url parameter set and behavior rules)
•Data extraction, HTML/XML scraping
•Graphical regular expression designer
•Text operations (cleansing)
•Term frequency models
•Pattern matching
•Job history / log files (3 log levels per filter)
•Executive jobs for sequential job execution
Administration / Portal
•User rights administration
•Job scheduler / dashboard
•User monitoring
•External viewer / Webservice permission management
Implementation / Deployment
•Plugin mechanism for filter types
•Filter type developers API
•Plugin mechanism for container types
•Container type developers API
•Plugin mechanism for viewers
•Web based multi user environment
•Parallel job execution
•JAVA application server
•Parallel and distributed deployment
•Hadoop and map/reduce components

6. © 2015 black swan
blackswan
6
Impressions

7. © 2015 black swan
blackswan
7
Impressions

8. © 2015 black swan
blackswan
8
Impressions

9. © 2015 black swan
blackswan
9
Impressions

10. © 2015 black swan
blackswan
10
Proposal
Analytic SEARCH on multidimensional SEARCHCORPORA1) that combine
structured and unstructured data from multiple sources like
…to find better quality answers faster.
• medical records
• drug databases
• scientific publications
• corporate financial filings
• news tickers
• company websites
• publications
• social media
• blogs
• data feeds
• uncounted internal data
sources that currently cannot
be searched at all...

11. © 2015 black swan
blackswan
11
Contact:
Klaus Kater
Email:
Website:
klaus.kater@blaswa.com
http://www.blaswa.com
Phone: +49 7441 5203720
Contact

12. © 2015 black swan
blackswan
12
Backup

13. © 2015 black swan
blackswan
13
Technology
Analytic SEARCH Product Architecure
 Building Filter Chains
 Scheduling Jobs and
Processing
 Crawling for Information
 Multidimensional
SEARCHCORPUS®
 Visualization / Searching
 Social Media Interfaces
 Data Warehouses