Content analysis for ECM with Apache Tika

Content analysis for ECM with Apache Tika Paolo Mottadelli -

[email_address]

ON BOARD!

Agenda

Main challenge Lucene index

Other challenges

A real world challenge ? ? ? Searching .docx .xlsx .pptx in Alfresco ECM

Agenda

What is Tika? Another Indian Lucene project? No.

What is Tika? It is a Toolkit

Current coverage

A brief history of Tika Sponsored by the Apache Lucene PMC

Tika organization Changing after graduation

Getting Tika … and contributing

Tika Design

The Parser interface <ul><li>void parse(InputStream stream, ContentHandler handler, Metadata metadata) throws IOException, SAXException, TikaException; </li></ul>

Tika Design

Document input stream

Tika Design

XHTML SAX events <ul><li><html xmlns="http://www.w3.org/1999/xhtml"> </li></ul><ul><li><head> </li></ul><ul><li><title>...</title> </li></ul><ul><li></head> </li></ul><ul><li><body> ... </body> </li></ul><ul><li></html> </li></ul>

Why XHTML? <ul><li>Reflect the structured text content of the document </li></ul><ul><li>Not recreating the low level details </li></ul><ul><li>For low level details use low level parser libs </li></ul>

ContentHandler (CH) and Decorators (CHD)

Tika Design

Document metadata

… more metadata: HPSF

Tika Design

Parser implementations

The AutoDetectParser <ul><li>Encapsulates all Tika functionalities </li></ul><ul><li>Can handle any type of document </li></ul>

Type Detection MimeType type = types.getMimeType(…);

tika-mimetypes.xml <ul><li>An example: Gzip </li></ul><ul><li><mime-type type="application/x-gzip"> </li></ul><ul><li><magic priority="40"> </li></ul><ul><li><match value="3713" type="string“ offset="0" /> </li></ul><ul><li></magic> </li></ul><ul><li><glob pattern="*.tgz" /> </li></ul><ul><li><glob pattern="*.gz" /> </li></ul><ul><li><glob pattern="*-gz" /> </li></ul><ul><li></mime-type> </li></ul>

Supported formats

A really simple example <ul><li>InputStream input = MyTest.class.getResourceAsStream("testPPT.ppt"); </li></ul><ul><li>Metadata metadata = new Metadata(); </li></ul><ul><li>ContentHandler handler = new BodyContentHandler(); </li></ul><ul><li>new OfficeParser ().parse(input, handler, metadata); </li></ul><ul><li>String contentType = metadata.get(Metadata. CONTENT_TYPE) ; </li></ul><ul><li>String title= metadata.get(Metadata. TITLE) ; </li></ul><ul><li>String content = handler.toString() ; </li></ul>

Demo ?

Future Goals

Who uses Tika?

Agenda

ECM: what is it?

ECM: Manage <ul><li>Indexing </li></ul><ul><li>Categorization </li></ul>* *

ECM: we love SEARCHING!

Don’t do it on your own Tika shields ECM from using many single components

Agenda

Alfresco: short presentation

Who uses Alfresco?

Alfresco Repository JSR-170 Level2 Compatible

Repository Architecture Services Components Storage Hibernate Content Lucene Content Index Database Search Node Node Content Query Index

Alfresco Search

Use case

Without Tika:

Step 1

Step 2 <ul><li>for (ContentTransformer transformer : transformers) </li></ul><ul><li>{ </li></ul><ul><li>long transformationTime = transformer.getTransformationTime(); </li></ul><ul><li>if (bestTransformer == null || transformationTime < bestTime) </li></ul><ul><li>{ </li></ul><ul><li>bestTransformer = transformer; </li></ul><ul><li>bestTime = transformationTime; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>return bestTransformer; </li></ul>ContentTransformerRegistry Provides the most appropriate ContentTransformer

Step 2 (explained) Too many different ContentTransformer implementations

Step 3 Transform public void transformInternal(ContentReader reader, ContentWriter writer, TransformationOptions options) throws Exception { ... HSSFWorkbook workbook = new HSSFWorkbook(is); ... for (int i = 0; i < sheetCount; i++) { HSSFSheet sheet = workbook.getSheetAt(i); String sheetName = workbook.getSheetName(i); writeSheet(os, sheet, encoding); } ... } Example: PoiHssfContentTransformer

Step 3 (explained) Too many different ContentTransformer implementations ... again !?!

Step 4 Lucene index creation ContentReader reader = contentService.getReader(nodeRef, propertyName); ContentTransformer transformer = contentService.getTransformer(reader.getMimetype(), MimetypeMap. MIMETYPE_TEXT_PLAIN); transformer.transform(reader, writer); reader = writer.getReader(); . . . . . . . . doc.add(new Field(attributeName, reader, Field.TermVector. NO));

Let’s do it using Tika

Step 1 + Step 2 + Step 3 String name = “resource.doc” InputStream input = getResourceAsStream(name); Metadata metadata = new Metadata(); ContentHandler handler = new BodyContentHandler(); new AutoDetectParser().parse(input, handler, metadata); String title = metadata.get(Metadata. TITLE); String content = handler.toString();

Step 1 to 4 (compressed) String name = “resource.doc” InputStream input = getResourceAsStream(name); Reader reader = new ParsingReader (input, name); . . . . . . doc.add(new Field(attributeName, reader , Field.TermVector. NO));

Results: 1 & 2

Extension use case Adding support for Microsoft Office Open XML Documents (Office 2007+)

Apache POI Apache POI provides Text Extraction support for Office OpenXML formats and An advanced coverage of SpreadsheetML specification (WordprocessingML & PresentationML to come)

Apache POI Apache POI status

Apache POI TextExtractors <ul><li>POIXMLDocument document; </li></ul><ul><li>Package pkg = Package.open(stream); </li></ul><ul><li>textExtractor = ExtractorFactory.createExtractor(pkg); </li></ul><ul><li>if (textExtractor instanceof XSSFExcelExtractor) { </li></ul><ul><li>setType(metadata, OOXML_EXCEL_MIMETYPE </li></ul><ul><li>document = new XSSFWorkbook(pkg); </li></ul><ul><li>} </li></ul><ul><li>else if (textExtractor instanceof XWPFWordExtractor){…} </li></ul><ul><li>else if (textExtractor instanceof XSLFPowerPointExtractor){…} </li></ul><ul><li>setPOIXMLProperties(metadata, document); </li></ul>

Can we find it?

Results: 3 & 4

Q & A [email_address]

Content analysis for ECM with Apache Tika

Du liest eine Vorschau.

Hier ansehen

Content analysis for ECM with Apache Tika

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Ähnlich wie Content analysis for ECM with Apache Tika (20)

Weitere von Paolo Mottadelli (11)

Aktuellste (20)

Content analysis for ECM with Apache Tika