Session 1 : "Intro to Apache Nutch" - Mr. Sungrem Thang (25 mins) Web crawling/Web scrapping Web crawlers,robots,spiders etc.

1. Introduction to Apache Nutch
Sung
sungrem@sigmoid.com
Sigmoid

2. Quick Introduction
Web crawling/Web scrapping
Web crawlers,robots,spiders etc.

3. Web as a directed graph
Nodes (vertices): URL-s as unique identifiers
Edges (links): hyperlinks like <a href=”targetUrl”/>
Edge labels: <a href=”..”>anchor text</a>
Often represented as adjacency (neighbor) lists
Traversal: follow the edges, breadth-first,
depthfirst,random

4. Apache Nutch
Open source web crawler written in Java
Founded in 2003 by Doug Cutting, the Lucene
creator, and Mike Cafarella
Apache project since 2004 (sub-project of
Lucene)

5. Features at a glance
Plug-in architecture,highly modular.
− Most behavior can be changed via plugins
Multi-protocol, multi-threaded, distributed
crawler
Plugin-based content processing
(parsing, filtering)

6. Some more features...
Scalable data processing framework
− Map-reduce processing
Full-text indexer & search engine
− Using Solr or ElasticSearch
− Support for distributed search
Robust API and integration options

7. Architecture Overview

8. Nutch Data Components

9. Nutch Data Components

10. Nutch Data Components

11. Nutch Workflow

12. Injecting New URL-s

13. Generating fetchlists

14. Fetching content

15. Content Processing

16. Link inversion

17. Page scoring or ranking

18. Indexing

19. Nutch Project Status & Conclusion
Two actively developed branches( nutch 1.x and
nutch 2.x)
Key difference(2.x from 1.x): Storage is
abstracted away from any specific underlying data
store by using Apache Gora for handling object to
persistent mappings.
Check out the source code to try on:
https://github.com/apache/nutch