Challenges with crawling AJAX pages on the web and their solutions.

1. Scraping AJAX
Pages
Big Data made small

2. What’s AJAX on a web page?
1. Filters 2. Load
more results
3. Forms
and others...

3. GET vs. POST
Client Server
Client Server
GET
POST
http://example.com?date=20140410
http://example.com
Payload
Form Data, JSON Strings, Query Parameters, View States, etc.

4. What makes crawling AJAX difficult?

5. Challenge 1- Javascript Calls
Solution- Emulate Javascript calls using headless browsers
Data fetched
from under
Javascript
code

6. Challenge 2- Fetch Bandwidths
Solution-
Optimize fetch limits
Incomplete page fetched
because of low fetch age
Image Credit: ticketmaster.com

7. Challenge 3- .NET Architectures
Solution- Track states, pass event validations, restore states for
mitigation
Viewstate

8. Challenge 4- Page Encoding
Solution- Send request (content type, media type,
accept field parameters) and parse responses in
same format as expected by server

9. Use Case- Crawl Ticketing Sites

10. Thank You!
Have specific queries on AJAX crawling?
Reach out to info@promptcloud.com.