1. Writing a distributed crawler system using gevent and redis Alex Dong @alexdong

3. Async 101

4. Gevent: the monkey king

5. Redis: data structure server

6. Lessons learned

8. How many links today?

9. What was the project Google employee #1 working on?

11. DNS resolution

12. Request HTTP Header

13. Download full content

14. Store to local file store, database and index

15. Scheduling, throttling, status monitoring, scale up by flicking on more servers.

17. Yield on IO/socket, but

19. Context switch overhead

21. Cooperative multitasking

22. epollfd = epoll_create ();

23. epoll_ctl ( epollfd , EPOLL_CTL_ADD, listen_sock , &ev)

24. epoll_wait ( epollfd , events , MAX_EVENTS, -1)

28. Working: Hashtable - HSET, HDEL, HEXISTS

29. One and only one: Sets - SADD, SPOP

32. 25% code for status update and monitoring

34. DNS is down?

35. ISP's bandwidth?

36. Large file download?

37. Scheduler re-submit tasks?

40. Carefully profile to make sure socket.recv becomes the main bottleneck.

41. The get_title crisis

42. Rule of thumb: load average < 1 to saturate 10M bandwidth

43. Twitter: @alexdong trunk.ly/?q= from:alexdong + gevent