Web crawl with Elixir
•
3 gefällt mir•719 views
스켈터랩스에서 Elixir 로 web site 를 크롤하면서 겪은 경험을 공유합니다. 의견은 이재철(mr.jechol@gmail.com) 으로 주세요.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Web crawl with Elixir
Ähnlich wie Web crawl with Elixir (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Web crawl with Elixir
- 2. Who am I? • Jechol Lee (mr.jechol@gmail.com) • Software engineer at Skelterlabs • Loves elixir, elm, ruby • We are HIRING!
- 3. First try
- 4. First try
- 5. First try Database Crawler JSONHTTP Crawler HTMLHTTP Products Products Save Queue Products
- 6. SQL Site A ES SaveQueue Task Page Task Item Task Item product req Sup Sup (site A) Sup (simple 1 for 1) Task.SupTask.Sup Sup (site B) progress start_link
- 7. Crawler speed > Site rate limit Crawler WebsiteHTTP 10/s
- 9. DB speed < Crawler speed Save QueueDatabase Crawler100/s 200/s
- 10. Memory exhaustion Save QueueDatabase Crawler100/s 200/s SaveQueue Out of Memory
- 11. Solution
- 12. OOM Problem : Producer driven 200/s Crawler Save QueueDatabase 100/s
- 15. Elixir GenStage (2016 / 7)
- 16. GenStage : Demand-driven Crawler Save Queue DEMAND 100 100Database 100/s
- 17. Memory usage Producer driven Demand driven
- 18. Site rate limit :TokenBucket
- 19. Site rate limit :TokenBucket Crawler WebsiteHTTP Burst Token Bucket 60/min
- 21. Network Requests Overflow Page 2 Page 1 item Page 1 item Page 1 item
- 22. Network Requests Overflow Page 1 item Page 2 item Page 3 Page 1 item Page 2 item Page 2 item
- 23. Network Requests Overflow Can't depend on random processing order. Page 1 item Page 2 item Page 4 Page 3 item Page 1 item Page 2 item Page 3 item Page 3 item
- 24. Priority Queue Page 1 item Page 2 item Page 3 Page 1 item Page 1 item Page 2 item Page 2 item
- 28. Rate limit byTokenBucket twotap.com c21stores.com WebProxy PriorityQueue + TokenBucket HTTP SPAWN HTML Parser C21 HTML Parser C21 HTML Crawler C21 JSON Crawler Lego
- 30. Fault-tolerance by SupervisionTree Supervisor Supervisor C21 Task Supervisor ErrorMonitor WebProxy PriorityQueue + TokenBucket HTML Parser C21 HTML Parser C21 SaveQueue HTML Crawler C21 JSON Crawler Lego
- 31. Tree for Multiple Crawlers JSON Crawler GNC Supervisor JCPenney Supervisor Supervisor C21 Task Supervisor ErrorMonitor WebProxy PriorityQueue + TokenBucket HTML Parser C21 HTML Parser C21 SaveQueue HTML Crawler C21 JSON Crawler Lego
- 32. Final JSON Crawler GNC Supervisor JCPenney Supervisor Supervisor C21 Task Supervisor ErrorMonitor sentry.io MONITOR {:DOWN, :page_not_found} twotap.com c21stores.com WebProxy PriorityQueue + TokenBucket HTTP SPAWN HTML Parser C21 HTML Parser C21 PRODUCT DEMAND DEMAND PRODUCT SQL ES SaveQueue HTML Crawler C21 JSON Crawler Lego
- 33. Building Blocks
- 35. GenServer vs Task • Tasks don't provide services. → No handle_call, etc. • Just run a function and exit.
- 36. Task.Supervisor.async • Not trap exit. • Caller process dies together. • Not restart task.
- 37. Task.async vs Task.Supervisor.async Only later builds supervision relationship so that visible using observer.
- 38. End