Slides from a presentation I gave to Austin on Rails on March 24, 2009. It describes building an application called TweetReach that uses Sinatra, Tokyo Cabinet/Tyrant and Grackle.

1. Off the Reservation with TweetReach Hayes Davis Co-Founder, Appozite [email_address] @hayesdavis

4. Be simple(ish)

5. Be reasonably fast for a resonable number of users

6. Fit within Twitter API limits

8. Persistence layer: Tokyo Cabinet + Tokyo Tyrant + Memcache-client

9. Twitter API: Grackle

13. Sinatra Example set :port, 3000 get '/' do erb :index end get '/reach' do @query = params[:q] tr = TweetReach.new(username,pass) @results = tr.measure_reach(@query) erb :reach_results end

15. Stats: 2.5M inserts/second, 3M queries/second, Store 8 exabytes

16. Has a server called Tokyo Tyrant

18. B-tree: functionally the same as the hash DB but with ordered keys based on a user-defined function

19. Fixed-length: basically a giant array which you index into by offset keys

20. Table: similar to a relational DB except no predefined schema (ala CouchDB). Can index columns and query them

22. Provides replication, failover, etc

24. Does not provide expiration

26. Reach calculations require O(n) looping and retrievals – much faster with a dedicated key-value store vs. a relational DB

27. Need a read-through cache so I don't have to hit Twitter so often

28. Can store stuff for relatively long periods that I may not want to keep solely in memory

30. I wrote it based on my experience building CheapTweet.com

31. Dynamically makes requests to the APIs via a generic syntax that maps to Twitter URIs

32. Builds OpenStructs of returned JSON or XML dynamically

33. Grackle Example client = Grackle::Client( :username=>'some_user', :password=>'secret' ) #GET http://twitter.com/users/show.json?id=hayesdavis client.users.show? :id=>'hayesdavis' #POST http://twitter.com/statuses/update.json client.statuses.update! :status=>'howdy world' #GET http://search.twitter.com/search.json?q=AoR client[:search].search? :q=>'AoR'

34. Building TweetReach

35. Basic Structure TweetReach Calculator Read-Through Cache Sinatra

37. Sinatra apps are a single ruby file (I called it app.rb)

38. Just require rubygems and sinatra

39. Run “ruby app.rb”

41. Run “ttserver -port <port> filename.<ext>”

42. The type of database comes from the filename extension. I used “tch” which is the Hash engine

43. Get Mike Perham's memcache-client gem

45. Install the hayesdavis-grackle gem

46. Just require “grackle”

47. Lessons Learned

49. No helpers for easy formatting (stole things from Rails)

51. You'll most likely need to roll one or two of your own things

53. Would definitely use it again for this type of task

55. Still a very new library so I'm sure there are places that need cleaning up

57. http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/

58. http://www.scribd.com/doc/12016121/Tokyo-Cabinet-and-Tokyo-Tyrant-Presentation

59. http://github.com/hayesdavis/grackle