Slides from the talk I gave at Web 2.0 Expo SF 2010.

1. Building Event Based
Systems for the Real-
Time Web
Paul Dix
http://pauldix.net
@pauldix

2. joiseyshowaa (http://www.flickr.com/photos/30201239@N00/3916914747/)

3. co-founder and CTO

5. The Talk

6. Event Based:
data update => do something

7. Batch:
need data => calculate

8. Real-Time:
data update => notification

9. Scheduled:
occasionally => do stuff

10. Event Based == Real Time

11. Monolithic
American Backroom (http://www.flickr.com/photos/41922098@N03/4247207167/)

12. Loosely
Coupled
alles-schlumpf (http://www.flickr.com/photos/29487767@N02/2855271953/)

13. Scale
'Playingwithbrushes' (http://www.flickr.com/photos/82518118@N00/2280744328/)

14. Scale with Complexity
PhOtOnQuAnTiQuE (http://www.flickr.com/photos/67968452@N00/1876685709/)

15. Scale with Team Size
alexkess (http://www.flickr.com/photos/34838158@N00/3370167184/)

16. Definition by Example

17. SQL Databases

18. events:
insert, update, delete
execution:
before, after, instead of
Event Based:
triggers

19. Batch:
count, sum, min, max

20. Real-Time?
counter cache

21. Scheduled?
cron, at, schtasks

22. Broken
nickwheeleroz (http://www.flickr.com/photos/nickwheeleroz/2474196275/in/photostream/)

23. Self Contained

24. Monolithic

25. Bad!
Road Fun (http://www.flickr.com/photos/21849473@N06/2550339131/)

26. Doesn’t
Scale
Darwin Bell (http://www.flickr.com/photos/53611153@N00/465459020/)

27. Distributed Systems
cygnoir (http://www.flickr.com/photos/35034356212@N01/163671482/)

28. Event Based:
messaging, publish
subscribe, amqp

29. Batch:
MapReduce, Hadoop

30. Real-Time:
messaging, publish/
subscribe

31. Scheduled:
delayed replication, data
backups

32. The Web
cygnoir (http://www.flickr.com/photos/35034356212@N01/163671482/)

33. Event Based:
web hooks,
pubsubhubub, rsscloud

34. Batch:
polling

35. Real-Time:
web hooks, web
sockets, pubsubhubbub,
rsscloud

36. Scheduled:
check every 1 hour

37. Distributed (internal)
Systems

38. Service Calls

39. Feed Reader (on fetch)

40. Feed Reader (on fetch)
• Update users reading list

41. Feed Reader (on fetch)
• Update users reading list
• Language Identification

42. Feed Reader (on fetch)
• Update users reading list
• Language Identification
• Named Entity Extraction

43. Feed Reader (on fetch)
• Update users reading list
• Language Identification
• Named Entity Extraction
• Cluster with other Articles

44. Feed Reader (on fetch)
• Update users reading list
• Language Identification
• Named Entity Extraction
• Cluster with other Articles
• Identify Trending Entities

45. Tightly
Coupled
Steve aka Crispin Swan (http://www.flickr.com/photos/26811962@N05/3704553536/)

46. Alejandro Groenewold (http://www.flickr.com/photos/17618485@N07/3294535473/)

47. Publish/Subscribe
(pubsub)

48. RabbitMQ

49. AMQP

50. Exchanges

51. Queues

52. Bindings

53. Routing Key

54. Exchange Type

55. Topic Exchange

56. <token>.<token>...

57. # - 0 or more wildcard

58. * - 1 wildcard

64. Include Data in the
Message

65. Data is the API

66. Another Example:
error logging

67. routing key:
domU-12-31-39-07.feed_fetcher

68. binding:
*.feed_fetcher

69. binding:
#

70. Loosely
Coupled
° ρЯίтΛм ° (http://www.flickr.com/photos/39046851@N08/3829500850/)

71. Programmable Web 2.0

72. Event-Based

73. Real-Time

74. Web Hooks

75. GitHub
Post-Receive Hooks

77. git push triggers hooks

78. POST to Post-Recieve
URLs

79. {
"before": "5aef35982fb2d34e9d9d4502f6ede1072793222d",
"repository": {
"url": "http://github.com/defunkt/github",
"name": "github",
"description": "You're lookin' at it.",
"watchers": 5,
"forks": 2,
"private": 1,
"owner": {
"email": "chris@ozmm.org",
"name": "defunkt"
}
},
"commits": [
{
"id": "41a212ee83ca127e3c8cf465891ab7216a705f59",
"url": "http://github.com/defunkt/github/commit/41a212ee83ca127e3
"author": {
"email": "chris@ozmm.org",
"name": "Chris Wanstrath"
},
"message": "okay i give in",
"timestamp": "2008-02-15T14:57:17-08:00",

80. Data is the API

81. Continuous Integration

82. Bug Tracking

83. Campfire

84. Continuous
Deployment

85. PubSubHubbub

86. RSS or Atom Feed
<link rel=”hub” href=”http://pubsubhubbub.appspot.com/”>

87. Subscriber tells Hub

88. Hub verifies with
Subscriber

89. Publisher notifies Hub

90. Hub gets from
Publisher

91. Hub sends to
Subscribers

92. Data is the API

93. OAuth + Web Hooks

94. Concerns

95. don’t make web-hook
callbacks while a user is
waiting
Responsive

96. what if the subscriber
doesn’t respond?
retry logic
notifications
Retries

97. use OAuth.
SSL for sensitive data
Security

98. Possibilities

99. Real-Time

100. Event Based

101. Communication

102. Are these really real-
time?

103. Data Consistency

104. in brewer’s CAP theorem he talked
about the relationship bet ween three
requirements when building
distributed systems. consistency,
availability, and partition tolerance.
Eric Brewer’s CAP
theorem

105. consistency means that an
operation either works completely
or fails. this is also referred to as
atomic.
Bug tracker example.
consistency

106. availability is pretty self
explanatory. a service is
available to ser ve requests.
Twitter example
availability

107. when you replicate data across multiple
systems, you create the possibility of forming
a partition. this happens when one or more
systems lose connectivity to other systems.
partition tolerance is defined formally as “no
set of failures less than total net work failure
is allowed to cause the system to respond
incorrectly”
partition tolerance

108. pick two

109. Can have all three
“is a special form of weak
consistency. if no new updates are
made to an object, eventually all
accesses will return the last
updated value.”
Werner Vogels’
eventual consistency

112. Design Around Eventual
Consistency

113. In Closing...

114. Event-Based == Real Time

115. Pubsub

116. Loosely Coupled

117. Expose Data

118. Third Party
Programmers

119. Extend and Improve

120. many heads are better...
shoothead (http://www.flickr.com/photos/66621443@N00/519240547/)

121. than one.
John Spooner (http://www.flickr.com/photos/29809546@N00/315157691/)

122. Questions?
Paul Dix
http://pauldix.net
@pauldix