This document discusses web hooks and the programmable web. It begins by defining web hooks as user-defined HTTP callbacks that can make web services extensible and empower users. The document then explores how web hooks enable a truly programmable web by allowing programs to extend web applications, unlike APIs which only allow programmatic use. It discusses how web hooks could enable applications to be combined like UNIX command line tools using input/output pipes. Finally, it argues that for the web to be fully programmable, web applications need to implement web hooks for common events.
1. WEB HOOKS
and the
Programmable World of Tomorrow
Jeff Lindsay
three years after coining web hooks, people are starting to get excited about them.
iâm going to share what they are, why theyâre signiïŹcant, and whatâs going on
in the growing web hooks ecosystem
2. WEB HOOKS
and the
Programmable World of Tomorrow
Jeff Lindsay
iâve given a version of this talk before, but this time iâd like to try and focus on this
âprogrammableâ idea that helped inspire web hooks.
3. WARNING:
Emergent dynamics described ahead.
was a guy really excited hearing about web hooks, but was disappointed to see what they are.
iâm describing a game changer based on a mechanism that takes one line of code to describe.
seems so simple, people assume you can only do simple things. quite the contrary...
4. go. simple mechanisms can create rich dynamics.
easily teach somebody the rules, doesnât mean they see the implications of those rules.
i want to share the implications of this simple mechanism.
5. in 1997, jon udell talks about websites as data sources that can be reused and remixed
today that idea isnât very novel
6. âa new programming paradigm
that takes the whole Internet
as its platformâ
he envisions the Internet as a programming paradigm
7. The Programmable Web
âThe Web as Platformâ
it starts to manifest as the programmable web and talk of âthe web as a platformâ
8. The Programmable Web
APIs and Mashups
john musser here starts to track apis and mashups in 2005. there are now over 1k apis and
increasingly more.
apis and mashups became the foundation of the programmable web...
9. Pro g ramm at ic
The Programmable Web
APIs and Mashups
but are they? after thinking about web hooks and what they can give us, i realized apis and mashups
donât make a programmable web. they make a programmatic web
10. Pro g ramm at ic
The Programmable Web
Twitter API lets you use Twitter programmatically.
It does not let you program Twitter to do more.
they let you use web apps programmatically. they donât let you program them to do more,
whereas web hooks can. so i like to argue that web hooks will bring about the *real* programmable
web.
11. Web Service
interwebz
User
so letâs get speciïŹc. hereâs an example scenario showing the use of web hooks.
12. Web Service
Hi, Iâm Twickr,
a new web service.
interwebz
User
22. âTwickrâ
It doesnât matter to you.
Whatever I want...
interwebz
User
web hooks are just user-deïŹned http callbacks
23. Web Service
EXTENSIBLE
interwebz
EMPOWERED
User
but do you see what they did? they made the web service extensible and empowered the
user.
24. a friend of mine got me to explain web hooks to her. not a programmer, but has iphone.
she compared it to jailbreaking the iphone: letting users do what they want, customize, add
apps, etc
having web hooks is like jailbreaking your web apps. opens functional extensibility.
25. a pseudo code example of what the heart of implementing web hooks looks like.
SIMPLE: make an HTTP request to a user-speciïŹed URL on major events
28. âWhen a customer pays you, PayPal posts a notiïŹcation to your server at a URL you specify.â
its framed as a notiïŹcation, but that doesnât properly imply the usage its intended for:
integration.
29. started thinking about this in 2006.
everything ïŹashed before my eyes and was very confused why it wasnât used more.
felt like i was taking crazy pills. today i know why...
31. both have been around longer. rest is simpler..
in fact, itâs almost described as âusing HTTP properlyâ
but not until it got a name could it be used in discourse to make it popular
32. REST
Hooks
rest apis and web hooks are two sides of the same coin
they complement each other in ways iâll get to later
but i just want to give this pattern a name, and start associating some ideas with it
33. Push
Pipes
Plugins
talk is split into three sections
ways to look at the use of web hooks
icons will hopefully make more sense as i talk about them
34. Push
letâs get started with push
people are starting to talk about push and pubsub on the web... although its not the ïŹrst
time
35. 1998 (predating rss) microsoft submitted an internet draft to extend http
to provide a basic pubsub framework called GENA
40. but then in 2000, rss 0.92 was released. ïŹve years later it gets an icon and widespread
adoption.
it started with blog feeds, then comment feeds...
41. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
42. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
43. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
44. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
45. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
46. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
47. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
48. it makes you think of feeds like in the telecom world
52. then it gives it to us. and we do this over and over.
are we there yet? are we there yet?
53. ?
feeds made sense in a world where feed readers ran on
desktops that couldnât be pushed to over http
54. ?
of course, now we have other web applications consuming feeds and it doesnât make sense.
even most of our feed readers have become web applications
55. SUP is a recent incremental solution to increase the eficiency of consuming large amounts of
feeds.
however, its based on polling and is essentially yet another feed,
so i couldnât help...
56. SUP DAWG, WE HEARD YOU LIKE FEEDS
SO WE MADE YOU A FEED OF OUR FEEDS
57. evan and kellan gave a great talk a while back about this push issue, and their proposed solution:
xmpp
i have a condensed version of this talk. the slides speak for themselves
58.
59.
60.
61.
62.
63.
64. (aka XMPP)
they have a great point. polling sucks, and xmpp is a pretty good solution for data streams
65. but itâs kind of heavy weight. it does a lot and makes a decently complex little system.
luckily itâs not *that* hard to use with todayâs library support
66. evan and kellan did point out there are extremes when it comes to data streams.
most data streams will probably fall somewhere in between,
but i do think xmpp is suited for the fast and furious end
67. joshua schachter of delicious responded to this talk in a blog post.
he basically suggests web hooks as an alternative
68. âOne solution that occurred to me at the time was to build a simple callback system over HTTP.
This would fall comfortably between full polling and full persistent publish/subscribe.â
he says they fall comfortably between polling and xmpp. i agree, and think they can cover
most use-cases
69. others seem to agree....
this is a standard for discovering and subscribing to content changes.
they let you use web hooks OR xmpp, which is nice.
good idea, but (like GENA) standard specs alone donât get very far
70. gnip is a service that some may have heard of (but donât understand)
72. Source Destination
Protocol + Mechanism Protocol + Mechanism
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
theyâre basically an adaptor for data streams, letting you pick your own protocol and mechanism, no
matter what the feed publisher is providing.
73. Source Destination
Protocol + Mechanism Protocol + Mechanism
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
however, they recently dropped support to consume via xmpp, making web hooks their primary push
mechanism for consuming data streams.
74. Source
Protocol + Mechanism Protocol + Mechanism
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
as an example, if i wanted to get digg updates via web hooks, gnip will poll digg for me and invoke my
callback with the content as new updates come in.
75. Source
Protocol + Mechanism Protocol + Mechanism
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
alternatively, i could even poll gnip for digg updates. although seemingly redundant, it helps ease the
load on digg and allows gnip to provide ïŹltering functionality.
76. Protocol + Mechanism Protocol + Mechanism
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
but another example might be if twitter provides an xmpp stream, but i already have a polling setup...
gnip would let me poll instead of integrate xmpp.
77. microformats
in promoting web hooks, i sort of look up to the way microformats work in values and process.
very ground up, grassroots... take existing popular use-patterns and make it a convention.
microformats can be viewed as an alternative xml+rdf
78. xml+rdf vs microformats
âHere's a new language we want you to learn,
and now you need to output these additional
ïŹles on your server. It's a hassle.
(Microformats) lower the barrier to entry.â
tantek is a big microformats evangelist. he says....
so i told him about web hooks. âwhat are they?â âpush over httpâ
âhow are they di than xmpp?â âtheyâre a lightweight alternativeâ lower the barrier to entry...
79. xmpp vs web hooks
âGood. XMPP needs a competitor.â
he says...
this was encouraging. i mean, when tantek talks, you listen...
if for no other reason than
81. unfortunately, xmpp has many features and proposals to do pubsub that web hooks doesnât.... after
all, itâs just a creative use of http requests, not a messaging protocol. so in order to do pubsub with
web hooks, you need more to be implemented. well, brad ïŹtzpatrick and brett slatkin are working on
that.
82. Push is good.
so the moral here is that push is good and thereâs ïŹnally a demand for it. we have some good
solutions waiting adoption...
83. XMPP is ideal when needed,
but Web Hooks generally do the job.
as far as xmpp vs web hooks, i think they both have their place.
web hooks are easier, so you might as well default to web hooks unless you really need
xmpp.
84. But push is not the point.
however, as i framed this talk, itâs really not about push. itâs a nice way to get the social
media kids into bed with web hooks, but hooks are really about more than pubsub and
notiïŹcations. otherwise i would have called them web notiïŹcations or something...
85. Pipes
instead of push, i was more drawn by the pipes metaphor. i wanted to integrate and
orchestrate the web applications i use, conceptually similar to unix pipelining.
86. so i thought about it. pipes were a really amazing feature that let you get more out of your commands
by letting you chain them together. it sort of let you program without programming, combining
commands in ways that werenât necessarily intended by the author.
87. Input Output
Program
all from a bit of infrastructure involving input and output
88. STDIN STDOUT
Program
STDERR
stdin, stdout were available to reroute wherever the user wanted
most common use was chaining commands together: piping
89. xargs
wget
echo
mail
grep
wc
cat
so you had all these simple little programs, that might not even be useful alone
90. cat
xargs
wget
echo
mail
grep
wc
string them together...
92. cat grep mail
xargs
wget
echo
wc
and you have something more useful than the sum of the parts
93. Write programs that do one thing and do it well.
Write programs to work together.
Write programs that handle text streams,
because that is a universal interface.
this helped put forth the unix philosophy and encouraged building these small composable
commands
94. STDIN
Program
but it doesnât work without the output. it just breaks.
95. API
Web App
unfortunately thatâs how the web is today.
we can talk to web apps, but they really canât talk to us. or anything else really.
96. API Hooks
Web App
itâs not that they canât, they just donât. we just need to start putting hooks in so they can.
those roles are best played by mechanisms that use the protocol the web is built on: http
99. cat grep mail
Basecamp
so we want to combine web applications like we can CLI programs.
get more than the sum of the parts.
web hooks open up this possibility, but need like APIs, need to be implemented
101. Basecamp
Project ïŹnished
Todo completed
Milestone created
Contact added
File uploaded
102. Basecamp
My handler
http://example.com/handler
users can write handlers that are just web scripts.
they have a url, and thats what you give basecamp
103. Basecamp
My handler
http://example.com/handler
itâs code. it can do anything from there.
integrate with other services, make a phone call, order pizza, whatever
104. Todos
Basecamp
for example, all these apps share data about todos. they each have respective specialized talents,
but all work with todos. by putting hooks on todo CRUD,
you can use their apis to keep them synced pretty well. magically. real-time.
107. Code as glue
based on the idea that web urls can run code. and code can do anything.
108. when i ïŹrst thought about this, cheap PHP hosting was all over, but it could be even simpler.
there are paste bins like this one. made to share formatted code with people over IRC or email
109. put in some code you want to share or ask a question about
113. Basecamp
Project finished
http://pastie.org/run/24576
for example, basecamp. now when you ïŹnish a project, everybody meets for shots in the break room.
115. just hit a button, write code, hit save, share the url. itâs javascript
116. obviously app engine, although itâs a little more involved than appjet for quick handlers.
but it is an option for python.
and there are ruby/rails hosts like heroku
117. one thing iâve been working on is an extension to play with ways end-users might interact
with web hooks
118. Hey, thereâs an
event hook here!
by detecting some markup in a page, it discovers hooks.
like say for new photos from contacts.
you want to do something when that happens, click it
119. Save
and write some code. hit save, it posts to AppJet (or wherever),
registers the handler (assuming a standard protocol), and done. all inline.
go back and change the code.
120. Real world examples
but these are all mockups and what-ifs...
there is a world of web hooks already evolving...
121. i started by exposing svn hooks as web hooks in devjavu
122. i talked about web hooks enough using pbwiki as an example,
their mysterious cto decided to implement them
124. went all out on hooked events. not sure if itâs made it to production, but really cool
125. âBuilding projects with web hooks in
mind lets me keep the core Lighthouse
source focused, while external
services live in their own libraries.â
--Rick Olson
the idea silently spread to rails guys.
rick olson used them in lighthouse
126. âWe implemented web hooks a while
ago and people have been building all
sorts of unexpected stuff on top of it.â
--Tobias LĂŒtke
tobias used them in shopify. iâm told heâs revamping their api to have more hooks. they were
one of the earliest adopters and recently had their 1 yr anniversary using web hooks
127. google code recently caught up other code hosts by providing a post commit hook.
it was very well done, as youâd expect from google and in particular shows that
authentication can be done with hmac signatures
128. github was one of the ïŹrst majorly popular site to use and promote web hooks.
129. theyâve been doing really well with their post-receive hook. users have used it to integrate
with mailing lists, chat, other project management tools, continuous integration, etc
130. they were so successful with the adhoc integration, they formalized it.
but in the best way! using their existing web hook infrastructure.
they just have modules running in a separate but local web service.
131. in fact, that lets them open source it. letting people fork, write new handlers, and push back.
this is probably going to be the standard model of service integration.
132. and a great example of services integrated with github, besides lighthouse, is runcoderun.
they run your regression tests for you. continuous integration in the sky. love it.
they sign you up automatically if you put their hook in github.
133. and of course i mentioned paypal. but i should mention, web hooks make so much sense
for paypal... theyâre not so much about pushing content, but INTEGRATING. thatâs what
web hooks are great for, even though they can be used for content push.
134. jott is another example of a web hook implementer that doesnât know it.
they parse voice over the phone and do stu with it, like post to twitter, etc
135. they do it with âLinksâ... which are just hooks. they post to a script that does something with
the parsed text. really cool for todos.
136. Jott reads back the
The message is Message is sent via User receives
User jotts to
response and sends
converted HTTP Post to a information back
a Jott Link
it via SMS SMTP
into text web page
hereâs their diagram. totally web hooks.
137. it become obvious making adaptor services would not only be useful in speciïŹc cases, but in
general, it makes sense to have services that will turn other protocols into inputs to the web
hook ecosystem. the ïŹrst was email... i built mailhook...
138. GAE community made one because GAE doesnât have a way to accept email (but will soon).
web hooks were the obvious solution.
139. rick olson has an open source non-hosted ruby version that will do xmpp.
he uses it for lighthouse.
140. but smtp2web is interesting because it was made because of the limitations of GAE...
141. in fact a lot of people made these kinds of âmicro webservicesâ to do simple things GAE
didnât do. it was the ïŹrst glimpse at small, focused services that are like the equivalent of
grep, cat, wordcount, etc in the command line piping ecosystem.
142. then thereâs martyn and andy. two guys in the uk that love web hooks.
they built this thing called spaghetti junction at a hackday. it involved into...
143. switchub. i REALLY love this. i knew this sort of thing would emerge, but i didnât think it would happen
until web hooks were more popular. kind of like the pastinbin code runner, they let you create hook
inputs with urls to put in apps that you can route to various output handlers: email, irc, etc
144. my example switchboard. this kind of feels like gnip, only more focused and more about web hooks.
so i like it lots.
145. opening handlers up like github. anybody can write handlers soon.
working with them a little to make it real awesome.
146. switchub would beneïŹt from builtin inputs from various other protocols like email, rss, etc...
but instead of having them builtin, they can work with other services to support those kinds
of inputs. for example, rssfwd could easily be modiïŹed to provide a web hook for rss
147. switchub is a lot like how i visualized a way for regular users to orchestrate web apps.
was inspired by reason: virtual rack mounts
148. ïŹip it around and wire them together however you like. totally cool.
153. tangent: this is a neat ïŹnd. was on reddit. andreessen proposing IMG tag.
people fought it, said it needed to be more generalized.
he just put it on mosaic and that was that
162. Extend
Integrate
Function
Content
but extending apps...
163. almost called this section Platforms.
platforms are really cool. we all love them.
i LOVE them, so fb platform was really cool.
asked a friend how it worked. he said âweb hooksâ
164. sure enough, this looks like web hooks to me. as long as itâs http, calling out... but then using the
results in their app? thats dierent...
165. in fact a few people have used web hooks for plugins. dabble is a great example.
166. they do online databased for people that use excel as a database.
168. âDabble plugins allow Dabble applications to create new, derived ïŹelds by calling out
to external HTTP-accessible applications. This solves the problem of safely enabling
extension of a centrally-located hosted application, in that, while youâre writing code
to extend and enhance the behavior of a Dabble application, your code never
actually runs inside Dabble.â
[General]
Name = Amazon Sales Rank
[Sales Rank by ISBN]
URL = http://chadfowler.com/dabble/amazon_sales_rank.cgi
Input = Text
Output = Number
only they have an extra layer for meta data. but thatâs a cool pattern.
169. âIf youâve used a UNIX-based operating system, youâre probably
familiar with the notion of pipes. The output of one program is
piped into the input of another, creating a ïŹlter chain. This is
conceptually the same as the way Dabbleâs plugin IO works.
Nice and simple.â
of course, they compare it to pipes. the simplicity. the natural ïŹt of it.
170. of course, i think they should have web hooks for all their standard CRUD events...
this way their database apps can integrate (like PayPal) with the rest of your workïŹow
171. in fact, all these âapp platformsâ like coghead and salesforce should have web hooks.
that would make them more useful, less siloâd o into just processing data in their world
172.
173.
174. IMiïŹed uses web hooks. sells the tech too: âallows anyone with basic web programming skills
to quickly and easily create ...â
176. General Systems
Theory
central tenet is value is not in the elements or parts of a system
177. General Systems
Theory
the real value is in the interactions, how they work together.
this creates the emergent phenomenon of a system, and deïŹnes its behavior
180. âa new programming paradigm
that takes the whole Internet
as its platformâ
this vision for a programming paradigm that IS the internet/web is very compelling... but weâre not
there yet.
184. âWhat is the equivalent of the pipe
in the age of the web?â
in 2000, just as rss 0.92 was being released, tim oreilly asks...
since we had nothing better, we assumed the answer was feeds.
185. this eventually gave us yahoo pipes. which just didnât seem to change the game like you would think...
perhaps feeds just arenât the answer. or maybe they are and the problem is using the pipes analogy.
itâs deïŹnitely a stretch...