REST in Action: the Live Coverage Platform at the New York Times

1. REST in Action: The Live
Coverage Platform at the New
York Times
A Day of REST
London, UK - January 28, 2016

2. Scott Taylor
• Core Developer, WordPress
Release Lead for 4.4
• Sr. Software Engineer, The
New York Times
• I love my Wife, Music, NYC,
and Mexican food

3. My Beautiful Wife
• Warrior for Animal Rights at
@NYCLASS
• @alliefeldman212 on Twitter
• Mother to our cat…

4. Willa

5. WordPress 4.4 “Clifford”
• REST API (Phase 1)
• Term Meta
• Responsive Images
• WordPress as oEmbed Provider
• Tons of under the hood stuff (2000+ commits, 400+
contributors)

6. 2013 - Present
Me at the New York Times

7. WordPress at the NYT now
• The Live Coverage platform
• some forthcoming
International projects
• some “legacy” blogs
• Lens - photography blog
• First Draft
• NYT Co.
• Women of the World
• Times Journeys
• some internal corporate sites

8. “Blogs Team” at The NYT
• NYT used WordPress very early
• NYT was an early investor in Automattic
• Multisite: ~80 blogs at the height of blog mania -
(the 00s were the glory days)
• Many blogs used “Live Blogging”

9. When I arrived:
Legacy Blogs Codebase
• Separate from the rest of the NYT’s PHP codebase
• Global NYTimes CSS and JS, CSS for all Blogs,
custom CSS per-blog
• A universe that assumed jQuery AND Prototype
were loaded on every page in global scope
• Challenging amounts of what could generously be
called “technical debt”

10. At the NYT
• No WordPress Comments: There is an entire team that deals
with these for the site globally, in a different system called CRNR
• No Media: There is another CMS at the Times, Scoop, which
stores the images, videos, slideshows, etc
• WordPress native post-locking: This only landed in WordPress
core in version 3.6 (we have yet to reconcile the differences)
• There is layer for Bylines which is separate from Users: Our
users are employees authenticated via LDAP, most post authors
don’t actually enter the content themselves

11. NYT5: The New
Frontier
My arrival at the New York Times coincided with
the NYT5 project, already in progress

15. NYT5 Dealbreakers
• We can’t just point at WordPress on every request
and have our code figure out routing. Routing
happens in Apache in NYT5 - most requests get
piped to app.php
• Because PHP Namespaces are used, WP has to
load early and outside of them (global scope)
• On the frontend, WP cannot exit prematurely before
hitting the framework, which returns the response
to the server via SymfonyHttpFoundation

16. Apache
NYT5: app.php
Route to NYT5 Blogs app
- Load initial files
- Bootstrap WP
- Capture WP content
- WP complete
- Initialize app

17. Blogs in Transition

18. Overall: Bad News for Blogs
• Blogs were duplicating Section Fronts, Columns:
Mark Bittman has column in the paper.
The column also exists on the web as an article.
He contributes to the Diner’s Journal blog.
There is a section front for dining.
He also has his own NYTimes blog. Why?
• Blogs and WordPress were combined in everyone’s
mind. So whenever WordPress was mentioned as a
solution for anything, the response was: aren’t blogs
going away? #dark

19. But we still had…
First Draft
Lens
Live Coverage

22. 2014 Midterm Elections
required new Live Blogging
tools

23. 2008: Live Blogs at the Times
• A Blog would create a post and check “Start Live
Blogging”
• the updates related to the post were stored in custom
tables in the database
• the APIs for interacting with these tables duplicated tons
of WordPress functionality
• Custom Post Types didn’t exist until WordPress 3.0 (June
2010) - the NYT code was never rewritten to leverage
them (would have required porting the content as well)

24. Live (actual) Blogs:
Dashboards/Dashblogs
• A Live Blog would be its own blog in the network, its own
set of tables
• A special dashboard theme that had hooks to add
custom JS/CSS for each individual blog, without baking
them into the theme
• Making an entirely new site in the network for a 4-hour
event is overkill
• For every 10 or so new blogs that are added, you are
adding 100 new database tables - gross!

25. What if…
• Instead of custom tables and
dupe’d API code, new object
types: events and updates!
• To create a new “Live Blog”: create
an event, then go to a Backbone-
powered screen to add updates
• If WP isn’t desired for the front end,
it could be the backend for
anything that wants a JSON feed
for live event data
• Using custom post types, building
a Live Event UI that looks like the
NYT5 theme would be nominal

27. • Built an admin interface with Backbone to quickly
produce content - which in turn could be read from
JSON feeds
• When saving, the updates post into a service we
have called Invisible City (wraps Redis/Pusher)
• Our first real foray into using the REST API
• Our plan was just to be an admin to produce data
via self-service URLs
What we did

28. Live Events, the new Live Blogs:
Complete Rewrite of 2008 code
• nytimes.com/live/{event} and nytimes.com/live/{event}/
{update}
• Brand new admin interface: Backbone app that uses the
REST API. Constantly updated filterable stream -
Backbone collections that re-fetch on Heartbeat tick
• Custom REST endpoints that handle processes that need
to happen on save
• Front end served by WordPress for SEO, but data is
received by web socket from Invisible City and rendered
via React

31. Responsive
on Mobile

32. An “Interactive Promo”
on an article page

33. 2015: “Blogs” team
becomes “WordPress”
team, joins the Interactive
News team

34. • Would rather use Docker instead of Vagrant
• PSR-0 is now PSR-4
• Grunt is now eschewed in favor of Gulp
• RequireJS is ok, but I’d rather use Browserify
• PHP is cool, but why don’t we use Node and
React?

35. What is a Live
Coverage platform?

36. nytimes.com/live/{event}
Request is served by WordPress,
PHP generates markup
React wraps the "posts" area
JS listens to Web Socket
Updates are added on the backend (OR via SLACK!)
React updates the content

37. the elections code
became useful for
Breaking News

38. WordPress + REST

39. Register a route:
register_rest_route(
‘nyt/v1’,
‘/live-events/(?P<post_id>d+)’,
[
‘callback’ => [ $this, ‘liveEventRoute’ ],
‘methods’ => WP_Rest_Server::READABLE
]
);

40. Handle the route:
public function liveEventRoute( $request ) {
$post_id = $request['post_id'];
$response = new WP_REST_Response( [
'results' => . . .,
] );
return $response;
}

41. Some REST API
gotchas…

42. Most plugins only handle POST
• WP-API and Backbone speak REST
• REST will send you requests via
GET, PUT, DELETE, POST and friends

43. $hook = add_menu_page( ... );
add_action( "load-$hook", 'callback' );
function old_custom_load() {
if ( 'POST' !== $_SERVER['REQUEST_METHOD'] ) {
return;
}
...
}
function new_custom_load() {
if ( 'GET' === $_SERVER['REQUEST_METHOD'] ) {
return;
}
...
}

44. WordPress becomes
a web service
• Monolithic mindset needs to transition into how to make
it into a bare metal service provider
• The serving of requests should be loosely coupled from
objects like WP_Query
• WordPress needs to become supportive of concurrency

45. Custom REST Endpoints for GET
• We do not hit these endpoints on the front-end
• We have a storage mount that is fronted via Varnish
and Akamai
• JSON feeds can show up on the homepage of the
NYT to dynamically render “promos” - these have
to massively scale

46. HTTP is time-consuming
• It is easy to lose track of how many things are
happening on the 'save_post' hook
• Admin needs to be fast
• The front end is typically cached, but page generation
shouldn’t be bogged down by HTTP requests
• Anything which is time-consuming should be
offloaded to a separate “process” or request who
response you don’t need to handle

47. wp_remote_post( $url, wp_parse_args( [
'timeout' => 0.01,
'blocking' => false
], $args ) );
Fire and Forget*
* Stolen from Mark Jaquith’s nginx cache invalidation technique:
wp_remote_get( $url, [
'timeout' => 0.01,
'blocking' => false,
'headers' => [ 'X-Nginx-Cache-Purge' => '1' ]
] );

48. Custom REST Endpoints for POST
• Use fire-and-forget technique on 'save_post',
instead of waiting for responses inline. You can still
log/handle/re-try responses in the separate request.
• Most things that happen on 'save_post' only
need to know $post_id for context, the endpoint
handler can call get_post() from there

49. Trigger the process:
NYTAdminREST::async_request(
'/varnish-invalidation',
[
'urls' => $urls
]
);

50. Questions?