Increasing engineering velocity through architecture. This presentation was given at the Beyond Agile meetup group in Seattle on Sept 24, 2014.

1. Engineering Velocity
Building Great Software
Through Rapid Iteration

2. Matt Schemmel
Engineering
Resume Products

3. I help
people
get jobs.

5. Indeed
#1 Job Site Worldwide
More people find jobs on Indeed than anywhere
else
Over 140M monthly visitors
Over 3B monthly searches

6. (other)

7. Late '90s Quality Products

8. High Performance
Late '90s
Early '00s
Quality Products

9. High Performance
Late '90s
Early '00s
Quality Products
Late '00s High Scalability

10. High Performance
Late '90s
Early '00s
Quality Products
Late '00s High Scalability
Early '10s Rapid Iteration

11. Rapid Iteration:
Better Products
Faster
http://go.indeed.com/tdwi
http://go.indeed.com/ddpd
http://go.indeed.com/tdwi

12. Indeed Product Development Principles
Measure Everything
Test Everything (A/B Experiments)
Iterate Quickly

13. Lean Startup Cycle
Build
Learn Measure

14. Indeed Product
Development Cycle
Deliver
Learn Measure

15. Indeed Product
Development Cycle
Deliver
Quickly!
Learn Measure

16. Indeed Product
Development Cycle
Deliver
Learn Measure

17. Engineering Velocity
Deliver
Learn Measure

18. Engineering Velocity (n)
[ˌenjəˈni(ə)riNG vəˈläsətē]
Our ability to quickly deliver implementations of
ideas to job seekers and employers

19. Deliver Developer Productivity

20. Software Architecture
Processes and Tools
Risk Management
Deliver

21. Software Architecture
Processes and Tools
Risk Management
Deliver

27. Job Search
Web App

28. STOP
EVERY
THING

29. Job Search
Web App

30. Job Search
Web App

31. Service Oriented Architecture:
Technical Challenges

32. Challenge: High Performance
● Speed is a feature
● 100ms response time
clicks
time-to-render

33. Boxcar
High Performance SOA Framework
http://go.indeed.com/boxcar
Image used with permission of O Scale Trains Magazine (oscalemag.com), photo credit Don McFall of Old Line Graphics

34. Boxcar
Load Distribution without Middleware
http://go.indeed.com/boxcar

35. http://go.indeed.com/boxcar

36. Challenge: Compatibility over Time
● Any service can be updated at any time
● Any client can be updated at any time
● Any service or client can be rolled back
unexpectedly
Every service conversation must be forward
and backwards compatible

37. Boxcar: API Compatibility
Represent complex data types
Extremely compact and performant
Forwards and backwards compatible

38. Challenge: Deploy Independence
Forward/backward-compatible API is great.
But what about implementations?

39. Hard Scenarios
New API
Old server doesn't understand new client
Same API, Different Behavior
Old client doesn't understand new server
New client doesn't understand old server

40. Proctor
Efficient A/B Experiment Management
http://indeedeng.github.io/proctor/

41. Proctor
● Visitor segmentation
● Population filters
● Contextual allocation
http://indeedeng.github.io/proctor/

42. Proctor: Filters
"rule" : "loggedIn"
"rule" : "smartphone &&
'JP' == country"
"rule" : "ua.os == 'ios' &&
ua.version >= '6.0'"

43. Proctor: Visitor Allocation
10% 10%
Sign Up Activate
25% 25%
Sign Up Activate
US
UK

44. Proctor
Manage Experiments → Manage Behavior

45. Job Search
Web App
Boxcar!

47. Decoupling
Web Front-Ends

55. Each page has
different features
different data
different operational requirements
different owners
Each page should be its own deliverable

57. Job Search
Trends
Company Info
Forums
Salary
Indeed

58. Indeed Web Properties Goals
Provide a consistent user experience
One implementation of shared functionality
Completely decoupled applications

60. Option: Shared Libraries?
Reduced flexibility of technology decisions
Requires a release of all projects
Occasional inconsistent user experience

61. Option: Shared Resources?
On every app, include:
http://www.indeed.com/global/nav.js
http://www.indeed.com/global/nav.css
http://www.indeed.com/global/nav

62. Option: Common Resource
Reduced flexibility of technology decisions
Requires a release of all projects
Occasional inconsistent user experience
No shared services

63. Solution: Navshell
Application-level Rewriting HTTP Proxy

64. Request
Navshell
HTTP
Web App

66. <head>
<title>Johnson & Johnson Careers</title>
<link type="text/css" href="common.css">
<link type="text/css" href="comp.css">
</head>
<body>
<div id="page-header">
......
</div>
<div id="page-body">
<div id="page-frame">
<div id="comp-header">
......
</div>
</div>
</div>
</body>

67. <head>
<title>Johnson & Johnson Careers</title>
<link type="text/css" href="common.css">
<link type="text/css" href="comp.css">
</head>
<body>
<div id="page-header">
......
</div>
<div id="page-body">
<div id="page-frame">
<div id="comp-header">
......
</div>
</div>
</div>
</body>

68. <head>
<title>Johnson & Johnson Careers</title>
<link type="text/css" href="common.css">
<link type="text/css" href="comp.css">
</head>
<body>
<div id="page-header">
......
</div>
<div id="page-body">
<div id="page-frame">
<div id="comp-header">
......
</div>
</div>
</div>
</body>

69. REQUEST HEADERS
Cookie, User-Agent, Referer, etc
X-Indeed-Shell: 1
X-Indeed-Logged-In: [01]
X-Indeed-Email: chris@indeed.com
X-Indeed-Locale: en_US
X-Indeed-Geo: US
Navshell
Web App

70. Navshell
RESPONSE HEADERS
Set-Cookie
X-Indeed-Page-Title: Indeed Careers
X-Indeed-Include-Js-Footer: /js/app.js
X-Indeed-Include-Css: /style/main.css
Web App

71. Navshell
No duplicate code
Centralized navigation changes
Plain HTTP with strong contract
Decoupled front-ends

72. Requests
Navshell
Services Web Apps

73. Web-Based Delivery of
Mobile App Features

74. Mobile
Mobile Browser Mobile App

75. Mobile
Mobile Browser Mobile App

76. Shared Code Across Platforms
● One code base for mobile web, iOS, Android
● Simultaneous release of features across
platforms
● No waiting for App Store approval
● No waiting for job seekers to upgrade

77. Easy To Reuse Infrastructure
● A/B testing in native apps challenging
● Activity tracking challenging
● Performance monitoring challenging
We have battle-hardened solutions to all these.

78. Desktop
Browsers
Mobile
Browsers
Mobile
Apps
Job Search
App

79. Software Architecture
Processes and Tools
Risk Management
Deliver

80. Continuous Collaboration
Product
Eng QA

81. Continuous Deployment Toolchain
Integration git
Verification Jenkins
JUnit, TestNG, Selenium, JaCoCo, …
Deployment Rundeck

82. But what about the hard stuff?
● Configuration changes
● Resource re-allocation
● Database schema updates
● Internationalization

83. But what about the hard stuff?
● Configuration changes
● Resource re-allocation
● Database schema updates
● Internationalization
http://go.indeed.com/i18n

84. <Indeed map of the world>

85. tek arama. bütün işler.
one search. all jobs.
uma busca. todos os empregos.
één klik. alle vacatures.
una búsqueda. todos los empleos.
Ein Klick. Alle Jobs.
一站搜尋，工作齊全
Jedno vyhledávání. Všechna pracovní místa.
あらゆる仕事まとめて検索
una ricerca. tutti i lavori.

86. Before Internationalization
Implement Commit
Deploy to
QA
Deploy to
Production
Verify

87. Internationalization Process Overhead
Implement Commit
Translate
Deploy to
Verify Production
Identify
Deploy to
QA
Commit

88. +1 day

89. +1 day (20%)

90. +1 day (minimum)

91. <Indeed map of the world>

92. <Indeed map of the world>
<but with only US in orange>

93. I help
[English-speaking
Americans]
get jobs?

95. gettext utilities
GNU Translation Project Tool Set
http://www.gnu.org/software/gettext/

96. Template
en.po
Extract

97. gettext utilities (with some help)
✓ Many text sources: jsp, ftl, soy, js, …
✓ Dialects: en_US, en_CA, en_GB, …
✓ Variable substitution
✓ Complex plural forms

98. Machine Translation
English
Portuguese

99. Machine Translation
Low quality translations are better than no
translations
But we needed something better

100. Human Translation Service

101. Combined Translation Service
Extract Improve Release

102. Localization before Translation Automation
Implement Commit
Identify Translate
Commit
Deploy to
Verify Production
Deploy to
QA

103. Localization with Translation Automation
Implement Commit
Deploy to
Verify Production
Extract
Deploy to
QA
Improve

104. And if there just isn't enough time…
"rule" : "language != 'es'"
"rule" : "locale != 'en_GB'"
"rule" : "country != 'IT'"

105. I help
everyone
get jobs!

106. Software Architecture
Processes and Tools
Risk Management
Deliver

107. Risk
Severity of issue
Probability of issue
Number of users affected

109. Smaller Releases
Probability of issue
More releases → less functionality per release

110. Proctor Feature Toggles
Probability of issue
"rule" :
"endsWith(email,'@indeed.com')"

111. Manage Cross-Product Releases
Probability of ISsue
Deploy products then activate features

112. Decouple Features from Releases
# of users affected
Rollback features and releases independently

113. Incremental allocation changes
Number of users affected
1% → 5% → 25% → 50% → 100%

114. Embedded QA
Probability of issue
Severity of issue

115. Monitoring
# of users affected

116. Monitoring and Alerting
● Identify runtime issues
● Alert Ops when an issue arises

117. Healthcheck Framework
● Applications and services self-report
internal state & health
● Divert traffic away from unhealthy instances
● Disable features for failing dependencies

119. Measure Everything

120. Software Architecture
Processes and Tools
Risk Management
Deliver

121. Better Product
● Emphasize goals over features
● Fail faster, cheaper
● Incremental progress live in production

122. Better Life
● No deadline crunches
● There's always another release coming
● More transparent schedules = less stress

123. We built all of this
incrementally, to meet
our own challenges

124. Yours are different

125. Q&A
Visit
http://engineering.indeed.com/
and
http://go.indeed.com/jobs