Lookout! Growing Pains was originally presented at Lookout's Scaling for Mobile event on July 25, 2013. Kyle Barton is a Senior Software Engineer at Lookout, Inc. Kyle's talk focused on the growing pains that Lookout has faced as we've scaled and engaged more users. Lookout has grown immensely in the last year. We've doubled the size of the company—added more than 80 engineers to the team, support 45+ million users, have over 1000 machines in production, see over 125,000 QPS and more than 2.6 billion requests/month. Our analysts use Hadoop, Hive, and MySQL to interactively manipulate multibillion row tables. With that, there are bound to be some growing pains and lessons learned.

1. LOOKOUT!
GROWING PAINS
KYLE BARTON, LOOKOUT, INC.

2. THEN AND NOW
• Getting a solid product off of the ground
• Dealing with a growing user base
• Where we are today

3. GETTING OFF THE GROUND
MVP

4. • Try to reproduce locally
with a test
• Production debugging
• Eye balling change sets
• Synclogs
HOW WE DEALT

5. WE STARTED GROWING
• More Users
• More Traffic
• More Developers
• More Problems

6. • Ran out of integer space for
primary keys
• Load spikes
• Clients checked in at the
same time
• No way to tell clients to
back off
PROBLEMS

7. • More code going out in
each deploy
• Difficult to diagnose
issues after deploy
• Pretty much guaranteed
rollbacks on deploy
PROBLEMS CONT...

8. • Implemented “load
shedding”
• Under high load
respond to client with
“come back later”
message
• Automate with ganglia
checks
• Randomized client check in
• Picture Backup disabled by
default
LOAD SPIKES

9. INTEGER OVERFLOW
• Created new tables
• Big int primary and foreign keys
• Application code to use new and old
tables
• New data goes to new table
• Old data moved on login
• Background Process to update untouched
records

10. GIANT DEPLOYS
• Feature rollouts
• Slowly enable a new features
• Configuration
• Ability to turn off new code without
rolling back

11. WHERE WE ARE TODAY

12. PROBLEMS
• Syncml
• One endpoint to service all types of
requests
• Bloated request and responses
• Team Structure
• Communication
• Too many meetings

13. • Versioned RESTful APIs on
existing architecture
• Eventually end of life old
clients that haven’t
upgraded
DEPRECATE SYNCML

14. • Persisted chat
• Smaller teams
• Better documentation
TEAM STRUCTURE AND COMMUNICATION

15. Keep in touch with
@lookout
/mylookout
blog.lookout.com
contact@lookout.com
http://bit.ly/scaling-for-mobile