10. AND HERE’S WHAT THEY FOUND OUT:
“Friends don’t let friends use RoR”
11. THINGS INSTAGRAM LOVES ABOUT PYTHON
Easy to become productive
Practicality
Easy to grow
engineering team
Popular language
12. THINGS INSTAGRAM LOVES ABOUT DJANGO
Maturity of the
language and
Django framework
Use Django user
model for
supporting 3B+
registered users
13. WE HAVE TAKEN OUR PYTHON+DJANGO STACK QUITE FAR
1 Sharded database support
2
Run our stack across multiple geographically distributed data centers3
Disable garbage-collection to improve memory utilization
16. PYTHON IS SIMPLE AND CLEAN, AND FAVORS PRAGMATISM.
1 Scope the problem, AKA, do simple things first
2 Use proven technology
3 User first: focus on adding value to user facing features
17. Subtitle
AT INSTAGRAM, OUR BOTTLENECK IS
DEVELOPMENT VELOCITY,
NOT PURE CODE EXECUTION
BUT PYTHON IS STILL SLOW, RIGHT?
18.
19. SCALING PYTHON TO SUPPORT USER AND FEATURE GROWTH
20
40
60
80
00
0 2 4 6 8 10 12 14 16 18 20 22 24Server growthUser growth
20. PYTHON EFFICIENCY STRATEGY
1 Build extensive tools to profile and understand perf bottleneck
2 Moving stable, critical components to C/C++, e.g., memcached access
3
Async? New python runtime?4
Cythonization
35. Create separate branch?
MIGRATION OPTIONS
• Branch sync overhead, error prone
• Merging back will be a risk
• Lose the opportunity to educate
36. MIGRATION OPTIONS
One endpoint at a time?Create separate branch?
• Common modules across end points
• Context switch for developers
• Overhead of managing separate pools
37. MIGRATION OPTIONS
One endpoint at a time?Create separate branch? Micro services?
• Massive code restructuring
• Higher latency
• Deployment complexity
57. from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
CHALLENGE: UNICODE/STR/BYTES
58. from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
CHALLENGE: UNICODE/STR/BYTES
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
59. CHALLENGE: UNICODE/STR/BYTES
mymac = hmac.new(‘abc’)
TypeError: key: expected bytes or bytearray, but got 'str'
value = ‘abc’
if isinstance(value, six.text_type):
value = value.encode(encoding=‘utf-8’)
mymac = hmac.new(value)
Error
Fix
68. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR
69. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR
70. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR
71. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = list(map(BuildProcess, CYTHON_SOURCES))
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR
80. Video View Notification
Save Draft
Comment Filtering
Story Viewer Ranking
First Story Notification
Self-harm Prevention
Live
81. MOTIVATION: TYPE HINTS
Type hints - 2% done
def compose_from_max_id(max_id: Optional[str]) -> Optional[str]:
MyPy and typeshed contribution
Tooling - collect data and suggest type hints
