Recommended
Recommended
More Related Content
Viewers also liked
Similar to Scaling Dubsmash's backend from 0 to 100+ million users
Similar to Scaling Dubsmash's backend from 0 to 100+ million users (20)
Recently uploaded
Recently uploaded (20)
Scaling Dubsmash's backend from 0 to 100+ million users
- 1. Scaling Dubsmash's backend from 0 to 100+ million users PYCON.DE Munich - Daniel Taschik – 10/29/2016
- 2. We hit a nerve. >100M Users 192 Countries 1.5B Videos
- 4. The Start
- 5. The Start Backend • Django-powered BE for content management • web-based Dubloader to add sounds • deployed on Heroku Content Delivery • sound files in S3 • meta information in JSON file in S3 • files served via Cloudfront CDN Metrics • Dubloader with < 100 req/min • >500 TB! of traffic in January 2015
- 6. Dubsmash Service Landscape Backend Router S3 sound storage Cloudfront CDN
- 8. New Features: Registration & Search User registration • API based on REST framework • Django user model • store user’s most like sounds • push notifications for new content Server-side Sound search • new Django-based service • search via ElasticSearch using Haystack • Celery-based indexing on RQ Metrics • 100.000 registrations within first 24h • >20.000 requests per minute on search service
- 9. caching Cloudfront CDN S3 sound storage Dubsmash Service Landscape Searc h Router main API
- 10. DubTalk Social Graph Service • friend relations on platform • Django • TitanDB on Cassandra • later DynamoDB DubTalk Service • group & video management • Django Service Communication • Async via Celery on RabbitMQ • Sync via internal HTTPS API Metrics • > 50.000 requests per min on both • > 150.000.000 videos stored
- 11. NoSQL Cloudfront CDN caching S3 sound storage Dubsmash Service Landscape GraphDubTalk Router Monolith relational DB
- 12. Large Scale Problems favorited sounds outgrew our PostgreSQL • > 1.000.000.000 favorited sounds • simple data model & access pattern • Premium-7 120GB RAM, 1TB disk instance dtaschik@unic0rn:~/dubsmash$ heroku pg:table-size -a dubsmash name | size -----------------------------------+------------ users_favs | 158 GB dtaschik@unic0rn:~/dubsmash $ heroku pg:index-size -a dubsmash name | size -----------------------------------+------------ users_favs_username_key | 132 GB ID username sound_id 1 daniel3 Dzdcjc 2 sarah 3jGYzH Let’s make it a new service!
- 13. Cloudfront CDNS3 sound storage Dubsmash Service Landscape Auth Graph DubTalk Favs Router Monolith relational DB caching NoSQL many more
- 14. Our Goal Interested? Come and join! 😜 Building the largest mobile video communication platform!
- 15. Questions? daniel@dubsmash.com | daniel3 | @dtaschik
- 16. Let’s say it with video! Thank you! daniel@dubsmash.com | daniel3 | @dtaschik
Editor's Notes
- Dubsmash is a product with global scale. 100m users, in 192 countries created over 1,5bn videos in total. most famous ones are jimmy fallon, Huge Jackman, Arnold Schwarzenegger or Jennifer Lopez and recently emma dickson?
- connect, create communicate withfamily and friends but started as a very simple, 3-screen protoypical creational tool.
- the cave – building Dubsmash out of a Berlin souterrain 50 sqm office
- early 2015
- started mid 2015
- mid 2016