Slides from the data plumbers episode about data engineering at OLX

1. Infrastructure for Image Models
at OLX
Alexey Grigorev
21.05.2019

2. About me
● Originally from Russia, live in Berlin
● Background: Databases
● Started as Software Engineer
● Masters in TU Berlin
● Data Science since 2013
● Book: Mastering Java for Data Science
● Kaggler in the past
● Now: Data Scientist at OLX
“ML Engineer”

3. OLX Group

4. OLX
● Tech hubs:
○ Poznan, Lisbon, Berlin
○ Buenos Aires, Delhi
● 30+ countries
● 200M MAU
● 10M sellers & 30M buyers
● 30M new listings per month
● Live listings:
○ OLX.PL (18M listings)
○ OLX.UA (12.5M listings)
○ OLX.IN (3M listings)

5. Apollo at OLX
Image uploads per day
● eu-west-1: up to 6.5 m
○ PL: 3 mln
○ UA: 1 mln
● ap-southeast-1: 2.2 m
○ IN: 0.7 mln
● eu-central-1: 1.6 m
● us-west-1: 55k
● total: ~10 m per day

6. Apollo X - the metadata service
● Data about images:
○ Category: what is on the image
○ Image quality: is the image good
● How to extract metadata?
● With machine learning!

7. Training
s3
SageMaker
we
Model

8. Serving
s3
event
listener
Apollo X
Category
Quality
tf-serving
mms
user
redis mysql
ObjectCreated:Put
Apollo bucket
Batch
Batch
AWS + Kubernetes

9. Sagemaker vs Self-hosted
Sagemaker
● Sagemaker is great for model training and testing in prod
● But it’s expensive for serving millions of requests
● Hard to monitor
● Hard to make Ops happy
Self-hosted (Kubernetes)
● Cheaper
● Fits existing infra
● Metrics, Logs, Alerts
● SREs know how to deal with issues

10. Image quality

11. Cover image

12. Forbidden items

13. Duplicates
● Duplicates is a problem
● 30% of images are duplicates
● 10% of ads are duplicates

14. Image hashes
● MD5 - cryptographic hash
● Perceptive hashes:
○ DHash, PHash, WHash (imagehash library)
● Take image
● Change pixel, resize, re-compress
● Different MD5, same phash

15. s3
ObjectCreate
ES
hashes
Image index
ObjectDelete
IngestorIngestor
Hash calculation
Image Index

16. Image Index
● Lambdas are super easy to scale
Images per hour
114 prs

17. Contact information
● http://alexeygrigorev.com & contact@alexeygrigorev.com
● https://github.com/alexeygrigorev
● https://www.linkedin.com/in/agrigorev

18. OLX is hiring
● https://www.olxgroup.com/search/engineering/all-locations/all-brands