Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Scaling tokopedia-past-present-future

27.739 Aufrufe

Veröffentlicht am

Leontinus Alpha Edison dalam webinar Startupbisnis x Codepolitan menjelaskan bagaimana Tokopedia.com melakukan scaling teknologinya

Veröffentlicht in: Business

Scaling tokopedia-past-present-future

  1. 1. Scaling Tokopedia Past, Present, Future
  2. 2. Once Upon a Time In Jakarta, Jan 2009
  3. 3. 1 Product Guy and 1 Half Engineer as co-founder
  4. 4. Never have experience to manage a high traffic website Don’t have business background AT ALL
  5. 5. Perl as back end Build our own perl framework Apache Mod Perl Oracle Express Edition
  6. 6. Hm… looks like we need a better front end designer
  7. 7. AwStats and A little bit Google Analytic
  8. 8. CBN apache server oracle server Network Topology Apps Topology Internet apache server oracle server http req http resp sql
  9. 9. 2 co-founder 1 real engineer 1 cust care
  10. 10. Hooray, WE LAUNCH!!
  11. 11. IDR 33 Mio of GMV in the first month
  12. 12. WE ARE SLOW!!!
  13. 13. * We didn’t have storage * pictures uploaded is stored on the same machine * Web page & static content is served by single apache * We didn’t use CDN * We didn’t even know what is CDN WHY??
  14. 14. Network Topology CBN apache app server oracle serverapache static server
  15. 15. Apps Topology Internet apache app server oracle server http req http resp sql Internet apache upload / statis server oracle server http upload http resp sql Internet apache upload / statis server http req http resp access web page upload pictures read static pictures css + js
  16. 16. We are back in business
  17. 17. BUT WE ARE SLOW AGAIN!!!
  18. 18. * Oracle express edition reach it’s limit * No Partition * No Replication * Poor indexing * Read/Write and Query on the same Master DB. WHY??
  19. 19. SO WE MIGRATE TO
  20. 20. Network Topology CBN apache app server PostgreSQL Masterapache static server PostgreSQL Slave
  21. 21. Apps Topology Internet apache app server PostgreSQL Master http req http resp sql insert sql update sql delete PostgreSQL Slave sql iquery WAL streaming Replication
  22. 22. We did it again!!!!
  23. 23. DAMN SEARCH IS SLOW!!!
  24. 24. * We have a lot of new products every second * We have to show search results in real time * But every second the sorting keep changing * PostgreSQL load is just too much!!! WHY??
  25. 25. And Many More……..
  26. 26. SEARCH IS EASY !!!!
  27. 27. Come on Man…. SLOW AGAIN??
  28. 28. * We were using apache + mod perl * Apache consume a lot of resource * Our code has a lot of memory leak WHY??
  29. 29. * We found out about NginX is very light and fast * We use nginx as load balancer * Replace apache modperl with nginx-perl * We have 1 nginx load balancer with several nginx-perl servers * For load balancing method, we mix round robin and clustering SOLUTION
  30. 30. siege -c100 -t5s -i -b -q 'http://www.tokopedia.com/ebenhaezer' siege: invalid option -- 'q' siege: invalid option -- 'q' ** SIEGE 2.72 ** Preparing 100 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 14788 hits Availability: 100.00 % Elapsed time: 4.59 secs Data transferred: 63.50 MB Response time: 0.03 secs Transaction rate: 3221.79 trans/sec Throughput: 13.83 MB/sec Concurrency: 87.52 Successful transactions: 7481 Failed transactions: 0 Longest transaction: 0.43 Shortest transaction: 0.00
  31. 31. Apps Topology PostgreSQL Master sql insert sql update sql delete PostgreSQL Slave sql iquery WAL streaming Replication Internet http req http resp NginX Load Balancer nginx-perl #1 nginx-perl #2 nginx-perl #3 nginx-perl #n proxy_pass SOLR Import SOLR query
  32. 32. Now what…. Storage??
  33. 33. * Hardware limitation * We used SATA HDD not SSD * Disk Utilities 100% * No back up, No Failover * Capacity is critical * Users keep uploading pictures WHY??
  34. 34. User
  35. 35. We also use CDN
  36. 36. AFTER ALL WE ARE STILL SLOW!!!
  37. 37. SOLUTION Internet nginx-perl #1 PostgreSQL Master http req http resp nginx-perl #2 nginx-perl #3 nginx-perl #n NginX Load Balancer proxy_pass PostgreSQL Slave replication MongoDB primary MongoDB secondary replication SOLR Redis query & update 3rd Party API such as Logistics, Banks, Payment Gw ETC Internet
  38. 38. We Start To Know About NginX, NoSQL In-Memory Storage GlusterFS Storage Scale out (not scale up) and many more….. Lesson Learn??
  39. 39. Thanks to our Awesome Engineers and many more…
  40. 40. We are back in business
  41. 41. BUT …………..
  42. 42. For the first time in our life we were doomed!!!
  43. 43. * One of our GlusterFS Server is broken. Image read/write is super slow. * We were using version of postgresql which has some bugs on indexing. WHY??
  44. 44. Another Awesome Engineers Mixed with International Team
  45. 45. Current State
  46. 46. New VP of Engineering
  47. 47. FUTURE
  48. 48. * Mobile First Company * Zero Downtime * Full to Cloud * Re-architech to SoA * Open API to Public * Deploy New Tech, such as replace perl with Go Lang * Advance Alert & Monitoring
  49. 49. * Redundancy and Failover * Multiple 3rd party * Datawarehouse such as Cubes, Pentaho etc * Machine Learning, Business Intelligence * Build things that can be share with others * Really pay attention on security * and many more……
  50. 50. What if the problems come from ISP? Unsolved Issues
  51. 51. * User cannot access Tokopedia * Pictures are not showing * css and js are not loaded * Sometime it just show a blank page * Some ISPs do Ads Injection * ALL WITHOUT REASONS FACTS
  52. 52. WHY?? WE DON’T KNOW BUT SOMETHING HAPPEN ON ISP SIDE
  53. 53. Works well * Using NginX Geo Module * All HTTPS since Q4 2014 * Try CDN Load balancing Don’t work at all * Talked to ISP * “Fight” in idEA What we’ve done
  54. 54. Don’t think “someone else will join and take care of this” — Mike Krieger of Instagram
  55. 55. Whether you think you can, or you think you can’t, you’re right — Henry Ford
  56. 56. THANK YOU ANY QUESTIONS?

×