18. 18
coming of age
During the US presidential election, 2012
Averaged 9,965 TPS from 8:11pm - 9:11pm PT
Peak of 15,107 TPS at 8:20pm PT
“In the past, we’ve generally experienced short-lived roars related
to the clock striking midnight on New Year’s Eve (6,939 TPS), the
end of a soccer game (7,196 TPS), or
Beyonce’s pregnancy announcement (8,868 TPS). Those spikes
tended to last seconds, maybe minutes at most. Now, rather than
brief spikes, we are seeing sustained peaks for hours”
- Mazen Rawashdeh, VP Infrastructure Operations Engineering
How? Migrating from Ruby to Scala and Java on the JVM
Java remains amazingly strong and vibrant. Resurgence underpinned by Big Data and microservices. Java and Javascript as the key skills package.
Twitter
Started as monolithic rails app. “Ruby will scale”
Discovering the JVM – hipster Scala
Enlightenment – pragmatic use of Java
Open source adoption and contribution
Summer 2010, Russian President Dmitry Medvedev – twitter had to “break twitter” and hack a dedicated server to meet demand for his first tweet
Facebook - notable “PHP shop.”
2008 needed a high scale, read intensive columnar data architecture for message search – created Cassandra. Wrote it in…Java
November 2010 facebook outgrows Cassandra, adopts Hadoop, written in… Java.
Cassandra adopted by Twitter, Instagram, Reddit, Netflix, Webex
Also talk to yahoo and why doug cutting chose java.
Etsy – another “PHP shop”. Begins life as a stored procedures monolith. Massive rewrite kicked off when Chad Dickerson joins in 2009. Move to devops, refactored code.
Adopts Hadoop in 2009/2010 before it was even a top level Apache project. https://codeascraft.com/2010/02/24/analyzing-etsys-data-with-hadoop-and-cascading/
2014, Cascading etc, Etsy datawarehousing team with extensive JRuby skills https://codeascraft.com/2010/02/24/analyzing-etsys-data-with-hadoop-and-cascading/
“When anyone asks what programming language to use, it is either PHP or Java because then anybody at the company can contribute.” John Allspaw SVP technical ops. 2015
http://www.nextplatform.com/2015/04/07/etsy-shows-how-to-be-just-crafty-enough-with-platforms/
Netflix. All about the culture. HR is everything. Amazing Relocation package, only pay above market rate for devops. Steeped in open source. Major contributors.
Started life as Java monolith running in tomcat
Code shipped every 2 weeks. All of it.
Ops was a separate group
Decomposition
smaller code bases, smaller teams, all with ops
from a few teams checking code into a large monolithic application running on tens of servers to having tens of engineering teams developing hundreds of component services that run on thousands of servers.
Disposability, the transition to Resilience with the expectation of breaking things. Chaos monkey, simian army.
Current definitions are hopelessly reductive. Should *not* specify implementation details (eg container-based)
CAP
12 factors
CTCF
Container packaged: Running in application containers as a unit of application deployment and as a mechanism to achieve high levels of resource isolation in order to improve the overall developer experience, foster code reuse and simplify operations.
Dynamically managed: Actively scheduled and managed by a central orchestrating process to radically improve machine efficiency, while reducing the cost associated with maintenance and operations.
Micro-services oriented: Loosely coupled with dependencies explicitly described through service endpoints with the goal of significantly increasing the overall agility and maintainability of applications
Paul Fremantle 2010 - Distributed / dynamically wired. Elastic. Multi-tenant. Self-service. Granularly metered and billed. Incrementally deployed and tested.
My definition- must be deployable on public cloud/standard infrastructure, expect effectively “infinite” resources, stateless/scale out, continuous deployment, microservices, DevOps, engineers for disposability.
Conways Law "Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design whose structure is a copy of the organization's communication structure."