Unlocked London - Technical Track

No 11
cavendish
square
11 july 2013
london
Thursday, July 11, 13

developer
VOCAB
Alex brandt
sr developer

MODULES
3

LIBRARIES
4

FRAMEWORKS
•The building blocks of an application - a set of common
software functions that take the work out writing all code
from scratch.
5

LANGUAGES
9

What is a data
structure?
•images
• url
• stars
• *title
• caption
• variants
•small
•medium
•big
•tags
•*_id
•rolestars
•*image_id
•*role_id
•roles
• *name
• privileges
• login
• description
• location
• child_roles
• _id
13

API tier/Application
STORAGE
APPLICATION
PRESENTATION
14

API tier/Application
STORAGE
APPLICATION
PRESENTATION
APPLICATION/API
14

Bindings
15

Bindings
16

Bindings
17

EVERYTHING
IS CODE
wayne walls
cloud evangelist
alex brandt
sr developer

Infrastructure
as code
19

Chef
Server
Chef
Workstation
Chef Client
knife
node
Chef Client
node
environment
22

•Nodes: Servers
•Environments: Groups of nodes (Dev/QA/Prod)
•Recipes: Describes the desired state of a node
•Cookbooks: Group of Recipes
•Roles: Group of Cookbooks (Web)
•Attribute: information about the current state of a node
23

Puppet
CA
Puppet
Master
Puppet
Agent
https
Puppet
Agent
Puppet
Agent
25

•Nodes: Servers
•Environments: Groups of
nodes (Dev/QA/Prod)
•Manifests: Describes the
desired state of a node
•Catalog: Group of Manifests
•Facts: information about the
current state of a node
26

Salt
Master
Salt
Minion
Salt
Minion
Salt
Minion
28

•Nodes: Minions
•Environments: Groups of nodes (Dev/QA/Prod)
•States: Describes the desired state of a node
•Grains: information about the current state of a node
29

Thinking like a developer
30

scrum Development
Product
Backlog
Sprint
Backlog
Sprint
Working increment
of the software
31
1. Design
5. QA 2. Build
4. Release 3. Configure

scrum Development
32
1. Design
5. QA 2. Build
4. Release 3. Configure

TYPES OF
DEVELOPMENT
33

TYPES OF
DEVELOPMENT
33
Test Driven Development

TYPES OF
DEVELOPMENT
33
Behavior Driven Development

TYPES OF
DEVELOPMENT
33
Behavior Driven Development Domain Driven Development

TYPES OF
DEVELOPMENT
33
Application Bliss
Domain Driven Development

Model View Controller (MVC)
UPDATES MANIPULATES
SEES USES
USER
MODEL
VIEW CONTROLLER
34

DEv tools

Gerrit
38

CODE
REVIEW
PROCESS

Our hero, the developer, submits code for review.
The code is reviewed in Gerrit by other developers, who
approve the change.
Jenkins runs automated tests against the code change.
Jenkins pushes the code to GitHub so others can build on the
change.
41

POLYGLOT
PERSISTENCE
wayne walls
cloud evangelist
technical track
alex brandt
sr developer

A database, in the most general
sense, is an organized collection of
data.
Database Defined
http://glenn-packer.net
43

ACID
•Atomicity: refers to a database’s ability to either fully process or fully roll back a
transaction
•Consistency: database should ensure that all data written therein follows all rules
and constraints specified in the database
•Isolation: transactions must be processed securely and independently, without
interfering with each other
•Durability: the database must ensure that all committed transactions are saved
permanently and cannot be accidentally erased, even in a database crash
44

•A relational database is essentially a group of tables
or, to use the technical name, entities.
•Each table is made up of rows (tuples) and columns
(attributes).
relational
databases
45

Some companies needed something more to accommodate their growing data
rdbms may not be
the best fit
46

Enter
Non-Relational database
management systems
47

INTRODUCTION TO
NOSQL
• NoSQL Database (1998)
• NoREL
• “Not only SQL”
• NoSQL (2009)
48

NoSQL is difficult to define but most NoSQL databases will
share common characteristics
1. Non-relational
2. Distributed
3. Open-source
4. Horizontally Scalable
5. Schema-less
NoSQL
49

An “aggregate” is a
collection of related
objects that we wish
to treat as a unit
orders
customers
order lines
credit cards
ID: 1001
line items:
customer: Ann
0321293533
0321601912
0131495054
2
1
1
$48 $48
$39
$51
$39
$51
payment details:
Card: Amex
CC #: 12345
expires: 04/2001
aggregate
50

There are several different
types of data stores that are
considered to be NoSQL
•Key/Value Databases
•Document Databases
•Column-Family Stores
•Graph Databases
51

Key/Value Database
•Primarily constructed of aggregates that have a Key or ID that’s
used to access the data
•The database cannot see structure within the aggregate
52

Key/Value Database
•Good Use Cases:
• Storing Session
Information
• User Profiles &
Preferences
• Shopping Cart Data
• Bad Use Cases:
• Data Relationships
•Multi-operation
Transactions
•Query by Data
53

•Like a Key-Value database, document databases primarily store
and access data using aggregates
•In contrast to the Key-Value database, document data bases can
see structure within the aggregate.
•You are able to retrieve the whole aggregate or specific parts of it
document Database
54

document Databases
•Good Use Cases:
• Event Logging
• Blogging Platforms
• Real-time Analytics
• Bad Use Cases:
• Complex Transactions
•Queries against Varying
Aggregate Structures
55

• Can be looked at as a two-level aggregate
structure
• The first key is often described as a row ID
picking up the aggregate of interest
• The second level values of the aggregates are
referred to as columns
• You are able to access a row as a whole or
operations will allow you to pick out a
particular column to get specific data
COLUMN FAMILY
56

COLUMN-FAMILY
datastores
•Good Use Cases:
• Event Logging
• Blogging Platforms
• Counters (page visits)
• Expiring Usage
• Bad Use Cases:
• ACID transactions
required
• Early prototype
applications
57

•Unique compared to most NOSQL types because it was not
developed with the intention of scaling horizontally into a cluster
•Data is stored as “nodes” that are connected by edges
•Edges are directional in nature and must be specified
•Queries need a starting point and them will traverse the edges to
gather the data requested
graph database
58

Amanda
KNOWS
KNOWS
KNOWS
Robert
Gerry
Lamar
KNOWS
App 1
BUILT
BUILT
KNOWS
App 2
USES
BUILT
Christie
KNOWS
Destiny
Jessica
KNOWS
App 3
BUILT
USES
USES
USES
KNOWS
graph database
59

graph databases
•Good Use Cases:
• Connected Data
• Location-based services
•Recommendation
engines
• Bad Use Cases:
• Ability to update
specific aggregates
60

Companies using
nosql
61
DATABASE COMPANIES

THE FUTURE IS POLYGLOT
62
Neo4J
database
Cassandra
database
Couch
database
MongoDB
database
RELATIONAL
DATABASE
Graph-structured
domain rules
Columnar data
access with
decentralization
Document
structures
Document structures
with offline
processing
(Actors) (Actors)
(Actors)(Actors)
Asynchronous message passing
MODULE 1
MODULE 2
MODULE 3
MODULE 4
IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg

63
POLYGLOT PATTERN: cache
ORDER
TAKING
RESTAURANT
MANAGEMENT
REDIS
CACHE
MYSQL
DATABASE
SECONDFIRST
CONSUMER
RESTAURANT
OWNER

64
POLYGLOT PATTERN:
write to sql and NOSQL
NEW
TWEET
INSERT
LPUSHX
LTRIM
MySQL
TWEETS
FOLLOWS
FOLLOWERS
Redis: Timeline
FOLLOWER 1
FOLLOWER 2
FOLLOWER 3
Avoid
expensive
MySQL joins

65
POLYGLOT PATTERN:
REPLICATE FROM
MYsql to NOSQL
QUERY
SERVICE
UPDATE
SERVICE
REDIS
MYSQL
DATABASE
SYSTEM
OF
RECORD
MATERIALIZED
VIEW
READER WRITER
replicate and
denormalize
query() update()

66
Atomic
Consistent
Isolated
Durable
ACID to base
Basically Available
Soft state
Eventually consistent

5 PILLARS OF
CLOUDINESS
wayne walls
cloud evangelist
alex brandt
sr developer
technical track

FIVE PILLARS
OF CLOUDINESS
1 2 3 4 5
Parallel Modular Horizontal Agile Secure
68

Parallel

Parallel - what
1M
Records
Answer
Node
70

Parallel - how
1M
Records
Answer
Node
1-250K
Node
251K-500K
Node
501K-750K
Node
751K-1M
71

Parallel - how
72
1M
Records
Answer
Node
1-250K
Node
251K-500K
Node
501K-750K
Node
751K-1M

Parallel - how
73

Parallel - how
DATA
CALCULATION
MAP
CALCULATION
REDUCE
DATA
CALC
D
C
R
D
D
C
R
D
D
C
R
D
SHUFFLE
DATA
73

Avoid Deadlock
74

Thread 1
Deadlock
75
Thread 2

File A
Thread 1
Deadlock
75
Thread 2

File A File B
Thread 1
Deadlock
75
Thread 2

File A File B
Thread 1
Deadlock
75
File B
Thread 2

File A File B
File A
Thread 1
Deadlock
75
File B
Thread 2

File A File B
File A
Thread 1
Deadlock
75
X
File B
Thread 2

File A File B
File A
Thread 1
Deadlock
75
X X
File B
Thread 2

Solution? Scheduling
Y
XA
B
A
B
Y
X
1
2
3
4
76

Parallel - why
1M
records
Answer
Node
1M
records
Answer
Node
1-250K
Node
251K-500K
Node
501K-750K
Node
750K-1M
77

Parallel - why
78

Parallel - why
79

ONE PERSON BUILDS A CAR
Modular - what
Example: Assembly Line
81

CHASSIS ENGINE BODY INTERIOR
Decoupled
Self-contained units
Data Communication
Modular - what
Example: Assembly Line
82

Modular - what
Modules within Modules
83
Nested Modularity

Modular - what
84
Nested Modularity

Modular - what
SEATS FLOOR
DASHBOARD
GAS
AND
BRAKE
PEDALS
84
Nested Modularity

Small units are more maintainable
Allows us to tolerate some partitions (but not all)
Allows us to be parallel!
modular - Why
85

Direct communication (1:1)
Topic communication (1:N)
Fanout communication (1:ALL)
modular - Why
86

•Decouple requests
•Message Queue
•Make small units of work
•Define tiers by function
modular - how
87

Potential for
Upstream Complexity
88

defect
Potential for
Upstream Complexity
88

defect defect
Potential for
Upstream Complexity
88

defect defect defect
Potential for
Upstream Complexity
88

defect defect defect defect
Potential for
Upstream Complexity
88

defect defect defect defect FAIL
Potential for
Upstream Complexity
88

defect defect defect defect FAIL
Mandelbugs
Potential for
Upstream Complexity
88

horizontal

2 GB
2 CORES
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
VERTICAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
SHARED NOTHING THROUGH STATELESSNESS
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
90
8 GB
4 CORES

2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
2 GB
2 CORES
VERTICAL
HORIZONTAL
Horizontal - What
2 GB
2 CORES
90
8 GB
4 CORES

Horizontal - why
91

Horizontal - why
92

ONE
SERVER
NEAR
CAPACITY
Horizontal - why
92

ONE
SERVER
NEAR
CAPACITY
ADD
ANOTHER
SERVER
Horizontal - why
92

ONE
SERVER
NEAR
CAPACITY
ADD
ANOTHER
SERVER
Horizontal - why
93

1234 MESSAGE QUEUE
94
Horizontal - HOW

1234MESSAGE QUEUE
95
Horizontal - HOW

1
2
3
4
96
MESSAGE QUEUE
Horizontal - HOW

1
2
3
4
96
56789101112 MESSAGE QUEUE
Horizontal - HOW

97
5678 9101112MESSAGE QUEUE
1
2
3
4
Horizontal - HOW

97
5678 9101112
ADD MORE SERVERS
BASED ON
MESSAGE QUEUE
MESSAGE QUEUE
1
2
3
4
Horizontal - HOW

MESSAGE QUEUE
98
5
6
7
8
9
10
11
12
13141516
ADD MORE SERVERS
BASED ON
MESSAGE QUEUE
1
2
3
4
Horizontal - HOW

99
5678 9101112
13141516
ADD MORE SERVERS
BASED ON
MESSAGE QUEUE
1234
MESSAGE QUEUE
Horizontal - HOW

100
5678 9101112
13141516
ADD MORE SERVERS
BASED ON
MESSAGE QUEUE
1234
MESSAGE QUEUE
Horizontal - HOW

101
5678 9101112
13141516
ADD MORE SERVERS
BASED ON
MESSAGE QUEUE
1234
MESSAGE QUEUE
Horizontal - HOW

101
5678 9101112
13141516
ADD MORE SERVERS
BASED ON
MESSAGE QUEUE
1234
MESSAGE QUEUE
Horizontal - HOW
DECREASE SERVERS
BASED ON
MESSAGE QUEUE

MESSAGE QUEUE
102
5678 9101112
13
14
15
16
ADD MORE SERVERS
BASED ON
MESSAGE QUEUE
1234
Horizontal - HOW
DECREASE SERVERS
BASED ON
MESSAGE QUEUE

Agile - WHAT
the power of moving
quickly and easily;
nimbleness
104

Features In Progress Blocked Tabled Testing Complete
Feature A
Feature B
Feature C
105
Agile - why Monday

Agile - why
Feature A
Feature B
Feature C
106
Tuesday

Agile - why
Feature A
Feature B
Feature C
Feature E
Feature D
106
Tuesday

Feature AFeature BFeature C
Feature E
Feature G
Agile - why
107
Wednesday

Feature AFeature BFeature C
Feature E
Feature D
Feature F
Feature G
Agile - why
107
Wednesday

Feature A
Feature B
Feature C
Feature E
Feature D
Feature F
Feature G
Agile - why
108
Friday

Feature A
Feature B
Feature C
Feature E
Feature D
Feature F
Feature G
Feature H
Feature I
Agile - why
108
Friday

Feature A
Feature B
Feature C
Feature EFeature D
Feature F
Feature G
Feature H
Feature I
Agile - why
109
Friday - WK 2

Agile - why
57%
14%
29%
Failed Successful Challenged
49%
42%
9%
Waterfall Agile
Source: The CHAOS Manifesto, The Standish Group, 2012
110

• CI/CD when
appropriate
• Quality is in the
eye of the
beholder
• Release fast,
release slow
Agile - How
111

IMAGE: blogs.bu.edu
Agile - How
• Increase quality
by reusing code
from partners
and libraries
112

Agile - How
113
Lint Testing
Functional Testing
Black Box
Integration Testing
Unit Testing
White Box
Smoke Testing
Acceptance Testing

ENSURES QUALITY
Agile - How
113
Lint Testing
Functional Testing
Black Box
Integration Testing
Unit Testing
White Box
Smoke Testing
Acceptance Testing

ENSURES CONFIDENCE
Agile - How
114
Lint Testing
Functional Testing
Black Box
Integration Testing
Unit Testing
White Box
Smoke Testing
Acceptance Testing

Security - what
116
•All about Trust
•Provider
•Customer
•User

117
SECURITY - HOW
•Security at every layer
PRESENTATION
APPLICATION
STORAGE

117
SECURITY - HOW
•Security at every layer
PRESENTATION
APPLICATION
STORAGE
INTERFACE
SECURITY
ACROSS ALL LAYERS

118
SECURITY - HOW
•Use industry standards
•SSL
•Encryption
•At rest
•In-transit

119
SECURITY - HOW
•Authentication
•User authentication
•Manage yourself
•Offload to 3rd party
•Host-based Authentication
•Security Groups
•Firewalls
•Configuration
Management Systems

120
SECURITY - HOW
•Hybrid Cloud
•Store different
types of data in the
right place
•Sensitive data in on-
premise cloud
•Non-sensitive on
public cloud

121
SECURITY - WHY
•Trustworthiness as an organization
•Cloud security is too important
to leave up to the end-user alone
•Ownership gaps that leads to
process and governance issues
•Compliance
•Logging and auditing
•Regulatory requirements

BREAK
3:00 - 3:15

APPLICATION
dissection
alex brandt
sr developer
technical track
wayne walls
cloud evangelist

Data Schema
•images
• url
• stars
• *title
• caption
• variants
•small
•medium
•big
•tags
•*_id
•rolestars
•*image_id
•*role_id
•roles
• *name
• privileges
• login
• description
• location
• child_roles
• _id
124

Requirements
•List stars
•List roles’ images
•CRUD
•Tag searching
•Who liked what
•Social media
•Image resizing
•Added by tag
•Hot tagged images
125

application decisions
API TIER
UT UR RS
WORKER
TIER
LB
AUTH
MQ DB
OBJECT
STORAGE
CDN
SOCIAL
MEDIA
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

API TIER
UT UR RS
WORKER
TIER
AUTH
MQ DB
OBJECT
STORAGE
CDN
SOCIAL
MEDIA
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

API TIER
UT UR RS
WORKER
TIER
MQ DB
OBJECT
STORAGE
CDN
SOCIAL
MEDIA
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

API TIER
UT UR RS
WORKER
TIER
DB
OBJECT
STORAGE
CDN
SOCIAL
MEDIA
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

API TIER
UT UR RS
WORKER
TIER
OBJECT
STORAGE
CDN
SOCIAL
MEDIA
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

API TIER
UT UR RS
WORKER
TIER
CDN
SOCIAL
MEDIA
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

API TIER
UT UR RS
WORKER
TIER
CDN
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

API TIER
UT UR RS
WORKER
TIER
126
INTERNET
•Load Balancer
•Infrastructure
•Authentication
•Message Queue
•Database
•Content Delivery
•Social Media

PARALLEL
API TIER
UT UR RS
INTERNET
WORKER
TIER
127

•API Tier
PARALLEL
API TIER
UT UR RS
INTERNET
WORKER
TIER
API TIER
127

•API Tier
PARALLEL
API TIER
UT UR RS
INTERNET
WORKER
TIER
API TIER
•Works in tandem
127

•API Tier
PARALLEL
API TIER
UT UR RS
INTERNET
WORKER
TIER
API TIER
•Works in tandem
•Doesn’t wait for workers
127

•API Tier
PARALLEL
API TIER
UT UR RS
INTERNET
•Update Tags, Update Relations,
Resize Workers operate in
tandem
WORKER
TIER UT UR RS
•Works in tandem
127

•API Tier
PARALLEL
API TIER
UT UR RS
INTERNET
•Update Tags, Update Relations,
Resize Workers operate in
tandem
•Non-blocking actions
WORKER
TIER UT UR RS
API TIER
•Works in tandem
127

MODULAR
API TIER
UT UR RS
WORKER
TIER
INTERNET
128

•Separate tiers by functionality
MODULAR
API TIER
UT UR RS
WORKER
TIER
API TIER
UT UR RS
INTERNET
128

MODULAR
API TIER
UT UR RS
•Use MQ to decouple API from
worker
WORKER
TIER
INTERNET
128

MODULAR
API TIER
UT UR RS
•Use MQ to decouple API from
worker
•Separate workers to simplify
units of work
WORKER
TIER UT UR RS
INTERNET
128

Horizontal
API TIER
UT UR RS
WORKER
TIER
INTERNET
129

•Restful API for shared nothing
scale
Horizontal
API TIER
UT UR RS
WORKER
TIER
API TIER
INTERNET
129

scale
Horizontal
API TIER
UT UR RS
•Workers share no state, simply
work on input
WORKER
TIER
INTERNET
129

scale
Horizontal
API TIER
UT UR RS
•Workers share no state, simply
work on input
•Scale data across nodes (even
DCs)
WORKER
TIER UT UR RS
API TIER
INTERNET
129

AGILE
API TIER
UT UR RS
WORKER
TIER
INTERNET
130

•Don’t re-invent ability or hosting
AGILE
API TIER
UT UR RS
WORKER
TIER
INTERNET
130

AGILE
API TIER
UT UR RS
•Split processes that are
individually testable
WORKER
TIER
API TIER
UT UR RS
INTERNET
130

AGILE
API TIER
UT UR RS
•CI/CD is achievable by
component
WORKER
TIER
INTERNET
130

AGILE
API TIER
UT UR RS
•CI/CD is achievable by
component
WORKER
TIER
API TIER
INTERNET
130

SECURE
API TIER
UT UR RS
WORKER
TIER
INTERNET
131

•Use HTTPs for secure data
transmission
SECURE
API TIER
UT UR RS
WORKER
TIER
API TIER
INTERNET
131

transmission
SECURE
API TIER
UT UR RS
•No host logins; managed by
Chef
WORKER
TIER
API TIER
UT UR RS
INTERNET
131

transmission
SECURE
API TIER
UT UR RS
•No host logins; managed by
Chef
•StormPath for user
authentication
WORKER
TIER
INTERNET
131

thank you!
@rackspaceUK
@Racker_beighton
@waynewalls
@alunduil
@jennbolestweets

Unlocked London - Technical Track

Recommended

Recommended

More Related Content

Similar to Unlocked London - Technical Track

Similar to Unlocked London - Technical Track (20)

More from Wayne Walls

More from Wayne Walls (8)

Recently uploaded

Recently uploaded (20)

Unlocked London - Technical Track