1. Cloud-Based Architecture and Big Data
Philip Balinov,
DevOps Engineer

2. SOME DEFINITIONS
►
def. CLOUD COMPUTING
l
- distributed computing over a network
l
- the ability to run a program or application on many connected computers at
the same time.
►
def. BIG DATA
l
data sets so large and complex that it becomes difficult to process using
traditional data processing applications

3. Retail
Finance
E-Commerce
Telecommunication
B2B
Publishing/Media
Government & NGO
Automotive
Travel
KOMFO'S CLIENTS

4. KOMFO PLATFORM WORKFLOW OVERVIEW

5. EXTERNAL PROVIDERS

6. MODERN COMPANY STRUCTURE
►
Agile software development: Scrum, Kanban
►
Short product feature lifecycle
►
Continuous delivery
►
Self-organizing teams
►
Cross-functional teams

7. AGILE DEVELOPMENT, CONTD.
►
Scrum
►
Excellent for product development
►
Short, iterative cycles (sprints), commit to deadlines
►
Self-organizing teams
►
Kanban
►
Developed at Toyota to optimize supply-chain
►
Good for ad-hoc tasks, e.g. support, bugfixing

8. AGILE DEVELOPMENT, CONTD.
►
Agile development requires agile operations
►
Underlying architecture must scale together with the product
►
Continuous integration
►
Quality assurance
►
Deployment
►
The solution: de-couple development and operations
►
DevOps

9. USE CASE
►
Fast and dependable communication with external providers
►
On-demand resource scaling
►
Flexibility
►
Storage, indexing & analysis of huge amounts of data
►
Dependability

12. CLOUD CONCEPTS
►
Three main service models
►
IaaS – EC2, Rackspace, Azure, HP, Oracle
►
PaaS – Google App Engine, Heroku
►
SaaS – Gmail, Wordpress.com, Salesforce, Office 365

13. CLOUD CONCEPTS, CONTD
►
Four deployment models
►
Public
►
Private
►
Community
►
Hybrid

14. CLOUD INTERNALS
SERVERS
STORAGE
EXTERNAL SERVICES
(CDN, NETWORKING, SaaS)
VMs VMs

15. PROS AND CONS
Dedicated Cloud
+ simplicity
+ performance
+ predictability
+ tried and tested
+ agility
+ ease of use
+ scalability
+ stability
- inflexible
- ineffective
- nightmare in case of disaster
- vendor lock-in
- blackboxed
- fluctuations
- helpless in case of disaster

16. CLOUD ARCHITECTURE
►
Application servers scale up and down based on load
►
Application software written for parallelism
►
Communication between nodes via messaging service
►
Write for eventual consistency

17. OK, so we have an (endlessly) scalable cloud app
now.
But didn't we just create a bottleneck elsewhere?

18. MIX & MATCH
►
Crunch numbers in the cloud
►
Application servers
►
Slow running tasks
►
Temporary services
►
Test servers

19. MIX & MATCH, CONTD.
►
Traditional servers for:
►
Incompatible apps (single-threaded, memory, disk intensive, specialized
hardware) do not work well in cloud environments
►
Database servers are best kept on dedicated machines in our experience

20. DATABASES, NOSQL
►
def. NoSQL
l
a mechanism for storage and retrieval of data that is modeled in means other
than the tabular relations used in relational databases.

21. DATABASES
• Use the best tool for the job depending on the task
• NoSQL Advantages
►
Social networks generate a lot of data
►
Complex interconnections, cyclical dependencies
►
Aggregations must be performed on both new and old data
►
Structure of foreign sources may change on short notice

22. DATABASES, NOSQL CONTD
►
Riak, Hadoop, MongoDB, Cassandra, Redis
►
In-memory dataset for faster operation
►
No predefined structure
►
Integrated sharding, load-balancing and failover
►
Versatility - can be used for anything from data storage to real-time messaging to search
indexes

23. DATABASES, LONG TERM STRATEGY
►
Data quickly becomes irrelevant
►
Archive it, but still be accessible
►
Online Data Warehouse solutions
►
Amazon Redshift
►
Keep Everything
►
Terabytes for pennies

24. WE ARE LOOKING FOR…
Senior Web Developer
Junior Web Developer
Junior QA
DevOps Engineer

25. Questions?