Speaker: Dmitri Chhtchourov

1. Dmitri Chtchourov,
MANTL Data Platform, Microservices and BigData Services
Innovation Architect, CIS CTO Group

2. Agenda
Problem & Opportunity
What do we want to do?
What is in it for us?
How does it work?
What have we done so far?
Anatomy of a Service
Reference Architectures and real use cases
Putting it all together

3. Problem & Opportunity
Rapid innovation in computing and application development services
No single service is optimal for all solutions
Customers want to run multiple services in a single cluster and
run multiple clusters in Intercloud environment
...to maximize utilization
...to share data between services
…Complex/BigData and Microservices together

4. Technologies matrix*
Service Product
Cloud/Virtualization CIS/AWS/Metacloud/UCS…
Provisioning Open Stack/Terraform
Automation Ansible
Clustering & Resource
Management Mesos, Marathon, Docker
Load Balancing Avi Networks
ETL & Data Shaping StreamSets
Log Data Gathering Logstash
Metrics Gathering CollectD, Avi Networks
Messaging Kafka, Solace
Data Storing (Batch) HDFS
Data Storing (OLTP/Real-time) Cassandra
Data Storing (Indexing) Elastic search
Data Processing Apache Spark
Visualization Zoomdata
*Subset example

5. Cloud
Management
Data Collect
Data
Storage
Data
Processing
Visualisation
Technologies stack

6. Datacenter and solution today
VM7
or
BM7
VM8
or
BM8
VM4
or
BM4
VM5
or
BM5
VM6
or
BM6
VM1
or
BM1
Visualization Service
Data Ingestion Service
Analytics Service
• Configuration and management
of 3 separate clusters
• Resources stay idle if service
is not active
• Need to move data between
clusters for each service
VM2
or
BM2
VM3
or
BM3
VM1
or
BM1
VM2
or
BM2
VM3
or
BM3

7. What do we want to do?
Data Ingestion Service
Analytics Service
Visualization Service
….to maximize utilization
...to share data between services
Shared cluster
Multiple clusters

8. Shared Cluster
CIS/AWS/Metapod/UCS…
VM1
or
BM1
VM2
or
BM2
VM3
or
BM3
VM4
or
BM4
VM5
or
BM5

9. What is in it for us?
Maximize utilization
Deliver more services with smaller footprint
Shared clusters for all services
Easier deployment and management with unified service platform
Shared data between services
Faster and more competitive services and solutions
Combine paradigms for flexibility and functionality
Run complex services and microservices in the single environment

10. How does this work?
Mesos Slave
Spark Task Executor Mesos Executor
Mesos Slave
Docker Executor Docker Executor
Mesos Master
Task #1 Task #2 ./python XYZ java -jar XYZ.jar ./xyz
Mesos Master Mesos Master
Spark Service Scheduler Marathon Service Scheduler
Zookeeper quorum

11. How does this work?
Mesos provides fine grained resource isolation
Mesos Slave Process
Spark Task Executor Mesos Executor
Task #1 Task #2 ./python XYZ
ComputeNode
Executor
Container
(cgroups)

12. How does this work?
Mesos provides scalability
Mesos Slave Process
Spark Task Executor
Task #1 Task #2 ./ruby XYZ
ComputeNode
Python executor finished,
more available resources
more Spark
Container
(cgroups)
Task #3 Task #4

13. How does this work?
Mesos has no single point of failure
Mesos MasterMesos Master
Mesos Master
VM1
or
BM1
VM2
or
BM2
VM3
or
BM3
VM4
or
BM4
VM5
or
BM5
Services keep running if VM fails!

14. How does this work?
Master node can failover
Mesos MasterMesos Master
Mesos Master
VM1
or
BM1
VM2
or
BM2
VM3
or
BM3
VM4
or
BM4
VM5
or
BM5
Services keep running if Mesos Master fails!

15. How does this work?
Slave process can failover
Tasks keep running if Mesos Slave Process fails!
Mesos Slave Process
Spark Task Executor
Task #1 Task #2 ./ruby XYZ
ComputeNode
Task #3 Task #4

16. How does this work?
Can deploy in many environments
Get orchestrated by Openstack, Ansible (scripts), Cloudbreak
True Hybrid Cloud deployment: CIS, AWS, UCS, vSphere,
other
CIS/
CIS/AWS/Metpod/vSphere/UCS…
Terraform
REST API
REST API
Scripted provisioning
Direct provisioning
Policy, Auto-scaling
VM1
or
BM1
VM2
or
BM2
VM3
or
BM3
VM4
or
BM4
VM5
or
BM5

17. How does this work?
Microservices managed and scaled separately
Microservices managed by Mesos in a single platform
Microservices architecture for Mesos frameworks and other
components
CIS/AWS/Metacloud/vSphere/UCS…
Terraform
Spark
Executor
N
Spark
Executor 1
Spark
Scheduler
Kafka
Broker N
Kafka
Broker 1
Kafka
Scheduler
Docker Docker
TraefikMicroservices …
REST API
REST API
Scripted provisioning
Direct
provisioning
Policy, Auto-scaling
VM1
or
BM1
VM2
or
BM2
VM3
or
BM3
VM4
or
BM4
VM5
or
BM5

18. What have we done so far?
Working with partners on adopting and co-developing services
Partners Co-development Partners

19. Anatomy of the service/framework
Riak is Basho Technologies distributed highly available database
Optimized Multi-Datacenter operation
We are working together with Basho Labs on developing and testing their
Mesos Service version of the product

20. Riak Use Cases
Online / Commerce
● Session Control
● Shopping Cart
● Product Ratings and Reviews
Internet of Things
● Connected Device Data
● Sensor Data
● Log Data
Content Management
● Storing Unstructured Data
● Content Personalization
● Advertising Data
Gaming
● Store Leaderboard Info
● Store Bet Transactions
● Online Chat
Digital Communications
● Online Community Chat
● Notification and Alerting
● Mobile Messaging

21. Development phases
Phase 0: Package application in Docker container to deploy on Mesos
Phase 1: Convert application to Microservices Architecture to deploy as
Mesos application with multiple components
Phase 2: Create an intelligent scalable Mesos service based on the
application

22. Riak Service: Components

23. Riak Service: Architecture

24. Riak Service: Persistence

25. Riak Service: Operational Simplicity

26. Riak Service: Highly Scalable
E-commerce Application with Varying Traffic

27. Anatomy of the service/framework
Zoomdata is distributed highly available large scale visualization platform
Optimized very big data set micro-query analytics
We are working together with Zoomdata on developing and testing their
Mesos Service version of the product

28. Zoomdata Service: Components

29. Zoomdata Service: Mesos + Kubernetes
Mesos Slave
Mesos Master
Mesos Slave
Mesos Slave
Zoomdata web app
Mongodb
Spark Worker
Spark Executor
Spark Executor
Proxy
(haproxy, nginx)
Kubernetes
Mongo
Service/RC
Kubernetes
Spark-Proxy
Service/RC
Spark-Proxy
Zoomdata web app
Zoomdata web app
Kubernetes
Framework
Kubernetes
Zoomdata
Scheduler
Service/RC
Zoomdata Scheduler
ProxyGen Script
User
● Every component (Zoomdata
App, MongoDB, Spark-Proxy,
Scheduler) must be started in
independent K8s Pod and there
must be exactly one MongoDB,
Spark-Proxy and Scheduler Pods
meanwhile Zoomdata App can be
scaled with help Kubernetes
Replication Controller.
● There must be defined
Kubernetes Service for MongoDB,
Spark-Proxy, Scheduler as they
will be used in Zoomdata’s App
Pod. Every docker container will
have env variables for every
present Service injected
automatically.

30. Anatomy of the service/framework
StreamSets is an open source continuous big data ingest infrastructure
Accelerates time to analysis with unprecedented transparency and
processing to data in motion.
Cluster deployments
JVM, Docker, Spark Streaming on Mesos
Continuous Operations to Minimize downtime

31. Advantages of Streamsets
Adaptable Data flow - Design and execute intent-driven data flows in a graphical
IDE
Instream Sanitization - transform and process the data on the fly
Intelligent Monitoring - Get early warnings, detect anomalies and take action
Link origins to destinations with in-stream data preparation

32. Streamsets Data Pipeline
MESOS
Streamsets Data Collector (SDC) Architecture
Cluster Streaming mode
Data Collector runs as an application within Spark Streaming,
Spark Streaming runs on Mesos cluster manager to process
data from a Kafka cluster.
The Data Collector uses a cluster manager and a cluster
application to spawn workers as needed.
Cluster Batch Mode :
Data Collector processes all available data from HDFS and then
stops the pipeline.
MapReduce generate additional worker nodes as needed.
Standalone mode
Single Data Collector process runs the pipeline. A pipeline runs
in standalone mode by default.

33. MANTL Data Platform Overview
A modern, batteries included platform for rapidly deploying globally distributed services.
Mantl’s goal is to provide a fully functional, instrumented, and portable container based PaaS for your
business at the push of a button
1) Easy deployment and configuration on different
platforms
2) High availability and self-healing
3) Multi-datacenter support
4) Linear scalability
5) Smart resource management
6) Wide range of supported frameworks

34. MANTL nodes
Consul for service discovery
Mesos cluster manager
Marathon for cluster management
Docker container runtime
Zookeeper for configuration
management
Docker containers
Any Mesos-based workloads
Traefik for proxying external traffic
into services running in the cluster

35. Security &
Operation
Frameworks
Platform
Support
Mantl Components
Core
Components
➢ Data Storage - Riak, Cassandra, HDFS
➢ Data processing - Spark
➢ Security - Vault
➢ Data ingestion – Kafka
➢ Metrics collection - Collectd
➢ Logs forwarding - Logstash
➢ Provisioning - Terraform, Ansible
➢ Cluster management – Mesos, Marathon
➢ Service discovery and configuration management - Consul, Zookeeper,
Traefik
➢ Container runtime - Docker
➢ Cisco Cloud Services, Cisco MetaCloud
➢ Amazon Web Services
➢ Google Compute Engine
➢ Openstack
➢ DigitalOcean
➢ Bare Metal
➢ Autoscaling and high availability
➢ Application load balancer
➢ Application dynamic firewall
➢ Manage Linux user accounts
➢ Authentication and authorization
for Consul, Mesos, Marathon

36. Long Running Services
Big Data Processing
Batch Scheduling
Supported Mesos Frameworks
Data Storage
Mesos makes it easy to develop distributed systems by providing high-level building blocks.

37. ANALYTICS PLATFORM MANAGEMENT
Data Ingestion
•Kafka, Streamsets configurators
Data Storage
Riak, Cassandra, HDFS
Model DevOps Machine learning
MLLib, Spark
Model Deployment
•Model loading, versioning
Cluster Management & Scheduling
Cluster manager
Mesos
Cluster Management long running
service
Marathon
Service Discovery
Consul
Distributed Virtual network
Calico ETCD
ADVANCED ANALYTICS APPS
Analytics Accelerators as Apps
•Forecasting, NLP, optimization, enrichment etc.
SPECIALIZED ADVANCED ANALYTICS MODELS
Consulting Services Design, Build, Deploy Maintain, Manage Performance
DASHBOARDS
ZoomData
Tableau, Qlik, Spotfire,
Excel/BI Cubes …
BUSINESS APPS
Custom ZoomData Visualizations
(D3)
Custom Applications Customer System Integration
CUSTOMIZATION&
MANAGEDSERVICES
CISCO INTERCLOUD
Customization MANTL Data Platform

38. Sample Architecture for Batch Data
Processing
Cassandra
Elastic
Search
Spark
Spark Mllib
Riak
Kibana
Dashboard
VisualisationStorage
Stream
Sets
I/P in multiple formats
Text, logs and json from
various storage source.
Spark application process data
and store to elastic search or
Cassandra or Riak storage for
visualization else it stores in
HDFS
Machine learning algorithm for
data science application
Zoomdata
Data
Discovery
D3
Web
Application
HDFS
StreamSets Data Collector
runs as an application in
Spark Streaming to pull
data from origin to spark
CSV, Tab
delimited etc.
LOG file
JSON
TEXT

39. Sample Architecture for Data Streaming
Kafka
Cassandra
Elastic
Search
Spark
Spark Mllib
Riak
Kibana
Dashboard
VisualisationStorage
Stream
Sets
Streaming network from
different sources
Kafka is used for collecting streaming data and
data is consumed through consumer API by
Streamset for further processing.
Spark application process data
and store to elastic search or
Cassandra or Riak storage.
Machine learning algorithm for
data science application
Zoomdata
Data
Discovery
D3
Web
Application

40. Use Case 1 - Shipped Analytics
Collect log metric from cluster to analyze and drive Alert/Recommend
engine
• Alert Engine - produces alert messages on a basis of some conditions.
• Trend Engine - produces trend messages related to data aggregation.
• Policy Engine – derives from Alert and Trend Engines produces policy
messages which contain recommendations.

41. Use Case 1 - Shipped Analytics Architecture
Central Cluster
Probe
Probe
Probe

42. DataCollector
DataCollector
DataCollector
node
node
node
node
node
Use Case 1 - Shipped Analytics Data Flow

43. • Identify the top technology trends by analyzing public data and open
source projects
• Use machine learning to process a wide range of public data available
on the world wide web and identify high potential emerging
technologies
• Publish results to a web-based dashboards and refresh results regularly
Use case 2 - Emerging Top technology using Public data

44. Use Case 2 - Analysis Through Public data

45. Use Case 2 - Dataflow
APIs
RSS feeds
Scraping
Numeric
Network Data
Text data (articles,
blogs)
Staging
tables
Interactive D3
Dashboards
Websites
Data Sources
Data
Extractors
Data
Storage
Data Processing Machine Learning Visualization
Below we used the framework to execute the project in CIS Data Platform

46. Lambda Reference Architecture
Monitoring / Analytics Cluster (local, Texas-3)
Global Monitoring / Analytics Cluster (global, Texas-1)
Monitoring / Analytics Cluster (local, Ams. -1 )
Monitoring / Analytics Cluster (local, Lon.-1)
Local components and deployment is the same as global, just smaller
Real-time and batch processing (Lambda), anomaly detection, visualization
SSL
Kafka
SSL
SSL
MQTT

47. MANTL Data Platform in Practice:
putting it all together
Working on advanced enabling technology – Mesos, K8S,
Orchestration
Working on developing individual components – dev & co-dev:
Zoomdata, Riak, Streamsets, etc.
Putting together reference architectures and real solutions to
test and further develop the technology
Provide innovation and advanced services to customers
Platform to develop and deliver Microservices and Data
applications

48. Q/A

49. Next steps
Continue partnerships and co-devlopment efforts with industry
leaders to deliver innovation
Continue applying new developed technology to real use cases
and PoC with customers and partners
Continue working closely with A&E and Product teams on
productization roadmap
Work with A&E team closely on prioritization of our R&D
activities to stay closely aligned

50. Anatomy of the service/framework
Elasticsearch is a highly scalable open-source full-text search and analytics
engine
Allows to store, search, and analyse big volumes of data quickly and in near
real time
Underlying technology in application to Optimize complex search in Big data
We are working together with Elastic developing and testing their UTILIZING
Mesos cluster to run Elasticsearch

51. Elasticsearch on Mesos Cluster
Elastic framework scheduler
Marathon framework scheduler
Chronos framework scheduler
Zookeper
Chronos Executor
Marathon Executor
HA Proxy node
Step 1: Mesos Cluster with Marathon &
Chronos running
Step 2:Elastic framework installation on
MESOS Master with a configured # of mesos
slaves to be launched
Step 3: Deploys the ES executore in MESOS
slaves
Step 4: ES nodes discovery and Zookeper
pugin in ES nodes
Step 5 Using plugin nodes find each other
and search is optimized at cluster level
Elasticsearch
executor &
Zookeper pugin

52. MANTL Architecture – Datacenters
Control nodes manage the cluster and resource nodes.
Containers automatically register themselves into DNS
so that other services can locate them.
Once WAN joining is configured, each cluster can locate
services in other data centres via DNS or Consul API
Single Datacentre Multiple Datacentre

53. Client Client Client
RPC
over DNSmask
RPC
over DNSmask
LAN gossip
over DNSmask
Server
Server
(Leader)
Server
replication replication
Lead
forwarding
Internet
Server
Server
(Leader)
Server
replication replication
Lead
forwarding
Datacenter 1
Datacenter 2
Remote DC
forwarding
WAN gossip
TCP&UDP
Consul
➢ Service discovery
➢ Client health-checking
➢ Key-value store for
configurations
➢ Multi-datacenter support

54. Mesos features
Mesos makes it easy to develop distributed systems by providing high-level building blocks.
➢ Scalability
➢ Fault-tolerance and self-healing
➢ Resource isolation
➢ Fine Grained resource elasticity

55. Mesos architecture

56. Mesos setup for developing application

57. ZK
ZK
ZK
Zookeeper quorum
JN
JN
JN
Shared edits
DataNode
DataNode
Active
NameNode
Zookeeper
Failover Controller
Active
NameNode
Zookeeper
Failover Controller
DataNode
Heartbeat Heartbeat
Write Read
Active NN state
monitoring
Standby NN state
monitoring
Monitor and
maintain active
lock
Monitor and try
to take active
lock
➢ Used to store and distribute
data accross a cluster
➢ Is a base for batch analytic
processing
➢ Is highly available and fault
tolerant
➢ Automatically scaled and self-
healing with Mesos
HDFS framework

58. * https://github.com/datastax/spark-cassandra-connector
Mesos Framework for Spark and Cassandra

59. * Smart broker.id assignment
* Preservation of broker placement
* Rolling restarts
* Easy cluster scale-up
Mesos framework for Kafka

60.  Fault tolerant job scheduler
 handles dependencies and ISO8601 based
schedules
 Flexible Job Scheduling
 Supports arbitrarily long dependency chain
 supports the definition of jobs triggered by
the completion of other jobs
Mesos framework for Chronos

61. How MANTL Data Platform for business
application
• Cisco Data Platform can be used to build custom applications or
service for various analysis and Data analytics initiative.
• Companies can streamline Data ingestion, process, manipulate
, analyse and visualize data all in single Infrastructure

62. Yali Load Testing Framework
Yali
Elasticsearch
Kafka
Cassandra
HDFS
Plugins
Kafka
Cassandra
HDFS
Storage
Elasticsearch
Generate data to load test storage

63. Elasticsearch Plugin Testing Results
Job Host
config
Elasticsearch
config
Execution
threads
Batches Records/batc
h/thread
Average
response
from ES, s
Records/s Record size,
b
Records
generated *
10^6
Execution
time, min
Win7, 4
cpu, 16 ram
Cluster: CentOs 6.7,
Elasticsearch 2.1.1,
VPN network, 2
master(4 cpu, 16
ram), 15 worker
nodes(8 cpu, 32
ram)
12 60 50000 78 6804 280 36 84
Local: CentOs 6.4,
Elasticsearch 2.1.1,
VMware virtual
network, single node
(2 Core cpu, 8 Gb
ram)
4 60 10000 1,6 14768 280 2,4 2,5
Records