This presentation covers architectural principles for Software defined "Everything", Microservices - their impact on Azure, a Geo-Spatial Fleet analysis using Spark and HDFS on ESRI and FlowBasedProgramming
2. Who am I ?
Sr. Solution Consultant with LastMileLink Technologies, a
CitySprint UK Innovation Lab, the leading Same Day UK
network, often cited as amongst top 5 in Europe.
: #Lazaredd
:https://uk.linkedin.com/in/lazared
3. Process Flow.
Topics
• Software Defined “Everything”
• Geo-Spatial Fleet Analysis using Big-Data
• Truly Reusable Code – FlowBaseProgramming
5. Community
Network
Marketplace
Technology
Infrastructure
Data
Source: Sangeet Paul Chaudry , http://platformed.info/platform-stack/
Community, Network,
Marketplace
Technology
Infrastructure
Data
Community
Network
Marketplace
Technology
Infrastructure
Data
Defunct or Trailing Hanging On Or Declining Thriving & Growing
Leveraging Data produced by Community, Network & Marketplace
is Key to
Platform Success.
From Pipes to Platforms & API Economy
14. 14
What a Cloud Native is NOT:
•Just Configuring Infrastructure
•Just Orchestrating Containers
•Composing Distributed Systems
•Supporting Ad-Hoc General Purpose Automation
15. Conway’s Law.
Any organization that designs a system (defined broadly)
will produce a design whose structure is a copy of the
organization's communication structure.
Melvyn Conway, 1967
24. Zeta Architecture - Technologies That Work
• Dynamic compute resources
• Common storage platform
• Real-time application support
• Flexible programming models
• Deployment management
• Solution based approach
• Applications to operate a
business
* This is a pluggable architecture
Image Source: courtesy of Jim Scott @ MapR
27. Process Flow.
Blue signals a pick-up
Red signals a drop-off
Sample of how one driver’s journey looks like
Used for:
• Viewing the base unit of analysis
31. K-means clustering analysis – 40 centres
Employed the K-means algorithm to identify clusters
of pickup points
Used for:
• Validating against current service centres map
• Identifying areas for potential expansion
33. K-means 100 centres
Higher granularity clustering
Used for:
• Assessing the frequency of pickups for micro-
clusters (e.g. villages, neighbourhoods)
• Directing drivers to hotter waiting areas
37. 0.0
4.5
9.0
13.5
18.0
0 3 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Expectedparcels
Time of day
Expected parcels allocated to cluster 41 (Stevenage)
Demand variation across time
Used for:
• Positioning couriers in the right place at the right time
For each demand cluster we calculated the
frequency of pickups per hour
39. The solution outline
• Data science capabilities of Spark, easy to use with SQL knowledge
• Map plotting on Esri– heat mapping, zoom in/out capabilities, real-time
• High-performance due to in-memory processing capabilities of Spark
• Can work with large data sets due to high performance disk-based data access
in Hadoop File System (HDFS)
• Can import data from EDW
• All delivered with Bigstep on their high performance Full Metal Cloud
Paraphrasing Commodus in “Gladiator”, who am I ?The general who became a slave. The slave who became a “Gladiator”. The gladiator who defied an emperor. Striking story!
I would like to touch on the following topics as I feel what happens these days is fundamentaly influencing the way we think , operate and deploy.
Simon Sinek - Ted talk.
Big shout out to Sangeet Paul- Platform Revolution for encoding the Platform Blueprint
What I want to emphasize here is that all 3 examples have the same technology and KNOWING the “WHY” they support and enble INTERACTION and VALUE creation and Exchange.
GDPR
Data In Motion , Data at Rest and all other aspects considering all it’s relationships with the other platform components.
We identified four key patterns. Speed, safety, scale, and what I’m now calling “ubiquity.” I used to call this mobile, but what I’m really trying to highlight is the idea that anybody, anywhere, can at anytime interact with your services. Speed is obvious - we can innovate, experiment, deliver value quickly. Safety balances speed with the simultaneous ability to maintain stability, availability, and durability. And scale refers to our ability to elastically respond to changes in demand.
Deliver Continuously using CI and CD. You see, it’s not enough to deploy something, that’s just Day One.
We also have to deploy again, and again, and again. And when you do that, you can’t break stuff. That’s day two…and beyond.
And so we have to figure out how we can build reliable systems from unreliable components.
Meaning that we MUST invite CHAOS into our daily grind. There are many way we can introduce chaos , Simian Army , Gauntl…, the main benefit being: operating at scale without disruption in a distributed manner.
Throught the young history of our industry, we have formulated and followed various methodologies like Waterfall ( represented by Monoliths, Paralysis), Agile (N-tier , Sprints ) and today Continuous Delivery (Micro-Sprints , daily deployments – enabled by Microservices, CI+CD, Automation/Orchestration, attributes of a Cloud native Company)
Cloud Native does not mean build a massive DevOps team and tell them to go configure all of the infrastructure, orchestrate all of the containers, compose the distributed systems, and generally support any kind of ad-hoc, general purpose automation you might dream up, like configuring load balancers or creating NFS mounts.
Cloud Native definition – reword !!!
Conway said you can’t help but reproduce in your architecture the structure of your organization. So now you’ve freed up brainpower to think about where you are and where you need to go.
Define your architecture and then morph your organization. You then have the ability to invoke the “Inverse Conway Maneuver,” taking steps like Netflix did. Once you decide the architecture that your system needs to have, you can restructure your organization to look like that architecture, and according to Conway, that architecture will emerge.
Quote alistair cockburn Hexangonal Architecture which paved the way towards Microservices and Continuous Delivery
The Scale Cube
Define your architecture and then morph your organization. You have the ability to invoke the “Inverse Conway Maneuver,” taking steps like Netflix did. Once you decide the architecture that your system needs to have, you can restructure your organization to look like that architecture, and according to Conway, that architecture will emerge.
Why do we do it this particular way, where should we focus our development effort-is it IP-able , ROI ? , any silver bulltes to watch out for when we consider COTS solutions for parts of our platform.
Mantl.io from Cisco
DevOps, It’s all about agility, so we’re told; Get your application up fast, be the first, pivot quickly, improve often.
But when you're deploying microservices, you'll soon run into problems. How to describe a deployment? Where should you run which service? How do you connect them together? Where should you store secrets? Where should the logs go?
These questions call for a new kind of stack and methology, aka “Continuos Delivery”.
Modern Microservice and Process/Container based applications have attributes that introduce challenges for operating and monitoring these environments:
ScaleModern applications can be constructed out of hundreds/thousands µServices and containers which are loosely coupled but communicate with each other. These applications can dynamically scale up and down depending on the load of the system and can spread all over the world in cloud datacenters and availability zones.
SpeedµServices can be deployed independently from each other and are build for Continuous Delivery processes, so that changes are happening at a high frequency.
SteadinessµServices are not steady anymore. They can be started and stopped just for a functional call and containers are moved around by tools like Kubernetes or Mesos to utilize the hardware in a better way. This requires that monitoring tools identify these services and understand their role in the overall context – without relying on manual tagging.
Golden Copy , Notary for example – Fingerprinted Images , Redeploy, forensics, learn, fix , Rinse and Repeat
So enter Omakase - This is not A La Carte - Rails for exmaple is Omakase because every component was carefully selected by the team who built it however this is not stopping people using their own.
<RIFF ON SPRING CLOUD>
About opinionated expression of distributed systems patterns. Drawing from existing implementations.
These technology concepts are now mature enough that they will stick around and the specific implementation is flexible.
Hexagonal – Alistair Cockborn
But what about polyglot programming? Isn’t one of the promises of microservices that everyone can write their services in whatever languages they choose? Sure. But polyglot is a RED HERRING. Polyglot is not what makes you productive. Polyglot actually makes it harder to enforce the simple rules and explicit contracts that make the platform so powerful. Is it impossible? No. But there’s definitely a cost to providing the same omakase experience to every taste in programming language.
PolyglotUsing the right tool for the problem is one of the paradigms of modern applications. This leads to multiple languages, frameworks and even persistence models for an application. In particular, new distributed caching and database models are getting adopted quickly.
Shows the hot points of pickup points along the uk. A good overview of the overall dataset.
Compared against our service center locations it shows a few differences. A clustering algorithm identifies ‘clusters’ of elements by it’s own. K-means needs to be told how many clusters to look for.
This is what happens if we tell k-means to split the dataset into 100 hot locations.
The blue dots are actual gps information of en-route drivers. Shows typical routes but only some routes go through hot areas.
A ‘cluster’ timetable is used to predict demand at a particular cluster on a particular time. Useful to instruct the driver if he is to stay or to go to it’s destination. This can help uberize the business.
Used a combination of technologies, mostly Spark on Hadoop on Bigstep. Imported data from production Postgres DB via Sqoop into avro and from there via spark into varous CSV files rendered by the ESRI (ARCGIS). Postgres concentrates information from mobile devices.