Testament - a blueprint for hyper-personalized services at web scale

Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software
License, version 2.0
introducing
the Testament Architecture
● An architecture for hyper-personalised services in an internet
of things that you can use.
Robert Gibbon – be.linkedin.com/in/robertgibbon
The content of these slides is my own original thinking; or derived from information freely available
in the public domain.
Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0

who am I?
● I am an independent information technology services consultant
● 15 years in the business
● 5 years doing stuff with big data
● And some more years on web-scale site management duty

why did I make this deck?
I made this deck because
● I had a collection of ideas that were buzzing around in my head and I wanted to get them
out onto paper.
● Neither the concepts and technologies, nor the way they are combined and presented in
this architectural paper are related to any of my past or present clients.
● But...if you try to implement a Testament architecture and it doesn't work out for you, don't
blame me!

contents
This slide set goes over
● The operational context of the Testament architecture
● The challenges to service delivery that the Testament approach prioritises
● A flythrough of the major features and technologies in a Testament implementation
● Some closing thoughts

who stands to benefit from Testament?
Testament is a high level architecture blueprint. It is oriented to multifaceted consumer services that
require
● global reach
● millions of users
● multiple devices per user
● always available
● long data retention times
● processing very big datasets
Testament proposes a set of building blocks on which to found certain aspects of such a service.

what challenges does Testament try to
address?
Testament prioritises these aspects of successful hyper-personalized service delivery at web scale.
● serving hyper-personalised data to millions+ concurrent users
● serving and reconciling data with split-brained services and offline-often devices
● serving “single record of truth” datums that must always be kept up to the minute
● cost-critical long term data storage and access for very big data

front end: conceptually
Content Distribution Network
Global Service Center 1
User X Database Branches User Y Database Branches
User X Managed Devices
CHICAGO
AMSTERDAM
TOKYO
Global Service Center 2 Global Service Center n
Communal Edge Service Nodes User Y Managed Devices

front end: logically
Distributed Filesystem
VCS, DDB, CEP
Offline-often Devices
CHICAGO
AMSTERDAM
TOKYO
VCS, DDB, CEP VCS, DDB, CEP
Hosted Application Servers Offline-often Devices
Standalone dbs Standalone dbs

front end: exemplar physical model
XtreemFS
Git, Hbase, Storm
Sunglasses, Phones, Wristbands
CHICAGO
AMSTERDAM
TOKYO
Git, Hbase, Storm Git, Hbase, Storm
Games, eShops, Content on-demand, Social TVs, Hi-Fis, Fridges
SQLite files SQLite files

front end: summary of the bits
Testimonies
● Are a user, session, and device bounded collection of events and facts
● eg. “A testimony of latlongs, browsing events, DNS lookups and app usage activities from Robert's phone for last
Friday”
● These testimonies are built to help better understand the user and consequently better serve the user, over time
Global Service Centers
● Handle asynchronously submitted testimonies
● Pull new testimonies from the CDN
● Automerge and tag them, eg. “Merge this testimony onto the <Robert's phone> branch and tag it <last
Friday>”
● Precompute ahead of pull requests from Edge Service Nodes for updated User Database Branches
● Run “single-version-of-the-truth” database services for those datums that must always be up to the minute, such
as authentication credentials, avatars, screen names etc.

● Common service backplane
● Write-locally, read-globally
● Ship User Database Branches between Global Service Centers and Edge Service Nodes
● Ship Testimonies between Edge Service Nodes and Global Service Centers
User Database Branches
● Provide a self-contained individual user database to enable hyper-personalized service delivery
● eg. attitude/trend/interest/demographic classifiers; motivators, intents and triggers; recencies; rising
and decaying trends
● Form a container for Testimonies to be backshipped via the Content Distribution Network

Communal Edge Service Nodes and Managed Devices
● Deliver actual application services to the user
● eg. Games, grocery shopping, grocery storage, streaming radio, TV on demand, household
automation, remote power/HVAC management, social etc.
● Are two-way, in that they build and backship Testimonies, to help better understand the user
and consequently better serve the user, over time

back end: conceptually
Global Service Center 1
CHICAGO
TOKYO
AMSTERDAM
VANCOUVER
HEBRIDES
ATLANTA
Global Service Center 2 Global Service Center n
Nearline Vault 1 Nearline Vault 2 Nearline Vault n

back end: logically
Distributed Filesystem
Compute Grid 1
CHICAGO
TOKYO
AMSTERDAM
VANCOUVER
HEBRIDES
ATLANTA
Compute Grid 2 Compute Grid n
Massive Array of Idle Devices Massive Array of Idle Devices Massive Array of Idle Devices

back end: exemplar physical model
XtreemFS
Spark on Mesos
CHICAGO
TOKYO
AMSTERDAM
VANCOUVER
HEBRIDES
ATLANTA
Spark on Mesos Spark on Mesos
Depreciated Commodity Systems Depreciated Commodity Systems Depreciated Commodity Systems

backend: summary of the bits
Global Service Centers
● Deliver the usual background batch processing services to build, refine and update apt and
timely user insights from available Testimonies, so that front-end applications can better
serve the user
● Identify Testimonies that the Global Service Centers cannot use at this moment and that are
therefore eligible to be archived to the Nearline Vault
● Eg. Analysis technology is not yet sufficiently mature, and/or Testimony content has
decayed in relevance
● Acts as a publish locally, read remotely conduit from Global Service Centers to Nearline Vaults

backend: summary of the bits
Nearline Vaults
● Load old Testimonies onto a storage device on a Depreciated Server, spin down the storage
device, and power off the Depreciated Server for long term storage
● Power up the Depreciated Server and spin up the required storage device on demand, for
retrieval of the old Testimonies in the future.
Depreciated Servers
● Old Servers that have depreciated, configured with high storage device density, often
racked and packed in self-contained DC freight containers.

closing thoughts
Testament only considers a small subset of architectural facets; its really only a teaser.
There is a spectrum of architectural facets to desigining a successful hyper-personalised, web-scale service that shall
be considered, and that are not considered by Testament.
For example, User Database Branches and Testimonies must be protected and able to defend against unauthorised
access, such as from competing service providers or from malicious users.
Also, the personally identifiable link between Testimonies and the user to which they relate and the legal and ethical
rights of users to sever that link, is another facet that these slides have not considered.
Importantly, “no man is an island”, and handling Testimonies of the relationships between users has not been
discussed in this slide deck.

slide deck license
Copyright 2014 Robert Gibbon
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Testament - a blueprint for hyper-personalized services at web scale

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Testament - a blueprint for hyper-personalized services at web scale

Ähnlich wie Testament - a blueprint for hyper-personalized services at web scale (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Testament - a blueprint for hyper-personalized services at web scale