See the video here: https://vimeo.com/131631801 IoT projects are really integration projects. This talk introduces Sesam and Data Oriented Architecture, useful for IoT, Micro Services, Master Data Management. Why DOA is better than SOA, and a new way of thinking! The talk will become available on Vimeo soon. I realize that the "SOA is dead commandment" is provocative. SOA is not dead but lacking when implemented with canonical data model. Perhaps we should make a Sesam manifesto instead, with "we value X over Y"

1. Hafslund AMS
Drinking from the fire hose
at a large IOT project
Jon Andreas Pretorius, Hafslund Nett
Axel Borge,Sesam
Simen Sommerfeldt, Bouvet
to NDC 2015

2. Hafslunds.2

3. s.3
Operational structure
Heat Network MarketsPower
Hydro&power& District&hea1ng&
produc1on&and&
distribu1on&
Bio7energy&
Network&
System&control&
centre&
Power&sales&
Billing&
Customer&
services&

4. • Hafslund Nett owns and operates Norway's largest
electricity grid and has long had one of the lowest net
rents
• Hafslund Nett owns and operates the regional grid in
Oslo, Akershus county and Østfold county
• Hafslund Nett owns and operates the distribution network
in Oslo and most of Akershus and Østfold counties
• Number of distribution network customers are 675,000
• The network consists of 37,000 km overhead lines and
underground cables
• Hafslund Driftssentral is one of Europe's most advanced
operating centers, that controls, monitors and optimizes
power to 1.4 million people, Hafslund Varme's district
heating plants in the Oslo area and Hafslund Produksjon's
power plants in Glomma
Business Area Network
s.4

5. Hafslund was founded more than 100 years
ago - in 1898
s.5

6. There will be more changes to the power grid
operation the next five years than the 100 last
years
1899 1911 2011 2020
?&

7. Changes in regulations will increase complexity
significant and increase demand for automation
s.7
35 000 enkle fjernavleste målere
Årlig driftskost pr måler: ca 950
700 000 komplekse fjernavleste
målere
Har i dag buffer for feilretting.
Stort sett bare ifm måler-
avlesning at programmet er tett
Årlig driftskost pr måler: 150
Alt online og ingen buffer eller
servicevindu. Alt må alltid være
tilgjengelig. Kan illustreres ved å
tenke at det var måleravlesning
hver dag hele døgnet
IT er i liten grad en trussel for
omdømme
Hacking, virus mm vil utgjøre en
mye større trussel generelt og
målere vil kunne hackes
Kompleksitet høy, men vi reddes
av rolige perioder
Kompleksitet vil være betydelig
høyere og konstant
I dag 2020
AMS
elHUB
Kundens forventning som i 1990
Hvordan vil kundens forventning
endres?

8. AMS:&
• The&new&smart&meters&detect&power&consump1on&on&
an&hourly&basis&and&automa1cally&sends&informa1on&
about&consump1on&to&the&grid&company&
• The&new&smart&meters&records&also&includes&data&
related&to&voltage&quality,&ground&faults&etc&
• The&new&meters&have&two7way&communica1on&
between&the&meter&and&grid&companies,&and&will&give&
customers&current&informa1on&about&their&own&
consump1on&and&instant&prices&for&electricity&and&grid&
transport.&
• Grid&companies&are&responsible&for&the&installa1on&and&
opera1on&of&the&AMS&system,&but&they&can&not&deny&
other&service&providers&to&offer&customers&management&
and&informa1on&services.&
&
Advanced&Metering&and&Control&Systems&(AMS)&are&introduced&within&2019.01.01&
AMS$%$the$founda/on$of$the$future$of$digital$
power$grid

9. s.9
Elhub and the supplier centric model
ElHub
Statnett has been commissioned by NVE to establish
Elhub.
Elhub shall collect all metering values for Norway and
makie these values available for power suppliers and
their end customers. Furthermore Elhub will support
processes for customers moving or switching suppliers,
and compile data for clearing between participants in
the electricity market
For Hafslund Nett this means that collected and verified
hour ly values from all AMS meters shall be transferred
to ElHub once a day
When vthe supplier centric model is established ,
customers will only deal with the electricity company
(example service and infrastructure provider of mobile
telephony)
The supplier centric model creates major changes in
business processes and data exchange in the industry
Drawing from elhub.no

10. s.10
System&D& System&…&
System&C&System&B&System&A&
System&N&
Hafslund investigated two
alternative solutions for
integration architecture that will
support the demands of new
AMS solution;
- ServiceBus
- Data hub(Semantic/RDF)
Hafslund has experience with
both solutions, but the project
consider a Data Hub based
solution most appropriate in this
context;
- Increased stability
(asynchronous data
exchange)
- Fewer integration points
- Similarly architecture chosen
for central El Hub
DataNAV&
Choice$of$integra/on$solu/on

11. s.11
IFS$ERP$
Warehousing&&&
Logis1cs&
Project&module& WO7module& 360º&Scheduling&
New&field&system&
Economy&installa1on&registry&
Documenta1on&
HR/resource&
Rollout$
AMS$
Opera/ons$
AMS$
Stage&Planning&and&
monitoring& Assign& Start&& Perform& Report&
Project,)Opera-ons,)
Maintanance)excis-ng)
Recep1on&7&
withdrawals&goods&
Data&Hub&
GeoNIS&
#installa1on&
Quant&
#AMS&&
Generis&
#old&meters&
CAB&
#Customer&&
Datawarehouse&/&
archive&
Consolidated&customer&and&
installa1on&data&from&Data&hub&
Data&sources&for&
rollout&
Data&Recipients&
rollout&
Historical&data&archive&
and&analysis&
Integration engine
All masterdata is
consolidated in
Data Hub
Data Hub is the
only source for all
business
applications
In the semantic
data base all data
are connected
Data Hub provides
great potential for
management of the
information model
and analysis
Established$applica/on$solu/on$design

12. www.hafslund.nowww.hafslund.no

13. Advisor and CTO, Bouvet Oslo
Lær Kidsa Koding!
Oslo IoT meetup
@sisomm

15. Master data:
The Elephant
in the room

16. System
System
System
System
System
System
System
System
System
System
System

17. I have given many talks and written
several articles about IoT

18. Hypothesis: Internet of Things
projects always involve
integration

20. Archive
HR
CMS
Finance
Payroll
Classic
Point-to-
Point
integration

21. Archive
HR
CMS
Finance
Payroll
ESBESB
Classic
integration
with ESB

22. The systems keep changing their
need for information from other
systems

23. Archive
HR
CMS
Finance
Payroll
Common data model?

26. Traditional
SOA with a
canonical
datamodel
PREVENTS
flexibility

27. DOA – Data Oriented
Architecture.
Enter
27

28. Archive
HR
CMS
Finance
Payroll
Keep the models, keep the data

29. We copy the
domain
context in the
systems into
SESAM

30. Convert data to triplets - RDF
ID Name Position Born E-mail Manager
101 Tim Berners-
Lee
Programmer 08061955
timbl@w3c.org 958
958 Vint Cerf Inventor 23061940 vint@stanford.edu 999
765 Pål Spilling Professor 04111940 pspilling@uio.no 765
Subject Predicate Object
101
101
101
101
Type Person
Name
Position
Born
E-mail
Manager
101
timbl@w3c.org08061955 958ProgrammerTim Berners-Lee101

31. Universally unique identifiers
Subject Predicate Object
www.org.no/data/system/person/1 Type Person
www.org.no/data/system/person/1 Name Tim Berners-Lee
www.org.no/data/system/person/1 Position Programmer
www.org.no/data/system/person/1 Manager www.org.no/data/system/person/2
www.org.no/data/system/person/2 Name Vint Cerf
www.org.no = Unique organisation on the internet
www.org.no/data/system/person/1 = unique id of the information element

32. Archive
HR
CMS
Finance
Payroll
Link data from different GRAPHS
Duplicates
Contact=Contact
Location= Depnr
Empnr=ID1
Org=Depnr

33. With DUKE we can link
elements with no common
identifiers – using statistical
algorithms and training on data
sets

34. Archive
HR
CMS
Finance
Payroll
Data sources can enrich eachother
HUB
Doc
Meta1
Meta2
Doc

35. Archive
HR
Turnus
Finance
Payroll
Keep data from ”dead” systems
Nav
HR
Payroll
Search

36. Archive
HR
CMS
Finance
Payroll
Hub
How things are connected

37. HR Dest
SDShare
Source
HUB
• Based on Atom: Pull data, don’t push
• Asynchronous
• Subscribers ask for data that has changed
since the last time
• Update frequencies are adjustable
• You can ask for changes or the whole dataset
• Data formats changed in transfer.

38. www.sdshare.org

39. Sesam uses a Hashtracker to
keep track of the latest changes,
or an explicit “latest changed”
field

40. Read using SQL/File acces,
Write using API’s

41. A push receiver is a http server
receiving RDF fragments. It
calls the API of the receiving
system. If the call fails, Sesam
can try again

42. DOA
Database'
Content'
Mngmt'
File'
System'
Enterprise'
Search'
Reporting'
Analytics'
Data'Hub'
System'X'
Public'
Open'
Data'
Content'
Mngmt'
Enhance'
and'
Connect'
COLLECT
CONNECT
SHARE
Transform'
Data objects flow NOT messages

43. Kafka for extra throughput
SDShare'Server'
KaDa'
Provider'
RDF'Store'
KaDa'Queue'
The Kafka Provider Pulls Information off from the Queue and can add
extra data from the RDF store before exposing it out via SDShare. It can
also apply filters based on data in the hub or the item on the queue.

44. • SQL Databases via jdbc
• CSV files
• RDF triple stores
• Sharepoint
• Kafka
• XML files
• LDAP providers
• Excel files
• MS Exchange server (mail and calendar)
• SDSHARE – anything! (MongoDB, etc)
Data sources and sinks

45. 45

46. Data Analytics & Enhancement Existing Systems
Processes Search and Reporting
All Data Indexed
Contribute data
Drive process through
state change
Models in data,
Constraints in data
Act on all data
Analytics results are
just more data
Complete
views of all
systems and
processes
Use Data
All people can ask
all questions
U
niform
ly
Structured
data
from
heterogeneous
sources
System Improved

47. Other systems can keep running
even if one is down. And you
can upgrade a system or install
a new with fewer impacts

48. The customer controls the
information model and
becomes more independent
from vendors

49. Customers can prototype
and combine data for new
solutions rapidly

50. 1. Thou shall only get data from other domains through Sesam
2. SOA is dead, long live DOA. Processes advance through state changes
3. There can never be a common data model in the company
4. Thou shall never query Sesam directly, but through SDSHARE
5. Thou shall be comfortable with eventual consistency
6. Thou will always get the same answer when you ask Sesam the same
question. And Sesam can say the same things many times
7. The world is asynchronus, as is Sesam. Don’t try to shoehorn synchronicity
8. Thou shall embrace that data can have different sources/master and values
9. The systems need not know about Sesam
10. Sesam is not a backup.
The Commandments of Sesam

51. • Runs in Docker containers
• Github and Saltstack are used to keep all
installations up-to-date
• At the core: Virtuoso Triple Store
• Includes a data browser
• Indexed with SOLR to provide universal search
• All communication happens with SDSHARE
• Configuration over coding
Sesam tech

52. A paradigm shift for developers
• Eventual consistency
• “Pilfering” of data
• RDF and SDShare
• Sparql is not SQL
• Idempotence: Sesam
can send duplicates
• No RPC calls or
message passing
• You need an information
architect in the project
• Don´t add more queues.

54. Pedal to the
metal! Now
implement
AMS with
SESAM!

55. A recap of the requirements
• Massive amounts of data
• Many systems must be coordinated
• Many stages in the deployment, with changing
needs
• Systems will be upgraded and changed
• The systems were not designed to cooperate with
each other
• Bugs and errors happen – in systems and human
actions.

60. 670.000 x 4 x 24
=
64.320.000 daily readings

61. YAY! Need for american-scale
technology
Yay! Need
for
AMERICAN
scale
technology!

62. The first challenge is to sync
information across systems
before the measurements can
begin

63. Construct the SQL for the output
feeds, implement push receivers
for input feeds.

64. All the
arrows are
feeds in/out
of Sesam

65. Slide source: Ståle Heitmann, Computas

66. Slide source: Ståle Heitmann, Computas

68. When to use Sesam
• When all else is tried – you are f***ked
• If you have many domains in the company
• If your integration work involves a lot of
data transformation, lookup and
conversion
• If the logic in the ESB rivals that of the
systems
• For Internet of things projects
• As a collector for big data projects

69. Micro services need to get
information from several places

70. Want to know more?
• contact us at info@sesam.no and we will
help you get started
• www.sesam.no
• www.sdshare.org

71. • Anders Volle
• Ståle Heitmann
• Steinar Rudsar
• Axel Borge
• Øystein Isaksen
• Graham Moore
• Lars Marius Garshol
• Steinar Rune Eriksen
Thanks to...

72. Questions?

73. Thanks!
Simen.Sommerfeldt@bouvet.no
@sisomm/995 07 733
Axel.borge@bouvet.no
@axelborge/905 92 955