Control system based on a Highly Abstracted and Open Structure (CHAOS) provides toolkits and services to build a scalable and fault-tolerant control system. The Common, Control Unit (CU), and User Interface toolkits abstract hardware resources and allow developers to focus on control algorithms, device drivers, and user interfaces. Services like the Directory provide device information and the CQL Proxy enables live and historical data access across nodes. CHAOS simplifies control system development through its modular and scalable architecture.
1. Control system based on a
Highly
Abstracted and
Open
Structure
seminar 2012
http://chaos.infn.it C.
Thursday, January 10, 13
2. What is
!CHAOS (Control system based on Highly Abstracted and Open Structure), is
an experiment of CSN 5(technological research experiments) at the LNF and
Roma-TV section.
The project has been appreciated by educational world and the University of
Università
TorVergata LNF - Roma2
Università
di Cagliari
Thursday, January 10, 13
3. Objective
!CHAOS has been designed to widely scale and for manage
large apparatus with different needs in term of storage
technology and speed for data acquisition
Thursday, January 10, 13
4. Objective
new topology in the CS that permit to integrate the triggered
DAQ.
define a new way to access realtime data
data acquisition, storage and indexing fully scalable
Thursday, January 10, 13
5. Objective
A CPP Framework and services that permit to have:
• 99% of adaptation time of instrument to !CHOAS done by
framework
• simplify the creation of the user interfaces
• simplify the creation of the control algorithms basing it on an
abstraction of the I/O channels or full dataset not on physical
device
Thursday, January 10, 13
7. Simplify Vision
User specialized
Boundary
Implementation
CHAOS Boundary Control's
Algorithm
Implementation
Implementation
Instrument's
network User Interface
Driver
Live Data
Directory
Data Cloud Server
Cloud
History Data
Implementation
Thursday, January 10, 13
8. Simplify Vision
User specialized
Boundary
Implementation
CHAOS Boundary Control's
Algorithm
Abstract RPC
Driver
Implementation
Implementation
Instrument's Abstract
network User Interface
Driver Event Driver
Abstract
Direct I/O
Live Data
Directory
Data Cloud Server
Cloud
History Data
Implementation
Thursday, January 10, 13
9. Simplify Vision
User specialized
Boundary
Implementation
CHAOS Boundary Control's
Algorithm
Abstract RPC
Driver
Implementation
Implementation
Instrument's Abstract
network User Interface
Driver Event Driver
Abstract
Direct I/O
Live Data
Live Data Live Data
Directory
Data Cloud Data Cloud Server
Data Cloud
Cloud
History History
Data Data
History Data
Implementation Implementation
Implementation
Thursday, January 10, 13
11. User User Interface for monitoring and controls
Customization
Abstraction
Boundary User Interface Toolkit
Common Toolkit
RPC - Event - Direct I/O
Data Boundary CQL Proxy Nodes Informatio
Execution Unit Toolkit
n
RPC - Event - Direct I/O
Indexer Nodes
Node
Common Toolkit
Management Nodes Information
HTTP - API
Live / History Security
domain
Controls Node
Algorithm Activity
Chaos
Live / History
Directory
domain
Nodes
RPC - Event - Direct I/O
Common Toolkit
Control Unit Toolkit
Driver for sensor and instruments
Thursday, January 10, 13
12. Toolkit and Services
!CHAOS is a set of toolkit and services that will
permit the realization of an control system
Thursday, January 10, 13
13. !CHAOS Toolkit and Services
Common Toolkit Directory
Services
CU Toolkit
Live and History Data
UI Toolkit
CQL Proxy
EU Toolkit
Indexer
Managment
Thursday, January 10, 13
15. !CHAOS Toolkit
Common implements the fundamental software's
layer to RPC, I/O, Event and other’s utility
Toolkit code and for logic
abstracts the !CHAOS resources to the
CU Toolkit device drivers developers.
abstracts the CHAOS resources to the
UI Toolkit developers that need to develop control
and monitoring user interface
abstracts the CHAOS resources to the
EU Toolkit developers that need to develop control
and computing algorithms
Thursday, January 10, 13
17. !CHAOS Services
Directory gives services via RPC (or with other
higher transport layers) for get device
Services structured information, system topology
give the access to insert and query
CQL Proxy operation on live and history data to
other CHOAS node using CHOAS Query
performs the indexing of the new
Indexer inserted data
performs the management of the datas
Management chunk saved on the storage (erase, move,
Thursday, January 10, 13
18. !CHOAS Scalability and Fault-
Some node needs to be scaled for performance reason or for
fault-tolerant:
• Directory Nodes
• Live and History nodes
Thursday, January 10, 13
19. !CHOAS Scalability and Fault-
Server P1
Server P2
Framework
Device 1
Driver X
X
Server P1
Server PN
X Server P1
Server P2
Client Reader
Server P2
Driver
Server PN
Framework
Clients that need to talk to scaled services, has more than one
ip address for every service
Thursday, January 10, 13
20. !CHOAS Scalability and Fault-
Server P1
Server P2
Framework
Device 1
Driver X
X
Server P1
Server PN
X Server P1
Server P2
Client Reader
Server P2
Driver
Server PN
Framework
Every ip address has a priority, the client, as first address, uses
the one with highest priority
Thursday, January 10, 13
21. !CHOAS Scalability and Fault-
Server P1
Server P2
Framework
Device 1
Driver X
X
Server P1
Server PN
X Server P1
Server P2
Client Reader
Server P2
Driver
Server PN
Framework
when a server goes down, another ip will be used
Thursday, January 10, 13
22. !CHOAS Scalability and Fault-
Server P1
Server P2
Framework
use at
ok
Device 1
Server P1
Driver
TS1
Server PN
use at
Server P1
TS1 Server P2
Client Reader
Server P2
Driver
Server PN
Framework
the re-balance occurs when a server with a highest priority,
previously goes down or stopped for maintenance, becomes
up and running
Thursday, January 10, 13
23. !CHOAS Scalability and Fault-
Server P1
Server P2
Framework
use at
ok
Device 1
Server P1
Driver
TS1
Server PN
use at
Server P1
TS1 Server P2
Client Reader
Server P2
Driver
Server PN
Framework
the new server will be used starting from a pre-determinated
time stamp (TS1)
Thursday, January 10, 13
24. !CHOAS Scalability and Fault-
Server P1
Server P2
Framework
use at
ok
Device 1
Server P1
Driver
TS1
Server PN
use at
Server P1
TS1 Server P2
Client Reader
Server P2
Driver
Server PN
Framework
all the operations and the calculation of TS1 are coordinated by
one of the directory nodes
Thursday, January 10, 13
26. Communication Systems
!CHOAS has three different communication system:
RPC, used for command (messages that need to be securely
received and recorded and for message that need a response
[ asynchronous ])
Events, used for inform to a wide range of nodes, that
something has happened (device heartbeat, alert on device
channel, received command, etc..)
Thursday, January 10, 13
27. Communication Systems
Every !CHAOS node has at last
one client and server instance
of any of the three system
Thursday, January 10, 13
28. RPC System
•RPC layer is implemented as a plug-ins System, it's is
abstracted to internal CHAOS framework so we con change it
• used to send important messages that need to be sure of it's
delivering status
• can be used to receive asynchronous responses
• the set command for input channel on control unit is done
with RPC
09/dic/2011
Thursday, January 10, 13
29. RPC System
Message Answer [optional]
Messagge forwarding Response forwarding
Node A Node B Node A Node B
ack [sync] ack [sync]
Thursday, January 10, 13
30. RPC System
RPC Message Flow
• request message is constructed with BSON, serialized and
sent to RPC end point
• if the requester want a response it put answer-code and
sender-address in BSON pack,and automatically RPC System
wait for the answer
• the request and response involves the forwarding of the
ack message, from the receiver to the sender in
Message Answer [optional]
Messagge forwarding Response forwarding
Node A Node B Node A Node B
ack [sync] ack [sync]
Thursday, January 10, 13
31. Event System
• Event is implemented as a plug-ins System it's is
abstracted to internal CHAOS framework so we con change it
• there are four type of event:
• Alert (highest priority)
• Instrument
• Command
• Custom
• all information are packed into 1024 byte and transported
by protocols that support multicast (UDP, PGM[http://
en.wikipedia.org/wiki/Pragmatic_General_Multicast])
09/dic/2011
Thursday, January 10, 13
32. Event System
Some numbers about event data
• Every event type has 2^16 sub type
• Every sub type can have up to 2^16 priority level
• Every subtype can have up to 255 byte of information
09/dic/2011
Thursday, January 10, 13
33. Direct I/O System
• Direct I/O is implemented as a plug-ins system
• used to make insert or select into the live and the history
data
• can be used as stream for receive result data of the query
or read the stored raw data file of the devices
09/dic/2011
Thursday, January 10, 13
35. Abstraction
!CHAOS use a structured serialization for data description
and serialization
• BSON (http://bsonspec.org/) has been chosen, very
similar to JSON but it is binary
• serialization is wrapped into a CPP class so it can be
changed infuture
• it is used in:
•Hardware dataset description's and acquired data
packaging (“channel desc” = value)
•RPC message
•Chaos Query Language
Thursday, January 10, 13
36. Hardware Dataset
•Hardware attributes and I/O channels are described into the
DATASET
•Each attribute is defined by
•Name
•Description
•Type (kind+basic type ex: VOLT32, CUSTOM_STR, etc...)
•Cardinality (single or array)
•Flow (Input, Output, Bidirectional)
•Range
11/dic/2011
Thursday, January 10, 13
40. CU Toolkit
CUToolkit help the Control Unit development
Developer need to extend only one class to create a
CHAOS driver, the “AbstractControlUnit”
AbstractControlUnit expose all the APIs needed for
interacting with !CHOAS
11/dic/2011
Thursday, January 10, 13
41. Control Unit
AbstractControlUnit is an abstract cpp class, that force
developer to implement some method
Control Unit Toolkit
Common Toolkit
11/dic/2011
Thursday, January 10, 13
42. Control Unit
AbstractControlUnit is an abstract cpp class, that force
developer to implement some method
AbstractControlUnit
Control Unit Toolkit •defineActionAndData
Common Toolkit set
•init
•run
•stop
•deinit
11/dic/2011
Thursday, January 10, 13
43. CUToolkit feature
multi-threading messaging dispatcher; every CU has it’s own
thread for dispatching messages after it has been received by
RPC System
cpp method:
setDatasetAttribute
CommonToolk
Control Unit
Toolkit
Common
BSON Toolkit
PACK
sent
from
Thursday, January 10, 13
44. CUToolkit feature
multi-threading messaging dispatcher; every CU has it’s own
thread for dispatching messages after it has been received by
RPC System
cpp method:
setDatasetAttribute
CommonToolk
Control Unit
Toolkit
Common
BSON Toolkit
PACK
the RPC dispatcher
find destination
sent
from
Thursday, January 10, 13
45. CUToolkit feature
multi-threading messaging dispatcher; every CU has it’s own
thread for dispatching messages after it has been received by
RPC System
cpp method:
setDatasetAttribute
CommonToolk
CU Message Queue is filled Control Unit
Toolkit
Common
BSON Toolkit
PACK
the RPC dispatcher
find destination
sent
from
Thursday, January 10, 13
46. CUToolkit feature
multi-threading messaging dispatcher; every CU has it’s own
thread for dispatching messages after it has been received by
RPC System
cpp method:
setDatasetAttribute
CommonToolk AbstractControlU
CU Message Queue is filled Control Unit
Toolkit
BSON Common
BSON PACK Toolkit
PACK
the RPC dispatcher
find destination
Pack is sent
to CUToolkit
BSON scheduler
sent PACK
from
BSON
PACK
Thursday, January 10, 13
48. CU Toolkit
UIToolkit helps the user interface development
Gives API that help to develop: monitoring processes,
device controller etc...
Gives an high performance queue using "locking free"
design. The queue can manage quantizations , over time,
for the dataset data acquisition
UIToolkit
DeviceControl DeviceControl
InterThread Cache Logic
Live Acquisition Thread Pool
AbsractIO
AbsractIO
Driver
Driver
Network
Thursday, January 10, 13
50. EU Toolkit
helps the development of distributed algorithms
(Execution Unit) that can be used to compute or automate
the devices control
The EU can have, in input the device channels or all the
datasset description, so it work on a "Class" of device not
directly on one device
Thursday, January 10, 13
51. EU Toolkit
scheme of flow of input or output of an EU
Specialized Execution Unit
Execution Abstract Execution Unit Result
Queue Queue
Input Parameter Output Parameter
Input Param Pack Output Result Pack
Thursday, January 10, 13
52. EU Toolkit
exception could be concatenated creating a sequence of any
combination of:
• Computing algorithm
• Control algorithm
Specialized Execution Unit
Execution Abstract Execution Unit Result
Queue Queue
Input Parameter Output Parameter
Input Param Pack Output Result Pack
Input Param Pack
Specialized Execution Unit
Execution Abstract Execution Unit Result
Queue Queue
Input Parameter Output Parameter
Thursday, January 10, 13
54. Data Management
!CHOAS has two different management system for the data
apparatus controlled environment data
Live Data
History Data
Thursday, January 10, 13
55. !CHAOS Live Data ui_1
push data
live
dev_1 ui_2
cache
eu_1
11/dic/2011 4
Thursday, January 10, 13
56. Live Data
The system of the Live Data, is ideated for the sharing of
the real time state of the instruments (CU) to other nodes
Thursday, January 10, 13
57. Live Data
Every instrument pushes the data of his I/O channels into
the shared cached through the controller CU
ui_1
push data
live
dev_1 ui_2
cache
eu_1
Thursday, January 10, 13
58. Live Data
1. every node can read data from the central cache at any
time[polling]
2. the nodes can register to central cache for receive
update[push]
ui_1
push data
live
dev_1 ui_2
cache
eu_1
Thursday, January 10, 13
59. Live Data
scalability & reliability
ui_1
push data
live
dev_1
All data of all instruments are
cache
ui_2
distributed across many cache
server
push data
live
dev_2
cache
eu_1
Thursday, January 10, 13
60. Live Data
scalability & reliability
X ui_1 with the algorithm previously
dev_1
live
cache shown, when one server goes
push data
ui_2
down, all client that push's, or
push data read, data on that server, use
live
another server, with highest
dev_2
cache
eu_1
priority
Thursday, January 10, 13
61. Live Data
The usage of the two modes permits to manage, more
efficiently, the bandwidth of the flows of the real time data
(read and write)
not all nodes need realtime data at the same speed:
• the mode (1) is for monitoring and control
• the mode (2) is for control algorithm that can’t lose
packages
Thursday, January 10, 13
63. Live & History Data
ChaosQL
Proxy CQL Interpreter
Abstract History Logic Cache Logic
Common Layer
Storage Driver
Cache Driver
Index Driver
the live data management is merged into history data
management into the ChaosQL node with the Cache Driver
Thursday, January 10, 13
64. History Data
The System for History Data has been designed to
efficiently store the serialized data pack, manage
different type of index on demand, and managing data
file
Thursday, January 10, 13
65. History Data
It is realized by three different services and two
ChaosQL Indexer Management
Storage Management Logic
Management Index Logic
Abstract History Logic
Abstract History Logic
Abstract History Logic
Storage Driver Storage Driver Storage Driver
CQL Interpreter
Index Driver Index Driver Index Driver
Common Layer Common Layer Common Layer
Storage Driver Index Driver
Storage Driver Index Driver
Thursday, January 10, 13
66. Chaos Query Language
ChaosQL implement the following operation on data
• push history/live data per device
• create index on device::attribute::{rule}
• delete index on device::attribute
• retrieve data with logic operation on attribute and index
Thursday, January 10, 13
67. History Data
Manages the insert and select
operations by parsing the Chaos
ChaosQL Query Language
Abstract History Logic
Storage Driver
CQL Interpreter
Index Driver
Insert - as fast as possible stores
Common Layer
the data into the storage system
select - decodes the query, asks at
the indexer the position in the
storage of the data that match the
Thursday, January 10, 13
68. History Data
UI or CU or EU
UI-CU-EU Toolkit
Chaos Common
All nodes, using the ChaosQL Driver
CQL I/O Driver
Direct I/O Driver (that uses direct I/O) can query the
history and live data.
Query
Result
Proxy CQL Interpreter the results are received in
Abstract History Logic
asynchronous mode
Common Layer
Storage Driver
Index Driver
Thursday, January 10, 13
69. Storage
oxy
Q L Pr
C
Managemen
t
Indexer
CUTK
xy Cluster
L Pro
CommonTK CQ
xy
ro
LP
CQ
Index
11/dic/2011
Thursday, January 10, 13
70. Storage
oxy
Q L Pr
C
Managemen
t
Indexer
CUTK
xy Cluster
L Pro
CommonTK CQ
xy
ro
LP
Push
CQ
Data
Index
11/dic/2011
Thursday, January 10, 13
71. Storage
Store
Data
oxy
Q L Pr
C
Managemen
t
Indexer
CUTK
xy Cluster
L Pro
CommonTK CQ
xy
ro
LP
Push
CQ
Data
Index
11/dic/2011
Thursday, January 10, 13
72. Storage
Store
Data
oxy
Q L Pr
C
Launch Managemen
Index t
Indexer
CUTK
xy Cluster
L Pro
CommonTK CQ
xy
ro
LP
Push
CQ
Data
Index
11/dic/2011
Thursday, January 10, 13
73. Storage
Store
Data
get data
pack
oxy
Q L Pr
C
Launch Managemen
Index t
Indexer
CUTK
xy Cluster
L Pro
CommonTK CQ
xy
ro
LP
Push
CQ
Data
Index
11/dic/2011
Thursday, January 10, 13
74. Storage
Store
Data
get data
pack
oxy
Q L Pr
C
Launch Managemen
Index Store t
Index
Indexer
CUTK
xy Cluster
L Pro
CommonTK CQ
xy
ro
LP
Push
CQ
Data
Index
11/dic/2011
Thursday, January 10, 13
75. Managemen
t
Storage
CUTK Indexer
CommonTK Cluster
oxy
Q L Pr
C
xy
L Pro
CQ
xy
ro
LP
CQ
Index
11/dic/2011
Thursday, January 10, 13
76. Managemen
t
Storage
CUTK Indexer
CommonTK Cluster
oxy
Q L Pr
C
xy
L Pro
CQ
EUTK
xy
CommonTK
ro
LP
CQ
Index
UITK
CommonTK
11/dic/2011
Thursday, January 10, 13
77. Managemen
t
Storage
CUTK Indexer
CommonTK Cluster
oxy
Q L Pr
C
xy
Pro
Query Live CQ
L
Data
EUTK
xy
CommonTK
ro
LP
CQ
Index
UITK
CommonTK
11/dic/2011
Thursday, January 10, 13
78. Managemen
t
Storage
CUTK Indexer
CommonTK Cluster
oxy
Q L Pr
C
xy
Pro
Query Live CQ
L
Data
EUTK
xy
CommonTK
ro
LP
CQ
Index
UITK
Query
CommonTK
History
Data
11/dic/2011
Thursday, January 10, 13
79. Managemen
t
Storage
CUTK Indexer
CommonTK Cluster
oxy
Q L Pr
C
execute
search on idx
DB
xy
Pro
Query Live CQ
L
send
Data search on
EUTK idx
xy
CommonTK
ro
LP
CQ
Index
UITK
Query
CommonTK
History
Data
11/dic/2011
Thursday, January 10, 13
80. Managemen
t
Storage
get pack
CUTK at location Indexer
CommonTK Cluster
oxy
Q L Pr
C
execute
search on idx
DB
xy
Pro
Query Live CQ
L
send
Data search on
EUTK idx
xy
CommonTK
ro
LP
CQ
Index
UITK
Query
CommonTK
History
Data
11/dic/2011
Thursday, January 10, 13
81. Managemen
t
Storage
get pack
CUTK at location Indexer
CommonTK Cluster
oxy
Q L Pr
C
execute
search on idx
DB
xy
Pro
Query Live CQ
L
send
Data search on
EUTK idx
Receive
History xy
CommonTK
ro
LP
Data
CQ
Index
UITK
Query
CommonTK
History
Data
11/dic/2011
Thursday, January 10, 13
82. History Data
UI or CU or EU
the query can be splitted in more than
UI-CU-EU Toolkit
QueryChannel
one time period from the receiving proxy.
Chaos
Common
The splitted query are sent to other proxy
CQL I/O Driver
Direct I/O Driver
CQLPack
that will process the sub query
Original
Re
ry
lt
Result
sul
ue
su t
Re
Q
Proxy CQL Interpreter Proxy CQL Interpreter
Proxy CQL Interpreter
Abstract History Logic Abstract History Logic
Abstract History Logic
Common
Common
Storage
Storage
Common
Driver
Driver
Driver
Driver
Layer
Layer
Index
Index
Storage
Driver
Driver
Layer
Index
CQLPack CQLPack CQLPack
1 2 3
Sub Query
Thursday, January 10, 13
83. History Data
Executes the necessary operations
Indexer to index the data saved on the
storage system
Management Index Logic
Abstract History Logic
Storage Driver
Index Driver
two type of index:
• TMI - Time Machine index
Common Layer
• DSI - Dataset Index
Thursday, January 10, 13
84. History Data
TMI - Time Machine index
•This index permits to catalog every data pack for a
device for the timestamp.
• all the data pack are recorded and associated to their
timestamp
• this index is managed by system
Thursday, January 10, 13
85. History Data
DSI - Dataset Index
•This index permits to catalog attributes of the dataset
for a determinated instrument
• there will be possibility of customize the index
algorithm
• this index is created on demand and on specific time
periods
Thursday, January 10, 13
86. History Data
Executes management operations
Management
on files that store the data packs
Storage Management Logic
Abstract History Logic
Storage Driver
Index Driver
some data pack could be subject to
Common Layer
ageing
the ageing state will determine the
Thursday, January 10, 13
87. Project Status
C User Interface LV User Interface MatLab UI
User Interface Toolkit 20%
Common Toolkit
distributed distributed Directory
Live Data History server
infrastructure service Data service
Common Toolkit
CU Toolkit
Control Unit Control Unit I/O Unit Exec Unit
CU-LV interface
CU modules (config)
CU modules (init)
CU modules (set)
CU modules (de-init)
CU modules (stop)
I/O modules
I/O modules
I/O modules
EU modules
EU modules
EU modules
CU modules (config)
CU modules (init)
CU modules (set)
CU modules (de-init)
CU modules (stop)
INFN-Vitrociset meeting 23/02/12
Thursday, January 10, 13
88. developments’ details
QT User Interface:
prototype ready (Qt)
UI Toolkit:
prototype ready
C wrapper UI Toolkit:
prototype ready
LV User Interface:
prototype in preparation (shlib for LV)
Live Data:
single-instance (no-distributed) prototype ready
RPC client/server:
prototype ready (msgpack and zmq impl)
Event client/server:
prototype ready (boost asio impl)
infrastructure Direct I/O c/s: todo
History Data service:
performance test (mongodb) ongoing (UniTV)
Directory services:
DS simulator ready; DS under development
scalability & availability:
test facility in preparation
Common Toolkit:
prototype ready
CU Toolkit:
prototype ready
CU modules:
prototype ready (C++)
CU-LV Interface
prototypes
& modules:
almost ready
INFN-Vitrociset meeting 23/02/12
Thursday, January 10, 13
89. ...closing
in the next year all development force will be focused to
realize the system for the history data and for the directory
service, a prototype for each system will be available for 2013
2Q
Thursday, January 10, 13
90. CHOAS Group
Claudio Bisegni, Luciano Catani,
Giampiero Di Pirro, Luca Foggetta,
Mazziteli Giovanni, Matteo Mara,
Antonello Paoletti, Stecchi
Alessandro, Federico Zani
thesis's
Flaminio Antonucci (tor Vergata
Ingengeria), Andrea Capozzi (Tor
Vergata Ingengeria), Francesco
Iesu (Cagliari Infromatica),
thanks for the time
Thursday, January 10, 13