Pi Day 2022 - from IoT to MySQL HeatWave Database Service

Frédéric Descamps
Community Manager
MySQL
From IOT to
MySQL HeatWave
Database Service in OCI
Oracle Pi Day - March 14th 2022

Who am I ?
about.me/lefred
Copyright @ 2022 Oracle and/or its affiliates.
2

@lefred
MySQL Evangelist
using MySQL since version 3.21
devops believer
living in
born at PiDay !
h ps://lefred.be

Frédéric Descamps
3

IoT
the Internet of Things
4

IoT - the Internet of Things
The Internet of Things describes the network of physical objects—“things”—that are
embedded with sensors, software, and other technologies for the purpose of connecting
and exchanging data with other devices and systems over the internet.
These devices range from ordinary household objects to sophisticated industrial tools. With
more than 7 billion connected IoT devices today, experts are expecting this number to grow
to 22 billion by 2025.
5

MySQL
the most popular Open Source Relational Database
Management System
6

MySQL HeatWave
MySQL is the most popular Open Source Database and it's developed by Oracle.
MySQL is a relational database management system (RDMBS) but also a NoSQL Document
Store.
7

MySQL HeatWave
MySQL is the most popular Open Source Database and it's developed by Oracle.
MySQL is a relational database management system (RDMBS) but also a NoSQL Document
Store.
MySQL is also available in Oracle Cloud Inftrastructure (OCI) as a managed service.
HeatWave is a massively, high performance, in-memory query accelerator for OCI MySQL
Database Service that accelerates MySQL performance by orders of magnitude for
analytics and mixed workload.
7

Raspberry Pi
hardware & software
8

Hardware
A Raspberry Pi Model B:
9

Hardware
A DHT22 sensor providing mesure for temperature and humidity:
10

Hardware
This is the assembly schema:
11

Software
On the Raspberry the following packages are required:
Python
pip
mysql-connector-python (from pip)
Adafruit_Python_DHT (from pip)
12

Software
On the Raspberry the following packages are required:
Python
pip
mysql-connector-python (from pip)
Adafruit_Python_DHT (from pip)
We will also create our own program and a systemd script to manage it (start/stop/status):
metrics_to_MySQL.py
piday_metrics@.service
12

Architecture
Deploy our Database in OCI
13

OCI Architecture - simpli ed
14

Deploy in OCI
Once you have an Oracle Cloud account, you can just use the link on the next slide to
deploy the required architecture using a Oracle Resource Manager Stack (Terraform).
This will create all the necessary resources:
A VCN
The Public and Private Subnets
An Internet Gateway
The Security Lists
A compute instance with MySQL Router installed and con gured
A MySQL Database Service Instance
15

Deploy in OCI - Stack
h ps://github.com/lefred/oci-iot-mds
16

Deploy in OCI - Stack (2)
17

18

19

20

21

22

23

24

Deploy in OCI - Created Resources
25

Deploy in OCI - Created Resources (2)
26

Database Design
Schema, tables, collections, ... how to store the data?
27

MySQL 8.0
With MySQL 8.0, you can mix relational tables and JSON Documents.
28

MySQL 8.0
With MySQL 8.0, you can mix relational tables and JSON Documents.
SQL + NoSQL = MySQL Document Store
28

Built on the
MySQL JSON
Data type
and Proven
MySQL
Server
Technology
Provides a schema exible JSON Document Store
No SQL required
No need to de ne all possible a ributes, tables, etc.
Uses new X DevAPI (available on premise and in OCI only !)
Can leverage generated column to extract JSON values into materialized
columns that can be indexed for fast SQL searches.
Document can be ~1GB
It's a column in a row of a table
It cannot exceed max_allowed_packet
Allows use of modern programming styles
No more embedded strings of SQL in your code
Easy to read
Also works with relational Tables
Proven MySQL Technology
29

Design
name
short_name
temperature
humidity
public_ip
...
{ {
30

Design - explanations
For realtime (current) information, we use a collection of JSON documents where each
document represents the state of a device with the lastest values it collected.
For historical data generally used for analytics, we use relational tables that we will use later
with HeatWave if we need to improve the performance.
31

Design - connection to the MySQL Instance
We use port 6448 (X Protocol) on the Public IP of the Compute Instance where MySQL
Router is deployed.
32

Design - schema and collection creation
33

Design - relational tables creation
34

At some point, dealing with historical data
like archiving, deleting old records can
become a heavy operation.
HeatWave has been designed to deal with a
huge amount of data but on MySQL,
partitioning is a solution to deal with these
operations.
Design - relational tables creation
35

Design - relational tables creation: partitioning
SQL
SQL>
> alter
alter table
table humidity_history
humidity_history add
add day_date
day_date date
date as
as (
(date
date(
(time_stamp
time_stamp)
))
) stored
stored,
,
drop
drop primary
primary key
key,
, add
add primary
primary key
key (
(id
id,
, day_date
day_date)
);
;
SQL
SQL>
> alter
alter table
table humidity_history
humidity_history partition
partition by
by range
range columns
columns(
(day_date
day_date)
) (
(
partition
partition p0_before2022
p0_before2022 values
values less than
less than(
('2022-01-01'
'2022-01-01')
),
,
partition
partition p2022_01
p2022_01 values
values less than
less than(
('2022-02-01'
'2022-02-01')
),
,
partition
partition p2022_02
p2022_02 values
values less than
less than(
('2022-03-01'
'2022-03-01')
),
,
partition
partition p2022_03
p2022_03 values
values less than
less than(
('2022-04-01'
'2022-04-01')
),
,
partition
partition p2022_04
p2022_04 values
values less than
less than(
('2022-05-01'
'2022-05-01')
),
,
partition
partition p2022_05
p2022_05 values
values less than
less than(
('2022-06-01'
'2022-06-01')
),
,
.
..
..
.
partition
partition p2023_12
p2023_12 values
values less than
less than(
('2024-01-01'
'2024-01-01')
),
,
partition
partition p9_future
p9_future values
values less than
less than(
(maxvalue
maxvalue)
))
);
;
see h ps://dev.mysql.com/doc/refman/8.0/en/partitioning-management-range-list.html
36

Dedicated User
MySQL user creation
37

MySQL Dedicated User for IoT
It's a best practice that every applications use their own user to access MySQL.
In our case we will create one for all our devices.
We create a user with the required priveleges to write data from our Rasbperry Pi:
SQL
SQL>
> create
create user
user piday identified
piday identified with
with 'mysql_native_password'
'mysql_native_password'
by
by 'Fred140376#'
'Fred140376#';
;
SQL
SQL>
> grant
grant select
select,
, insert
insert,
, update
update on
on piday
piday.
.*
* to
to piday
piday;
;
38

Collecting Data
from Raspberry Pi to MySQL
39

We need Adafruit_DHT module to read the sensor:
import
import Adafruit_DHT
Adafruit_DHT
sensor
sensor=
=Adafruit_DHT
Adafruit_DHT.
.DHT22
DHT22
gpio
gpio=
=4
4
humidity
humidity,
, temperature
temperature =
= Adafruit_DHT
Adafruit_DHT.
.read_retry
read_retry(
(sensor
sensor,
, gpio
gpio)
)
Reading from the Sensor
I am using Python on the Raspberry Pi.
40

Connection to MySQL
Of course all the connectors maintained by Oracle MySQL support the X Dev API
41

Connection to MySQL - Python
import
import mysqlx
mysqlx
connect_args
connect_args =
= {
{
'host'
'host':
: '130.61.38.177'
'130.61.38.177',
,
'port'
'port':
: 6448
6448,
,
'user'
'user':
: 'piday'
'piday',
,
'password'
'password':
: 'Fred140376#'
'Fred140376#',
,
'ssl-mode'
'ssl-mode':
: 'DISABLED'
'DISABLED',
,
}
};
;
session
session =
= mysqlx
mysqlx.
.get_session
get_session(
(**
**connect_args
connect_args)
)
42

The code : /opt/PiDayMySQL/metrics_to_MySQL.py
import
import sys
sys
import
import time
time
import
import urllib
urllib.
.request
request
from
from datetime
datetime import
import datetime
datetime
import
import Adafruit_DHT
Adafruit_DHT
import
import mysqlx
mysqlx
rasbpy_id
rasbpy_id =
= "1"
"1"
if
if len
len(
(sys
sys.
.argv
argv)
) >
> 1
1:
:
rasbpy_id
rasbpy_id =
= str
str(
(sys
sys.
.argv
argv[
[1
1]
])
)
sensor
sensor =
= Adafruit_DHT
Adafruit_DHT.
.DHT22
DHT22
gpio
gpio =
= 4
4
43

The code : /opt/PiDayMySQL/metrics_to_MySQL.py (2)
connect_args
connect_args =
= {
{
'host'
'host':
: '130.61.46.6'
'130.61.46.6',
,
'port'
'port':
: 6448
6448,
,
'user'
'user':
: 'piday'
'piday',
,
'password'
'password':
: 'Fred140376#'
'Fred140376#',
,
'ssl-mode'
'ssl-mode':
: 'DISABLED'
'DISABLED',
,
'schema'
'schema':
: 'piday'
'piday'
}
};
;
44

def
def connect
connect(
()
):
:
global
global external_ip
external_ip
global
global session
session
global
global schema
schema
global
global tbl_temperature_history
tbl_temperature_history,
, tbl_humidity_history
tbl_humidity_history,
, tbl_publicip_history
tbl_publicip_history
global
global col_devices
col_devices
external_ip
external_ip =
= urllib
urllib.
.request
request.
.urlopen
urlopen(
(
'https://api.ipify.org'
'https://api.ipify.org')
).
.read
read(
()
).
.decode
decode(
('utf8'
'utf8')
)
session
session =
= mysqlx
mysqlx.
.get_session
get_session(
(**
**connect_args
connect_args)
)
schema
schema =
= session
session.
.get_schema
get_schema(
('piday'
'piday')
)
tbl_temperature_history
tbl_temperature_history =
= schema
schema.
.get_table
get_table(
('temperature_history'
'temperature_history')
)
tbl_humidity_history
tbl_humidity_history =
= schema
schema.
.get_table
get_table(
('humidity_history'
'humidity_history')
)
tbl_publicip_history =
= schema
schema.
.get_table
get_table(
('publicip_history'
'publicip_history')
)
col_devices
col_devices =
= schema
schema.
.get_collection
get_collection(
('devices'
'devices')
)
connect
connect(
()
)
45

while
while(
(True
True)
):
:
if
if not
not session
session.
.is_open
is_open(
()
):
:
connect
connect(
()
)
humidity
humidity,
, temperature
temperature =
= Adafruit_DHT
Adafruit_DHT.
.read_retry
read_retry(
(sensor
sensor,
, gpio
gpio)
)
if
if humidity
humidity is
is not
not None
None and
and temperature
temperature is
is not
not None
None:
:
# Find the device and its last ip
# Find the device and its last ip
res
res =
= col_devices
col_devices.
.find
find(
("_id='{}'"
"_id='{}'".
.format
format(
(rasbpy_id
rasbpy_id)
))
).
.fields
fields(
(
'_id'
'_id',
, 'publicip'
'publicip')
).
.execute
execute(
()
)
device
device =
= res
res.
.fetch_one
fetch_one(
()
)
if
if device
device:
:
old_public_ip
old_public_ip =
= device
device[
['publicip'
'publicip']
]
else
else:
:
print
print(
("device with _id: {} not found !"
"device with _id: {} not found !".
.format
format(
(rasbpy_id
rasbpy_id)
))
)
print
print(
("sleeping 1 min..."
"sleeping 1 min...")
)
time
time.
.sleep
sleep(
(60
60)
)
continue
continue
46

patch_json
patch_json =
= {
{}
}
humidity_s
humidity_s =
= "{0:0.1f}%"
"{0:0.1f}%".
.format
format(
(humidity
humidity)
)
temperature_s
temperature_s =
= "{0:0.1f}C"
"{0:0.1f}C".
.format
format(
(temperature
temperature)
)
patch_json
patch_json[
['humidity'
'humidity']
] =
= humidity_s
humidity_s
patch_json
patch_json[
['temperature'
'temperature']
] =
= temperature_s
temperature_s
patch_json
patch_json[
['publicip'
'publicip']
] =
= external_ip
external_ip
patch_json
patch_json[
['last_update'
'last_update']
] =
= datetime
datetime.
.now
now(
()
).
.strftime
strftime(
(
"%Y-%m-%d %H:%M:%S"
"%Y-%m-%d %H:%M:%S")
)
print
print(
('Temp={} Humidity={}'
'Temp={} Humidity={}'.
.format
format(
(temperature_s
temperature_s,
, humidity_s
humidity_s)
))
)
col_devices
col_devices.
.modify
modify(
("_id='{}'"
"_id='{}'".
.format
format(
(rasbpy_id
rasbpy_id)
)
)
).
.patch
patch(
(patch_json
patch_json)
).
.execute
execute(
()
)
if
if old_public_ip
old_public_ip !=
!= external_ip
external_ip:
:
tbl_publicip_history.
.insert
insert(
([
['device_id'
'device_id',
, 'ip_address'
'ip_address']
])
).
.values
values(
(
rasbpy_id
rasbpy_id,
, external_ip
external_ip)
).
.execute
execute(
()
)
47

tbl_temperature_history
tbl_temperature_history.
.insert
insert(
([
['time_stamp'
'time_stamp',
, 'device_id'
'device_id',
, 'value'
'value']
])
).
.values
values(
(
datetime
datetime.
.now
now(
()
).
.strftime
strftime(
("%Y-%m-%d %H:%M:%S"
"%Y-%m-%d %H:%M:%S")
),
,
rasbpy_id
rasbpy_id,
, temperature
temperature)
).
.execute
execute(
()
)
tbl_humidity_history
tbl_humidity_history.
.insert
insert(
([
['time_stamp'
'time_stamp',
, 'device_id'
'device_id',
, 'value'
'value']
])
).
.values
values(
(
datetime
datetime.
.now
now(
()
).
.strftime
strftime(
("%Y-%m-%d %H:%M:%S"
"%Y-%m-%d %H:%M:%S")
),
,
rasbpy_id
rasbpy_id,
, humidity
humidity)
).
.execute
execute(
()
)
else
else:
:
print
print(
('Failed to get reading. Try again!'
'Failed to get reading. Try again!')
)
time
time.
.sleep
sleep(
(1
1)
)
48

Systemd script
/etc/systemd/system/piday_metrics@.service
[Unit]
Description=Sending DHT22 metrics from Raspberry Pi to MySQL Database Service
After=network.target
[Service]
Type=simple
User=root
Restart=on-failure
ExecStart=/usr/bin/python /opt/PiDayMySQL/metrics_to_MySQL.py %s
[Install]
WantedBy=multi-user.target
49

Testing
running our script
50

Start the service on the Raspberry Pi
[
[raspberry ~
raspberry ~]
]$
$ sudo
sudo systemctl start piday_metrics@00006227543c0000000000000001
systemctl start piday_metrics@00006227543c0000000000000001
51

Checking the data
52

Checking the data
without any single line of SQL ;-)
52

Improvements
using MQTT Broker
53

OCI Architectures - improved (1)
54

OCI Architectures - improved (2)
55

HeatWave
MySQL on Steroid
56

Analyzing the collected data
Let's get some statistics on the metrics we collected so far:
SQL
SQL >
> select
select *
* from
from (
(select
select device_id
device_id,
, max
max(
(value
value)
) as
as `
`max temp
max temp`
`,
, min
min(
(value
value)
) as
as `
`min temp
min temp`
`,
,
avg
avg(
(value
value)
) as
as `
`avg temp
avg temp`
` from
from temperature_history
temperature_history group
group by
by device_id
device_id)
) a
a
natural
natural join
join (
(select
select device_id
device_id,
, max
max(
(value
value)
) as
as `
`max humidity
max humidity`
`,
, min
min(
(value
value)
) as
as `
`min humidity
min humidity`
`,
,
avg
avg(
(value
value)
) as
as `
`avg humidity
avg humidity`
` from
from humidity_history
humidity_history group
group by
by device_id
device_id)
) b
b ;
;
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
|
| device_id
device_id |
| max
max temp
temp |
| min
min temp
temp |
| avg
avg temp
temp |
| max humidity
max humidity |
| min humidity
min humidity |
| avg humidity
avg humidity |
|
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
|
| 00006227543
00006227543c0000000000000001
c0000000000000001 |
| 142.94
142.94 |
| 9.00
9.00 |
| 11.711163
11.711163 |
| 169.86
169.86 |
| 40.00
40.00 |
| 45.206324
45.206324 |
|
|
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| 29.99
29.99 |
| 20.00
20.00 |
| 22.513379
22.513379 |
| 65.00
65.00 |
| 55.00
55.00 |
| 60.000385
60.000385 |
|
|
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| 814.36
814.36 |
| 0.00
0.00 |
| 3.114047
3.114047 |
| 800.21
800.21 |
| 1.00
1.00 |
| 6.609432
6.609432 |
|
|
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| 288.44
288.44 |
| 10.00
10.00 |
| 12.567651
12.567651 |
| 279.32
279.32 |
| 30.00
30.00 |
| 35.062746
35.062746 |
|
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
4
4 rows
rows in
in set
set (
(2.0747
2.0747 sec
sec)
)
57

Get the Query Execution Plan
Let's have a look at the EXPLAIN plan (format=tree):
EXPLAIN
EXPLAIN:
: -
->
> Nested
Nested loop
loop inner
inner join
join (
(cost
cost=
=1348953.36
1348953.36 rows
rows=
=0
0)
)
-
->
> Filter:
Filter: (
(a
a.
.device_id
device_id is
is not
not null
null)
) (
(cost
cost=
=155455.11
155455.11.
..58090
.58090.86
.86 rows
rows=
=516341
516341)
)
-
->
> Table
Table scan
scan on
on a
a (
(cost
cost=
=0.01
0.01.
..6456
.6456.76
.76 rows
rows=
=516341
516341)
)
-
->
> Materialize
Materialize (
(cost
cost=
=155455.31
155455.31.
..161912
.161912.06
.06 rows
rows=
=516341
516341)
)
-
->
> Group
Group aggregate:
aggregate: max
max(
(temperature_history
temperature_history.
.`
`value
value`
`)
),
, min
min(
.`
`value
value`
`)
),
,
avg
avg(
.`
`value
value`
`)
) (
(cost
cost=
=103821.20
103821.20 rows
rows=
=516341
516341)
)
-
->
> Index
Index scan
scan on
on temperature_history
temperature_history using
using device_id_idx
device_id_idx (
(cost
cost=
=52187.10
52187.10 rows
rows=
=516341
516341)
)
-
->
> Index
Index lookup
lookup on
on b
b using
using <
<auto_key0
auto_key0>
> (
(device_id
device_id=
=a
a.
.device_id
device_id)
)
-
->
> Materialize
Materialize (
(cost
cost=
=0.00
0.00.
..0
.0.00
.00 rows
rows=
=0
0)
)
-
->
> Table
Table scan
scan on
on <
<temporary
temporary>
>
-
->
> Aggregate
Aggregate using
using temporary
temporary table
table
-
->
> Table
Table scan
scan on
on humidity_history
humidity_history (
(cost
cost=
=52191.40
52191.40 rows
rows=
=516384
516384)
)
58

Analyzing the collected data (2)
SQL
SQL >
> select
select *
* from
from (
(
select
select date
date(
(time_stamp
time_stamp)
) as
as `
`day
day`
`,
, device_id
device_id,
, count
count(
(*
*)
) as
as `
`tot
tot`
`,
,
max
max(
(value
value)
) as
as `
`max hum
max hum`
`,
, min
min(
(value
value)
) as
as `
`min hum
min hum`
`,
, avg
avg(
(value
value)
) as
as `
`avg hum
avg hum`
`
from
group by
by device_id
device_id,
, day
day)
) a
a
natural
natural join
join (
(
select
select date
date(
(time_stamp
time_stamp)
) as
as `
`day
day`
`,
, device_id
device_id,
, count
count(
(*
*)
) as
as `
`tot
tot`
`,
,
max
max(
(value
value)
) as
as `
`max temp
max temp`
`,
, min
min(
(value
value)
) as
as `
`min temp
min temp`
`,
, avg
avg(
(value
value)
) as
as `
`avg temp
avg temp`
`
from
group by
by device_id
device_id,
, day
day)
) b
b order
order by
by day
day,
, device_id
device_id;
;
+
+------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
|
| day
day |
| device_id
device_id |
| tot
tot |
| max hum
max hum |
| min hum
min hum |
| avg hum
avg hum |
| max
max temp
temp |
| min
min temp
temp |
| avg
avg temp
temp |
|
+
+------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
|
| 2022
2022-
-03
03-
-09
09 |
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| 14534
14534 |
| 65.00
65.00 |
| 55.00
55.00 |
| 60.009273
60.009273 |
| 29.99
29.99 |
| 20.00
20.00 |
| 22.597118
22.597118 |
|
|
| 2022
2022-
-03
03-
-09
09 |
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| 31605
31605 |
| 800.21
800.21 |
| 1.00
1.00 |
| 8.570861
8.570861 |
| 814.36
814.36 |
| 0.00
0.00 |
| 5.079733
5.079733 |
|
|
| 2022
2022-
-03
03-
-09
09 |
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| 31284
31284 |
| 279.32
279.32 |
| 30.00
30.00 |
| 35.294440
35.294440 |
| 288.44
288.44 |
| 10.00
10.00 |
| 12.797445
12.797445 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000001
c0000000000000001 |
| 114906
114906 |
| 50.00
50.00 |
| 40.00
40.00 |
| 45.001613
45.001613 |
| 14.00
14.00 |
| 9.00
9.00 |
| 11.499796
11.499796 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| 100913
100913 |
| 65.00
65.00 |
| 55.00
55.00 |
| 59.999105
59.999105 |
| 25.00
25.00 |
| 20.00
20.00 |
| 22.501319
22.501319 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| 101465
101465 |
| 11.00
11.00 |
| 1.00
1.00 |
| 5.998472
5.998472 |
| 5.00
5.00 |
| 0.00
0.00 |
| 2.501763
2.501763 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| 101044
101044 |
| 40.00
40.00 |
| 30.00
30.00 |
| 34.991012
34.991012 |
| 15.00
15.00 |
| 10.00
10.00 |
| 12.496505
12.496505 |
|
+
+------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
7
7 rows
rows in
in set
set (
(1.2717
1.2717 sec
sec)
)
59

Get the Query Execution Plan (2)
EXPLAIN
EXPLAIN:
: -
->
> Sort: a
Sort: a.
.`
`day
day`
`,
, a
a.
.device_id
device_id
-
->
> Stream results
Stream results (
(cost
cost=
=1374880.15
1374880.15 rows
rows=
=0
0)
)
-
->
> Nested
Nested loop
loop inner
inner join
join (
(cost
cost=
=1374880.15
1374880.15 rows
rows=
=0
0)
)
-
->
> Filter:
Filter: (
((
(b
b.
.`
`day
day`
` is
is not
not null
null)
) and
and (
(b
b.
.device_id
device_id is
is not
not null
null)
))
) (
(cost
cost=
=0.11
0.11.
..59207
.59207.65
.65 rows
rows=
=526268
526268)
)
-
->
> Table
Table scan
scan on
on b
b (
(cost
cost=
=2.50
2.50.
..2
.2.50
.50 rows
rows=
=0
0)
)
-
->
> Materialize
Materialize (
(cost
cost=
=2.50
2.50.
..2
.2.50
.50 rows
rows=
=0
0)
)
-
->
> Table
Table scan
scan on
on <
<temporary
temporary>
>
-
->
> Aggregate
Aggregate using
using temporary
temporary table
table
-
->
> Table
Table scan
scan on
on temperature_history
temperature_history (
(cost
cost=
=53179.80
53179.80 rows
rows=
=526268
526268)
)
-
->
> Index
Index lookup
lookup on
on a
a using
using <
<auto_key0
auto_key0>
> (
(day
day=
=b
b.
.`
`day
day`
`,
, device_id
device_id=
=b
b.
.device_id
device_id,
, tot
tot=
=b
b.
.tot
tot)
)
-
->
> Materialize
Materialize (
(cost
cost=
=0.00
0.00.
..0
.0.00
.00 rows
rows=
=0
0)
)
-
->
> Table
Table scan
scan on
on <
<temporary
temporary>
>
-
->
> Aggregate
Aggregate using
using temporary
temporary table
table
-
->
> Table
Table scan
scan on
on humidity_history
humidity_history (
(cost
cost=
=53184.10
53184.10 rows
rows=
=526311
526311)
)
60

Enable HeatWave
Let's boost MySQL's performance with HeatWave !
We have two options to enable HeatWave:
. Editing the Stack (if we used OCI Resoure Manager)
. Using MySQL Database Service's console
61

Enable HeatWave - with Stack
62

Enable HeatWave - with Stack (2)
63

Enable HeatWave - with Stack (3)
64

Enable HeatWave - with MDS Console
65

Enable HeatWave - with MDS Console (2)
66

67

68

SQL
SQL >
> show
show global
global status
status like
like 'rapid_%er%_status'
'rapid_%er%_status';
;
+
+----------------------+--------+
----------------------+--------+
|
| Variable_name
Variable_name |
| Value
Value |
|
+
+----------------------+--------+
----------------------+--------+
|
| rapid_cluster_status
rapid_cluster_status |
| ON
ON |
|
|
| rapid_service_status
rapid_service_status |
| ONLINE
ONLINE |
|
+
+----------------------+--------+
----------------------+--------+
HeatWave Ready
69

Load Data to HeatWave
We now use the load command printed during the Estimate Node Count step:
SQL
SQL >
> CALL
CALL sys
sys.
.heatwave_load
heatwave_load(
(JSON_ARRAY
JSON_ARRAY(
('piday'
'piday')
),
, NULL
NULL)
);
;
70

Load Data to HeatWave (2)
This operation returns a report:
.
..
..
.
+
+-------------------------------------------------------------------------------+
-------------------------------------------------------------------------------+
|
| LOAD
LOAD SUMMARY
SUMMARY |
|
+
+-------------------------------------------------------------------------------+
-------------------------------------------------------------------------------+
|
| |
|
|
| SCHEMA
SCHEMA TABLES
TABLES TABLES
TABLES COLUMNS
COLUMNS LOAD
LOAD |
|
|
| NAME LOADED FAILED LOADED DURATION
NAME LOADED FAILED LOADED DURATION |
|
|
| ------ ------ ------ ------- -------- |
------ ------ ------ ------- -------- |
|
| `
`piday
piday`
` 3
3 0
0 12
12 1.84
1.84 s
s |
|
|
| |
|
+
+-------------------------------------------------------------------------------+
-------------------------------------------------------------------------------+
6
6 rows
rows in
in set
set (
(1.9237
1.9237 sec
sec)
)
71

... and now ?
SQL
SQL >
> select
select *
* from
from (
(select
select device_id
device_id,
, max
max(
(value
value)
) as
as `
`max temp
max temp`
`,
, min
min(
(value
value)
) as
as `
`min temp
min temp`
`,
,
avg
avg(
(value
value)
) as
as `
`avg temp
avg temp`
` from
group by
by device_id
device_id)
) a
a
natural
natural join
join (
(select
select device_id
device_id,
, max
max(
(value
value)
) as
as `
`max humidity
max humidity`
`,
, min
min(
(value
value)
) as
as `
`min humidity
min humidity`
`,
,
avg
avg(
(value
value)
) as
as `
`avg humidity
avg humidity`
` from
group by
by device_id
device_id)
) b
b ;
;
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
|
| device_id
device_id |
| max
max temp
temp |
| min
min temp
temp |
| avg
avg temp
temp |
| max humidity
max humidity |
| min humidity
min humidity |
| avg humidity
avg humidity |
|
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
|
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| 814.36
814.36 |
| 0.00
0.00 |
| 3.114047
3.114047 |
| 800.21
800.21 |
| 1.00
1.00 |
| 6.609431
6.609431 |
|
|
| 00006227543
00006227543c0000000000000001
c0000000000000001 |
| 142.94
142.94 |
| 9.00
9.00 |
| 11.708011
11.708011 |
| 169.86
169.86 |
| 40.00
40.00 |
| 45.203256
45.203256 |
|
|
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| 29.99
29.99 |
| 20.00
20.00 |
| 22.513379
22.513379 |
| 65.00
65.00 |
| 55.00
55.00 |
| 60.000384
60.000384 |
|
|
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| 288.44
288.44 |
| 10.00
10.00 |
| 12.567650
12.567650 |
| 279.32
279.32 |
| 30.00
30.00 |
| 35.062746
35.062746 |
|
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
4
4 rows
rows in
in set
set (
(0.1049
0.1049 sec
sec)
)
72

... and now ?
SQL
SQL >
> select
select *
* from
from (
(select
select device_id
device_id,
, max
max(
(value
value)
) as
as `
`max temp
max temp`
`,
, min
min(
(value
value)
) as
as `
`min temp
min temp`
`,
,
avg
avg(
(value
value)
) as
as `
`avg temp
avg temp`
` from
group by
by device_id
device_id)
) a
a
natural
natural join
join (
(select
select device_id
device_id,
, max
max(
(value
value)
) as
as `
`max humidity
max humidity`
`,
, min
min(
(value
value)
) as
as `
`min humidity
min humidity`
`,
,
avg
avg(
(value
value)
) as
as `
`avg humidity
avg humidity`
` from
group by
by device_id
device_id)
) b
b ;
;
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
|
| device_id
device_id |
| max
max temp
temp |
| min
min temp
temp |
| avg
avg temp
temp |
| max humidity
max humidity |
| min humidity
min humidity |
| avg humidity
avg humidity |
|
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
|
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| 814.36
814.36 |
| 0.00
0.00 |
| 3.114047
3.114047 |
| 800.21
800.21 |
| 1.00
1.00 |
| 6.609431
6.609431 |
|
|
| 00006227543
00006227543c0000000000000001
c0000000000000001 |
| 142.94
142.94 |
| 9.00
9.00 |
| 11.708011
11.708011 |
| 169.86
169.86 |
| 40.00
40.00 |
| 45.203256
45.203256 |
|
|
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| 29.99
29.99 |
| 20.00
20.00 |
| 22.513379
22.513379 |
| 65.00
65.00 |
| 55.00
55.00 |
| 60.000384
60.000384 |
|
|
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| 288.44
288.44 |
| 10.00
10.00 |
| 12.567650
12.567650 |
| 279.32
279.32 |
| 30.00
30.00 |
| 35.062746
35.062746 |
|
+
+------------------------------+----------+----------+-----------+--------------+--------------+--------------+
------------------------------+----------+----------+-----------+--------------+--------------+--------------+
4
4 rows
rows in
in set
set (
(0.1049
0.1049 sec
sec)
)
from 2.0747 sec to 0.1049 sec !!
20x faster !
72

the second example:
SQL
SQL >
> select
select *
* from
from (
(
select
select date
date(
(time_stamp
time_stamp)
) as
as `
`day
day`
`,
, device_id
device_id,
, count
count(
(*
*)
) as
as `
`tot
tot`
`,
,
max
max(
(value
value)
) as
as `
`max hum
max hum`
`,
, min
min(
(value
value)
) as
as `
`min hum
min hum`
`,
, avg
avg(
(value
value)
) as
as `
`avg hum
avg hum`
`
from
group by
by device_id
device_id,
, day
day)
) a
a
natural
natural join
join (
(
select
select date
date(
(time_stamp
time_stamp)
) as
as `
`day
day`
`,
, device_id
device_id,
, count
count(
(*
*)
) as
as `
`tot
tot`
`,
,
max
max(
(value
value)
) as
as `
`max temp
max temp`
`,
, min
min(
(value
value)
) as
as `
`min temp
min temp`
`,
, avg
avg(
(value
value)
) as
as `
`avg temp
avg temp`
`
from
group by
by device_id
device_id,
, day
day)
) b
b order
order by
by day
day,
, device_id
device_id;
;
+
+------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
|
| day
day |
| device_id
device_id |
| tot
tot |
| max hum
max hum |
| min hum
min hum |
| avg hum
avg hum |
| max
max temp
temp |
| min
min temp
temp |
| avg
avg temp
temp |
|
+
+------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
|
| 2022
2022-
-03
03-
-09
09 |
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| 14534
14534 |
| 65.00
65.00 |
| 55.00
55.00 |
| 60.009272
60.009272 |
| 29.99
29.99 |
| 20.00
20.00 |
| 22.597117
22.597117 |
|
|
| 2022
2022-
-03
03-
-09
09 |
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| 31605
31605 |
| 800.21
800.21 |
| 1.00
1.00 |
| 8.570860
8.570860 |
| 814.36
814.36 |
| 0.00
0.00 |
| 5.079732
5.079732 |
|
|
| 2022
2022-
-03
03-
-09
09 |
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| 31284
31284 |
| 279.32
279.32 |
| 30.00
30.00 |
| 35.294440
35.294440 |
| 288.44
288.44 |
| 10.00
10.00 |
| 12.797445
12.797445 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000001
c0000000000000001 |
| 115609
115609 |
| 50.00
50.00 |
| 40.00
40.00 |
| 45.001736
45.001736 |
| 14.00
14.00 |
| 9.00
9.00 |
| 11.499157
11.499157 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| 100913
100913 |
| 65.00
65.00 |
| 55.00
55.00 |
| 59.999104
59.999104 |
| 25.00
25.00 |
| 20.00
20.00 |
| 22.501318
22.501318 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| 101465
101465 |
| 11.00
11.00 |
| 1.00
1.00 |
| 5.998472
5.998472 |
| 5.00
5.00 |
| 0.00
0.00 |
| 2.501762
2.501762 |
|
|
| 2022
2022-
-03
03-
-10
10 |
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| 101044
101044 |
| 40.00
40.00 |
| 30.00
30.00 |
| 34.991011
34.991011 |
| 15.00
15.00 |
| 10.00
10.00 |
| 12.496504
12.496504 |
|
+
+------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
------------+------------------------------+--------+---------+---------+-----------+----------+----------+-----------+
7
7 rows
rows in
in set
set (
(0.1267
0.1267 sec
sec)
)
73

More Data : more di erence !
Now let's use again the second example with more data (7 days):
Without HeatWave:
90
90 rows
rows in
in set
set (
(5
5 min
min 3.3237
3.3237 sec
sec)
)
74

Now let's use again the second example with more data (7 days):
Without HeatWave:
90
90 rows
rows in
in set
set (
(5
5 min
min 3.3237
3.3237 sec
sec)
)
With HeatWave:
90
90 rows
rows in
in set
set (
(0.8620
0.8620 sec
sec)
)
from 363.3237 sec to 0.8620 sec ! 421x faster !!
74

And this is with 5GB to parse:
SQL
SQL>
> select
select format_bytes
format_bytes(
(sum
sum(
(data_length
data_length)
))
) DATA
DATA,
,
format_bytes
format_bytes(
(sum
sum(
(index_length
index_length)
))
) INDEXES
INDEXES,
,
format_bytes
format_bytes(
(sum
sum(
(data_length
data_length +
+ index_length
index_length)
))
) 'TOTAL SIZE'
'TOTAL SIZE'
from
from information_schema
information_schema.
.TABLES
TABLES order
order by
by data_length
data_length +
+ index_length
index_length;
;
+
+----------+----------+------------+
----------+----------+------------+
|
| DATA
DATA |
| INDEXES
INDEXES |
| TOTAL SIZE
TOTAL SIZE |
|
+
+----------+----------+------------+
----------+----------+------------+
|
| 4.23
4.23 GiB
GiB |
| 1.37
1.37 GiB
GiB |
| 5.60
5.60 GiB
GiB |
|
+
+----------+----------+------------+
----------+----------+------------+
75

Without HeatWave:
44
44 rows
rows in
in set
set (
(10
10 min
min 14.1022
14.1022 sec
sec)
)
With HeatWave:
45
45 rows
rows in
in set
set (
(1.6051
1.6051 sec
sec)
)
Let's double the data:
+
+----------+----------+------------+
----------+----------+------------+
|
| DATA
DATA |
| INDEXES
INDEXES |
| TOTAL SIZE
TOTAL SIZE |
|
+
+----------+----------+------------+
----------+----------+------------+
|
| 8.66
8.66 GiB
GiB |
| 2.93
2.93 GiB
GiB |
| 11.59
11.59 GiB
GiB |
|
+
+----------+----------+------------+
----------+----------+------------+
69M ROWS
76

MySQL & HeatWave for your IoT Projects
77

MySQL & HeatWave
When having to deal with large amount of data for Analytics, this is the traditional
architecture:
78

MySQL & HeatWave - no more ETL required
Single MySQL database for OLTP & Analytics applications
All existing applications work without any changes
Extreme performance
79

MySQL & HeatWave - one solution !
80

Using HeatWave ?
SQL
SQL>
> show
show global
global status
status like
like 'rapid_query_offload_count'
'rapid_query_offload_count';
;
+
+---------------------------+-------+
---------------------------+-------+
|
| Variable_name
Variable_name |
| Value
Value |
|
+
+---------------------------+-------+
---------------------------+-------+
|
| rapid_query_offload_count
rapid_query_offload_count |
| 17
17 |
|
+
+---------------------------+-------+
---------------------------+-------+
The EXPLAIN command for a query will have the following value for Extra when using
HeatWave:
Using secondary engine RAPID
Using secondary engine RAPID
81

More with Autopilot Machine Learning
HeatWave includes several Autopilot processes, the following ones are avaible from the
SQL prompt when HeatWave is enabled:
Autopilot Encoding: Determines the optimal representation of columns loaded into
HeatWave by analyzing HeatWave query history, which improves query performance
and minimizes the required cluster size.
Autopilot Placement: Recommends how tables should be partitioned in memory to
achieve the best query performance, and estimates the expected performance
improvement
More Info: h ps://dev.mysql.com/doc/heatwave/en/heatwave-autopilot.html
82

More with Autopilot Machine Learning (2)
Autopilot Encoding:
CALL
CALL sys
sys.
.heatwave_advisor
heatwave_advisor(
(JSON_OBJECT
JSON_OBJECT(
("auto_enc"
"auto_enc",
,JSON_OBJECT
JSON_OBJECT(
("mode"
"mode",
, "recommend"
"recommend")
))
))
);
;
Autopilot Placement:
CALL
CALL sys
sys.
.heatwave_advisor
heatwave_advisor(
(JSON_OBJECT
JSON_OBJECT(
("target_schema"
"target_schema",
, JSON_ARRAY
JSON_ARRAY(
("piday"
"piday")
))
))
);
;
83

HeatWave and Oracle Analytics Cloud
fast realtime analytics
84

Comprehensive cloud-native analytics
platform
Data visualization, data preparation,
data quality and data Flows
ML and AI embedded throughout
Natural language processing and
generation
Self upgrading, self patching, self
securing
Oracle Analytics Cloud
Complete Analytics Platform for Business Users, Developers and IT
85

Oracle Analytics Platform
Delivering at every stage of the analytics journey
86

For Data Connectivity, OAC supports…
87

From Multiple Native Connectors including
MySQL !
Creating a new connection to our MySQL
database instance:
Using Oracle Analytics Cloud with MySQL HeatWave
88

89

For Data Modeling, OAC supports...
90

Now we can create our dataset:
91

We search and select our connection
(piday):
92

93

94

We join using device_id eld:
95

96

Don't forget to specify that we want live data !
97

Retrieving data from a JSON document in a collection.
Let's return the device_id as VARCHAR and its short_name using VIRTUAL COLUMNS:
SQL
SQL>
> alter
alter table
table devices
devices add
add device_id
device_id varchar
varchar(
(32
32)
)
generated always
generated always as
as (
(doc
doc-
->>
>>'$._id'
'$._id')
)
SQL
SQL>
> alter
alter table
table devices
devices add
add short_name
short_name varchar
varchar(
(20
20)
)
generated always
generated always as
as (
(doc
doc-
->>
>>'$.short_name'
'$.short_name')
);
;
98

We can now retrieved them (and index them if needed) without having to use the special
JSON notation:
SQL
SQL>
> select
select device_id
device_id,
, short_name
short_name from
from devices
devices;
;
+
+------------------------------+------------+
------------------------------+------------+
|
| device_id
device_id |
| short_name
short_name |
|
+
+------------------------------+------------+
------------------------------+------------+
|
| 00006227543
00006227543c0000000000000001
c0000000000000001 |
| lefredPi
lefredPi |
|
|
| 00006227543
00006227543c0000000000000002
c0000000000000002 |
| fake01
fake01 |
|
|
| 00006227543
00006227543c0000000000000003
c0000000000000003 |
| fake02
fake02 |
|
|
| 00006227543
00006227543c0000000000000004
c0000000000000004 |
| fake03
fake03 |
|
+
+------------------------------+------------+
------------------------------+------------+
4
4 rows
rows in
in set
set (
(0.0284
0.0284 sec
sec)
)
99

100

101

102

103

For Data Preparation, OAC supports...
104

For Data Exploration, OAC supports...
105

For Data Visualization, OAC supports...
106

For Data Visualization, OAC supports... (2)
As a developer you could even check the performance parameters of your visualizations:
107

For Advances Analytics, OCA supports...
108

For Citizen Data Science, OAC supports...
109

For Collaboration, OAC Supports...
110

Easy creation of real-time dashboard of all the data collected by the Raspberry Pi
111

Summary
our journey from IoT device to HeatWave and Analyitcs
112

Summary of our journey
113

We started from a simple Raspberry Pi
113

We collected a lot of data
113

We ran queries super fast and were ready for Analitycs without any ETL service
113

We used the limitless OAC capabilities to process, vizualize and forecast the data
113

We used the limitless OAC capabilities to process, vizualize and forecast the data
and all this thanks to MySQL HeatWave !
113

Want more
Swag ?
Download a free trial
and get bonus swag on top of the Pi Day swag kit!
114

Once your email is whitelisted, go to bit.ly/piday-freetrial and click the Sign Up bu on
115

Click Verify my email bu on
Enter your Country/Territory, First Name, Last Name, and Email
Check the box if you are a human and complete the Captcha authentication
116

Click the Verify Email bu on
You will be presented with an o er for $500 in free credits, click the Select O er bu on
Go to the email account that was used to register for this event and look for the Oracle
Cloud email asking to verify the email address and create your Oracle account.
117

Enter required elds under Account
information
Read the Terms of Use and click the
Continue bu on
118

Enter your Address and Phone Number
Click the Continue bu on
Read the terms and conditions and
acknowledge the Agreement.
Click the Start my free trial bu on
119

Your Oracle Cloud Infrastructure (OCI) tenancy is now being provisioned.
120

Welcome to your Oracle Cloud Infrastructure (OCI) tenancy
121

Oracle Cloud Free Tier
bit.ly/piday-freetrial
122

Share our event!
#piday
#odevrel
#mysql
Join our slack channels!
bit.ly/odevrel-slack
bit.ly/mysql-slack
123

www.oracle.com/events/run-workloads-your-way-on-oci/
124

h ps://bit.ly/Pi-Quiz-2022
Test your Pi knowledge. ¶'s the limit.
125

Questions ?
126

Resources
h ps://github.com/lefred/oci-iot-mds
h ps://github.com/lefred/oracle-piday-2022
h ps://dev.mysql.com/doc/heatwave/en/
h ps://www.mysql.com/cloud/
h ps://www.oracle.com/in/business-analytics/data-visualization/examples.html
h ps://www.oracle.com/in/business-analytics
127

Resources (2)
Try a Live Lab: MySQL Database Service powered by HeatWave and Oracle Analytics Cloud:
h ps://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/workshop-a endee-2?
p210_workshop_id=870&p210_type=1&session=2731606019312
128

Pi Day 2022 - from IoT to MySQL HeatWave Database Service

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Pi Day 2022 - from IoT to MySQL HeatWave Database Service

Ähnlich wie Pi Day 2022 - from IoT to MySQL HeatWave Database Service (20)

Mehr von Frederic Descamps

Mehr von Frederic Descamps (17)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Pi Day 2022 - from IoT to MySQL HeatWave Database Service