How to Find Patterns in Your Data with SQL

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
How to Find Patterns in Your Data with SQL
Chris Saxon, @ChrisRSaxon & @SQLDaily
blogs.oracle.com/sql
youtube.com/c/TheMagicofSQL
asktom.oracle.com

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

This presentation contains <regular expressions>!

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | blogs.oracle.com/sql www.youtube.com/c/TheMagicOfSQL @ChrisRSaxon
I thought
this was
about SQL!
Ryan McGuire / Gratisography

* => zero or more matches
+ => one or more matches
{n,m} => N through M matches
(either optional)

Am I Improving?
Can Beat My PB?
Am I Training Regularly?

Am I running
every day?

RUN_DATE TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1

01 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1
#1
#3
#2
#4

How I know if rows are consecutive?

current value = previous value + 1

lag ( run_date ) over
( order by run_date )
Get the previous row's date

RUN_DATE RN TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 1 420 1
02 Jan 2018 2 2,400 5
03 Jan 2018 3 4,932 10
06 Jan 2018 4 2,350 5
07 Jan 2018 5 410 1
10 Jan 2018 6 400 1
13 Jan 2018 7 2,300 5
14 Jan 2018 8 425 1
15 Jan 2018 9 422 1
consecutive =>
constant gap

RUN_DATE RN TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 1 420 1
02 Jan 2018 2 2,400 5
03 Jan 2018 3 4,932 10
06 Jan 2018 4 2,350 5
07 Jan 2018 5 410 1
10 Jan 2018 6 400 1
13 Jan 2018 7 2,300 5
14 Jan 2018 8 425 1
15 Jan 2018 9 422 1
-
-
-
-
-
-
-
-
-

RUN_DATE RN RUN_DATE - RN TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 1 31 Dec 2017 420 1
02 Jan 2018 2 31 Dec 2017 2,400 5
03 Jan 2018 3 31 Dec 2017 4,932 10
06 Jan 2018 4 02 Jan 2018 2,350 5
07 Jan 2018 5 02 Jan 2018 410 1
10 Jan 2018 6 04 Jan 2018 400 1
13 Jan 2018 7 06 Jan 2018 2,300 5
14 Jan 2018 8 06 Jan 2018 425 1
15 Jan 2018 9 06 Jan 2018 422 1
-
-
-
-
-
-
-
-
-

Tabibitosan Method

row_number ()
over ( order by run_date )

run_date -
row_number ()
over ( order by run_date ) grp

with grps as (
select run_date ,
run_date -
row_number ()
over ( order by run_date ) grp
from running_log r
)
select min ( run_date ), count (*)
from grps
group by grp

12c Pattern Matching

01 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1
this = prev + 1

01 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1
this = prev + 1
this = prev + 3

01 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1
this = prev + 1
this = prev + 3
this ≠ prev + 1

define
consecutive as
run_date = prev ( run_date ) + 1

pattern ( strt consecutive* )
define
consecutive as

define
consecutive as
"Always true"
> 0 matches
Any row <>

RUN_DATE VARIABLE TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 STRT 420 1
02 Jan 2018 CONSECUTIVE 2,400 5
03 Jan 2018 CONSECUTIVE 4,932 10
06 Jan 2018 STRT 2,350 5
07 Jan 2018 CONSECUTIVE 410 1
10 Jan 2018 STRT 400 1
13 Jan 2018 STRT 2,300 5

define
consecutive as
Which row is prev?!

order by run_date
define
consecutive as

match_recognize (
order by run_date
measures
first ( run_date ) as start_date,
count (*) as days
define
consecutive as
);
How many consecutive rows?
From first row in group

START_DATE DAYS
01 Jan 2018 3
06 Jan 2018 2
10 Jan 2018 1
13 Jan 2018 3

So which is
better?
Pixabay
pattern
matching

Performance ~ similar
Availability MR 12c vs 8i*
Readability personal pref
Match_recognize vs. Tabibitosan

Am I running >= 3
times/week?
Pixabay

How I know if runs are in the same week?

last Monday = prev last Monday

trunc ( run_date , 'iw' )
Return the start of the ISO week…
…Monday!

RUN_DATE TRUNC(RUN_DATE, 'IW') TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 01 Jan 2018 420 1
02 Jan 2018 01 Jan 2018 2,400 5
03 Jan 2018 01 Jan 2018 4,932 10
06 Jan 2018 01 Jan 2018 2,350 5
07 Jan 2018 01 Jan 2018 410 1
10 Jan 2018 08 Jan 2018 400 1
13 Jan 2018 08 Jan 2018 2,300 5
14 Jan 2018 08 Jan 2018 425 1
15 Jan 2018 15 Jan 2018 422 1

select trunc ( run_date , 'iw' ),
count(*)
from running_log
group by trunc ( run_date , 'iw' )

select trunc ( run_date , 'iw' ),
count(*)
from running_log
group by trunc ( run_date , 'iw' )
having count (*) >= 3

define
same_week as
trunc ( run_date, 'iw' ) =
prev ( trunc ( run_date, 'iw' ) )

match_recognize (
order by run_date
measures
count (*) as days
pattern ( strt same_week* )
define
same_week as
);

match_recognize (
order by run_date
measures
count (*) as days
pattern ( strt same_week {2, } )
define
same_week as
);
Two or more matches

match_recognize (
order by run_date
measures
count (*) as days
define
same_week as
);

match_recognize (
order by run_date
measures
count (*) as days
define
consecutive as
);

How I can I find consecutive weeks
with >= 3 runs?

Find weeks >= 3 runs
then
check these are consecutive!

with min_three_runs as (
select * from running_log
match_recognize (
order by run_date
measures
first ( trunc ( run_date, 'iw' ) )
as week_start,
count(*) as days
define
same_week as
)

How I can I find consecutive weeks?

current Monday = prev Monday + 7

WEEK_START RN WEEK_START – RN RUNS
01 Jan 2018 1 31 Dec 2017 5
08 Jan 2018 2 06 Jan 2018 3
15 Jan 2018 3 12 Jan 2018 4
05 Feb 2018 4 01 Feb 2018 3
12 Feb 2018 5 07 Feb 2018 3
05 Mar 2018 6 28 Feb 2018 3
12 Mar 2018 7 07 Mar 2018 4

WEEK_START RN * 7 RUNS
01 Jan 2018 7 5
08 Jan 2018 14 3
15 Jan 2018 21 4
05 Feb 2018 28 3
12 Feb 2018 35 3
05 Mar 2018 42 3
12 Mar 2018 49 4

WEEK_START RN * 7 WEEK_START – ( RN * 7 ) RUNS
01 Jan 2018 7 25 Dec 2017 5
08 Jan 2018 14 25 Dec 2017 3
15 Jan 2018 21 25 Dec 2017 4
05 Feb 2018 28 08 Jan 2018 3
12 Feb 2018 35 08 Jan 2018 3
05 Mar 2018 42 22 Jan 2018 3
12 Mar 2018 49 22 Jan 2018 4

select run_date - ( 7 *
row_number()
) grp
from min_three_runs

match_recognize (
order by week_start
measures
first ( week_start ) as start_date,
count (*) as weeks
pattern ( strt consecutive_weeks* )
define
consecutive_weeks as
week_start = prev ( week_start ) + 7
);

Pixabay
Am I running >= 3
times in 7 days?

current day < first day + 7

01 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1
#1
#2

01 Jan 2018 01 – 07 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 08 – 14 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 15 – 21 Jan 2018 422 1

01 Jan 2018 01 – 07 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 2,350 5
07 Jan 2018 410 1
10 Jan 2018 10 – 17 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1

select r.*, min ( run_date ) over (
order by run_date
) mn_date
from running_log r

RUN_DATE MN_DATE TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 01 Jan 2018 420 1
02 Jan 2018 01 Jan 2018 2,400 5
03 Jan 2018 01 Jan 2018 4,932 10
06 Jan 2018 01 Jan 2018 2,350 5
07 Jan 2018 01 Jan 2018 410 1
10 Jan 2018 01 Jan 2018 400 1
13 Jan 2018 01 Jan 2018 2,300 5
14 Jan 2018 01 Jan 2018 425 1
15 Jan 2018 01 Jan 2018 422 1

Windowing clause to the rescue…

select r.*, min ( run_date ) over (
order by run_date
range between 7 preceding
and current row
) mn_date
from running_log r
Consider rows with values
in the previous 7 days

01 Jan 2018 01 Jan 2018 420 1
02 Jan 2018 01 Jan 2018 2,400 5
03 Jan 2018 01 Jan 2018 4,932 10
06 Jan 2018 01 Jan 2018 2,350 5
07 Jan 2018 01 Jan 2018 410 1
10 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1

01 Jan 2018 01 Jan 2018 420 1
02 Jan 2018 01 Jan 2018 2,400 5
03 Jan 2018 01 Jan 2018 4,932 10
06 Jan 2018 01 Jan 2018 2,350 5
07 Jan 2018 01 Jan 2018 410 1
10 Jan 2018 03 Jan 2018 400 1
13 Jan 2018 2,300 5
14 Jan 2018 425 1
15 Jan 2018 422 1

01 Jan 2018 01 Jan 2018 420 1
02 Jan 2018 01 Jan 2018 2,400 5
03 Jan 2018 01 Jan 2018 4,932 10
06 Jan 2018 01 Jan 2018 2,350 5
07 Jan 2018 01 Jan 2018 410 1
10 Jan 2018 03 Jan 2018 400 1
13 Jan 2018 06 Jan 2018 2,300 5
14 Jan 2018 07 Jan 2018 425 1
15 Jan 2018 10 Jan 2018 422 1

Windowing clause to the rescue!

11.2 Recursive With

with rws as (
select r.*, row_number() over ( order by run_date ) rn
from running_log r
)

with rws as (
from running_log r
), within_7 (
run_date, time_in_s, distance_in_miles, rn, grp_start
) as (
select run_date, time_in_s, distance_in_miles, rn, run_date grp_start
from rws
where rn = 1

with rws as (
from running_log r
), within_7 (
) as (
from rws
where rn = 1
union all
select r.run_date, r.time_in_s, r.distance_in_miles, r.rn,
from within_7 w
join rws r
on w.rn + 1 = r.rn
)

with rws as (
from running_log r
), within_7 (
) as (
from rws
where rn = 1
union all
case
when r.run_date < w.grp_start + 7 then grp_start
else r.run_date
end grp_start
from within_7 w
join rws r
on w.rn + 1 = r.rn
)

with rws as (
from running_log r
), within_7 (
) as (
from rws
where rn = 1
union all
case
when r.run_date < w.grp_start + 7 then grp_start
else r.run_date
end grp_start
from within_7 w
join rws r
on w.rn + 1 = r.rn
)
select grp, w.*
from within_7 w

10g Model

select * from running_log
model
dimension by ( row_number() over ( order by run_date ) rn )
measures ( run_date, 1 grp, run_date grp_start )
rules (
grp_start[1] = run_date[cv()],
grp_start[any] =
case
when run_date[cv()] < grp_start[cv()-1] + 7 then
grp_start[cv() - 1]
else run_date[cv()]
end ,
grp[any] =
case
when run_date[cv()] < grp_start[cv()-1] + 7 then
grp[cv() - 1]
else nvl(grp[cv() - 1] + 1, 1)
end
);

define
within7 as
run_date < first ( run_date ) + 7

pattern ( within7+ )
define
within7 as
> 1 matches

match_recognize (
order by run_date
measures
count (*) as days
pattern ( within7+ )
define
within7 as
);

Am I getting
faster? stocksnap.io

current time < prev time

Analytic Functions

lag ( time_in_s )

select r.*, case
when time_in_s <
lag ( time_in_s )
then 'FASTER'
else 'SLOWER'
end faster
from running_log r

define
faster as
time_in_s < prev ( time_in_s )

pattern ( slower faster* )
define
faster as

match_recognize (
order by run_date
measures
classifier () as faster
define
faster as
);

FASTER
SLOWER
SLOWER
FASTER
FASTER

match_recognize (
order by run_date
measures
one row per match
define
faster as
);

match_recognize (
order by run_date
measures
all rows per match
define
faster as
);

RUN_DATE FASTER TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 SLOWER 420 1
02 Jan 2018 SLOWER 2,400 5
03 Jan 2018 SLOWER 4,932 10
06 Jan 2018 FASTER 2,350 5
07 Jan 2018 FASTER 410 1
10 Jan 2018 FASTER 400 1
13 Jan 2018 SLOWER 2,300 5
14 Jan 2018 FASTER 425 1
15 Jan 2018 FASTER 422 1

01 Jan 2018 SLOWER 420 1
02 Jan 2018 SLOWER 2,400 5
03 Jan 2018 SLOWER 4,932 10
06 Jan 2018 FASTER 2,350 5
07 Jan 2018 FASTER 410 1
10 Jan 2018 FASTER 400 1
13 Jan 2018 SLOWER 2,300 5
14 Jan 2018 FASTER 425 1
15 Jan 2018 FASTER 422 1
SLOWER!

01 Jan 2018 420 1
07 Jan 2018 410 1
10 Jan 2018 400 1
14 Jan 2018 425 1
15 Jan 2018 422 1
02 Jan 2018 2,400 5
06 Jan 2018 2,350 5
13 Jan 2018 2,300 5
03 Jan 2018 4,932 10

match_recognize (
partition by distance_in_miles
order by run_date
measures
all rows per match
define
faster as
);

01 Jan 2018 SLOWER 420 1
07 Jan 2018 FASTER 410 1
10 Jan 2018 FASTER 400 1
14 Jan 2018 SLOWER 425 1
15 Jan 2018 FASTER 422 1
02 Jan 2018 SLOWER 2,400 5
06 Jan 2018 FASTER 2,350 5
13 Jan 2018 FASTER 2,300 5
03 Jan 2018 SLOWER 4,932 10

Can I run 10k in
< 50 minutes?

Sum the total distance for runs
with a total time < 50 minutes

cumulative time <= 3,000 seconds

pattern ( fifty_minutes+ )
define
fifty_minutes as
sum ( time_in_s ) <= 3000
Returns the running total

match_recognize (
order by run_date
measures
first ( run_date ) as strt ,
sum ( time_in_s ) as total_time,
sum ( distance_in_miles ) as dist
pattern ( fifty_minutes+ )
define
fifty_minutes as
sum ( time_in_s ) <= 3000
);

STRT TOTAL_TIME DIST
01 Jan 2018 2,820 6
06 Jan 2018 2,760 6
10 Jan 2018 2,700 6
14 Jan 2018 847 2
Where's my 10 mile run?

any runs cumulative time < 3,000
and
one run cumulative time >= 3,000

pattern ( )

pattern ( under_fifty* over_fifty )

define
under_fifty as
sum ( time_in_s ) < 3000,
over_fifty as
sum ( time_in_s ) >= 3000
);
Includes under_fifty values

match_recognize (
order by run_date
measures
define
under_fifty as
over_fifty as
);

01 Jan 2018 7,752 16
06 Jan 2018 3,160 7
13 Jan 2018 3,147 7
Hmmm….

match_recognize (
order by run_date
measures
after match skip past last row
define
under_fifty as
over_fifty as
);

match_recognize (
order by run_date
measures
after match skip to next row
define
under_fifty as
over_fifty as
);

01 Jan 2018 7,752 16
02 Jan 2018 7,332 15
03 Jan 2018 4,932 10
06 Jan 2018 3,160 7
07 Jan 2018 3,110 7
10 Jan 2018 3,125 7
13 Jan 2018 3,147 7

Pixabay

How often did I run 5 miles
Followed by 2+ 1 mile runs
Within 7 days?

pattern ( five_mile one_mile {2,} )
define

define
five_mile as distance_in_miles = 5,

define
one_mile as distance_in_miles = 1

define
and run_date < first ( run_date ) + 7

match_recognize (
order by run_date
measures
count (*) as total_runs
define
);

STRT TOTAL_RUNS
06 Jan 2018 3
13 Jan 2018 3

Why would I want to do that?!

How do
I debug it?
Gratisography

(Regular) [exprsion]+ are easy to missteak

regex101.comregex101.com

classifier
=> Which variable matched?

classifier
match_number
=> Which group is this?

classifier
match_number
all rows per match

classifier
match_number
all rows per match with unmatched rows
=> Show me everything!

match_recognize (
order by run_date
measures
classifier () as var,
match_number () as grp
all rows per match with unmatched rows
define
);

RUN_DATE VAR GRP TIME_IN_S DISTANCE_IN_MILES
01 Jan 2018 420 1
02 Jan 2018 2,400 5
03 Jan 2018 4,932 10
06 Jan 2018 FIVE_MILE 1 2,350 5
07 Jan 2018 ONE_MILE 1 410 1
10 Jan 2018 ONE_MILE 1 400 1
13 Jan 2018 FIVE_MILE 2 2,300 5
14 Jan 2018 ONE_MILE 2 425 1
15 Jan 2018 ONE_MILE 2 422 1

Want more?
Pixabay

iTunes & PDF
FREE!
SQL for Data Warehousing and Analytics
https://oracle-big-data.blogspot.co.uk
Keith Laker
Analytic SQL PM

How to Find Patterns in Your Data with SQL

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to Find Patterns in Your Data with SQL

Similar to How to Find Patterns in Your Data with SQL (20)

More from Chris Saxon

More from Chris Saxon (8)

Recently uploaded

Recently uploaded (20)

How to Find Patterns in Your Data with SQL