Finite state machines (FSM) are a common technique for recognizing patterns over streams of data. However basic FSM cannot easily handle noisy data such as sensor data from a wearable device that is generated using human limb movement. This session presents a novel variation of FSM using techniques from functional programming to construct 'functional' state machines (FnSM) that are computationally as efficient as FSM but can handle probabilistic patterns almost as well as more elaborate techniques such as Hidden Markov Models. A real-world application of FnSM is described; an Android smartwatch app that recognizes gestures performed using wrist and forearm movements. Also described is the use of an evolutionary computing algorithm to optimize the performance of the FnSM by selecting better parameter values for the various state-transition decisions. We are increasingly living in a word where data processing needs to happen in real-time to enable quicker decision making. FSM are an essential technique for recognizing patterns over sequential data. This talk describes techniques that leverage FP to succinctly define complex FSM. Additionally, such constructed FSM can be trained with the help of a suitable evolutionary computing algorithm to also handle noisy data.

1. RECOGNIZING PATTERNS
IN NOISY DATA USING
TRAINABLE ‘FUNCTIONAL’
STATE MACHINES
Faisal Waris

2. OUTLINE

3. BACKGROUND & MOTIVATION -
3 TRENDS
Android Auto
Cadillac Gesture Pad
BMW In-the-air Gestures

4. IDEA – WEARABLE DEVICE
GESTURE CONTROL OF HMI

5. Twist
Tap Swipe Left Right Escape
Twist gesture
recognized
Escape gesture recognized
IMPLEMENTATION
Simple Gesture
Vocabulary
HMI Menu
Hierarchy

6. GESTURE RECOGNITION ≈
PATTERN RECOGNITION IN
NOISY (SEQUENTIAL) DATA
Sensors Inherent Noise
Fusion: Accelerometer, Gyro, Magnetometer
Limb
Movement
Not precise
Variations
Vehicle
Motion
Acceleration
Cornering
Bumps

7. SEQUENTIAL DATA, PATTERN
RECOGNITION TECHNIQUES

8. FINITE STATE MACHINES
(FSM)

9. OUTLINE

10. PATTERN RECOGNITION WITH
FSM
FSM to recognize “…aaab…” pattern
a1 a2 a3
a a a
not a
b

11. OUTLINE

12. ‘FUNCTIONAL’ STATE
MACHINES (FNSM*)
FSM expressed using functional programming constructs
* FnSM is both singular and plural

13. NATURAL TRANSFORMATION TO
F# (ML)
type M<'s> = M of ('s -> M<'s>)
let rec start = function
| 'a' -> M a1
| _ -> M start
and a1 = function
| 'a' -> M a2
and a2 = function
| 'a' -> M a3
and a3 = function
| 'b' -> M ``end``
and ``end`` _ = M ``end``
a1 a2 a3
a a a
not a
b
FnSMforrecognizingthe“aaab”pattern

14. STATE-SPACE EXPLOSION
How about 3 ‘a’s and 3 ‘b’s in any order?
Are 6 states enough?
a1 a2 a3
b1 b2 b3
Need exponential # of states to recognize this pa

15. FNSM FOR 3 ‘A’S AND 3 ‘B’S IN
ANY ORDERlet rec fStart = function
| 'a' -> fAB 1 0 |> M
| 'b' -> fAB 0 1 |> M
| _ -> fStart |> M
and fAB countA countB = function
| 'a' when countA + 1 = 3 -> fB countB |> M
| 'a' -> fAB (countA + 1) countB |> M
| 'b' when countB + 1 = 3 -> fA countA |> M
| 'b' -> fAB countA (countB + 1) |> M
| _ -> failwith "invalid input"
and fA count = function
| 'a' when count + 1 = 3 -> fEnd |> M
| 'a' -> fA (count + 1) |> M
| _ -> failwith "invalid input"
and fB count = function
| 'b' when count + 1 = 3 -> fEnd |> M
| 'b' -> fB (count + 1) |> M
| _ -> failwith "invalid input"
and fEnd _ = printf "done"; fEnd |> M
1 0

16. FNSM FOR WEARABLE DEVICE
SENSOR DATA
type SnsrEvent =
{
Snsr : SensorTypeEnum
Ticks : long
X : float; Y : float; Z : float}
let rec start = function
| {Snsr=LinearAcceleration; Z=z}  …
| {Snsr=Gyroscope; Z=z; X=x}  …
| {Snsr=s; Ticks=t} when …  …
Matches an event where
the sensor is Linear
Acceleration and binds
the Z-Axis value to the
variable ‘z’
Given an event data
structure of this type
(record)...
We can define FnSM with
Pattern Matching code
like this

17. F# FNSM MODULE IN
PRACTICE - ANDROID ‘function-state’
recursive type
Evaluation function takes a
function-state and an event
and returns the new
function state PLUS whether
something was recognized
or not

18. OUTLINE

19. HANDLING NOISY DATA
let rec start = function
| {Snsr=LinearAcceleration; Z=z} when abs z > 5.0 // m/sec2
 possible_swing z 1 |> M
| _  start |> M
and possible_swing prev_z count = function
| {Snsr=LinearAcceleration; Z=z} when
count < 30

let new_z = updateAvg (prev_z, count, z)
possible_swing new_z (count + 1) |> M
|{Snsr=LinearAcceleration; Z=z} ->
when <avg z greater some threshold> -> …
Compute avg. linear
acceleration over
some count number
of events

20. PARAMETERIZE FNSM FOR
TRAINABILITY
let rec start cfg = function
| {Snsr=LinearAcceleration; Z=z} when abs z > cfg.MIN_Z_THRESHOLD
 possible_swing cfg z 1
| _  start
and possible_swing cfg prev_z count = function
| {Snsr=LinearAcceleration; Z=z} when
count < cfg.COUNT_LIMIT

let curr_z = updateAvg (prev_z, count, z)
possible_swing cfg curr_z (count + 1)
| …
FnSM parameterized
with configuration
data
Optimal
configuration values
can be determined
through machine
learning

21. FNSM – OTHER
FnSM as partially applied values in
other FnSM
 Hierarchical organization of state machines
 Call-return semantics / modularization
 Parallel execution of multiple ‘child’ state machines
 E.g. recognize 1 of many gestures at the same time
Reactive Extensions Operators
 Write more compact and/or efficient custom
operators for Rx
 (used with Apache Storm topology)
 Future: Project Brillo – Android for IoT

22. FNSM CONCEPTS – INTERIM
SUMMARY

23. OUTLINE

24. FNSM TRAINING – PARAMETERS
z_accel_front_thrsld = 2.74647951f;
z_accel_back_thrsld = -4.25041485f;
z_accel_avg_front_thrsld = 2.39706802f;
z_accel_avg_back_thrsld = -4.60432911f;
xy_accel_tolerance = 2.69525146f;
avg_over_count = 1.0f;
gstr_time_limit = 781282632L;
xz_rot_tolerance = 2.24759841f;
y_rot_tolerance = 3.57676268f;
gstr_time_limit = 460464225L;
x_accel_tolerance = 3.22107768f;
y_accel_tolerance = 2.86686087f;
ret_x_accel_tolerance = 2.9499104f;
ret_y_accel_tolerance = 3.1444633f;
xz_rot_tolerance = 2.97020698f;
y_rot_tolerance = 0.526834428f;
low_z_accel_limit = 3.09724712f
Sample
configuration
parameters

25. FNSM TRAINING – PROCESS
2
Labelled training
data: Perform
each gesture X
times and record
sensor output
Analyze gesture data and
define parameterized
FnSM for each gesture
with initial (best guess)
parameter values
1
3
Formulate a suitable
optimization problem
(fitness function) that
uses the FnSM
parameter values – this
is key
Run the evolutionary computing
optimization algorithm (EA) – given
the:
1) Training data
2) Parameterized gesture FnSM
3) Optimization formulation
To search for parameter values that
yield the best value of the objective
function
4
arg 𝑚𝑎𝑥 𝑝 𝐹 𝑝 = W1 ∗ 𝑓𝑖𝑡 𝑝, 𝑔
𝑔
− W2 ∗ 𝑐𝑜𝑢𝑛𝑡𝐷 𝑝 − 𝐷𝐹 𝑝

26. FNSM TRAINING – OPTIMIZATION
FORMULATION
p = parm array values from a population individual
g ∈ G ={Left, Right, Tap, Swipe}, the action gestures
fit(p, g) = GC(g) – | count(p, g) – countNot (p, g) |
is the fitness function for a single gesture. This function’s value is calculated by running the FnSM
parameterized with p over the training set for g. (The Ideal score is 0 when all of the 10 gesture instances
are recognized and none any other gestures are recognized).
GC(g) = the # of gestures of type g in the training set for g (constant 10 for all gestures).
count(p, g) = count of gesture of type g recognized when the p parameterized FnSM is run over the
training set for g. (Ideal value is 10).
countNot(p,g) = count of all gestures of type [G – {g}] recognized when the p parameterized FnSM is run
over the training set for g. (Ideal value is zero).
countD(p) = total count of any gesture in G recognized in the negative (driving) training set with a p
parameterized FnSM. Ideal score is zero. This term is to reduce inadvertent recognition of any gestures.
DF(p) = 𝑖,𝑗∈𝐺,𝑖 ≠𝑗 𝑓𝑖𝑡 𝑝, 𝑖 − 𝑓𝑖𝑡 𝑝, 𝑗
The sum of absolute differences between all individual fitness scores. This term favors p values that
result in even gesture recognition performance across all gesture types in G.
W1 and W2 are weighting factors to control the influence of some of the terms.
In summary the fitness function is construed so that a perfect score of 120 is achieved when all 10
instances of a gesture type are recognized with the corresponding training set and none of any other type
are recognized, for each of the action gesture types.
arg 𝑚𝑎𝑥 𝑝 𝐹 𝑝 = W1 ∗
𝑔
𝑓𝑖𝑡 𝑝, 𝑔 − W2 ∗ 𝑐𝑜𝑢𝑛𝑡𝐷 𝑝 − 𝐷𝐹 𝑝
Step (before / after
CA)
Fitness
Score (max
120)
Best guess
parameter values
-1.5
After CA
optimization
92.5

27. SUMMARY
Functional programming provides a powerful
means of modeling FSM
 State | Events | Transitions  function | pattern match | return a
function
Partial function application
 Model complex patterns (state-space compression)
 Handle probabilistic patterns through aggregation
 Make FnSM amenable to training or machine learning through
parameterization
Training: Training data + Optimization + EA
Human modelled, machine optimized, computationally
efficient pattern recognition
Upcoming paper in IEEE Conference Swarm / Human Blended
Workshop
 “Human-Defined, Machine-Optimized - Gesture Recognition using a mixed
Approach”
 SHBI2015 (Cleveland, Sept. 28, 29)

28. OUTLINE

29. A TAXONOMY OF EVOLUTIONARY COMPUTING
ALGORITHMS
EvolutionaryComputing
Biological
Genetic
Algorithm
Genetic
Programming
Evolutionary
Algorithm
Neural Net
Artificial Immune
System
Social
Ant Colony
Optimization
Particle Swarm
Optimization
Artificial Bee
Colony
Cultural
Algorithm
Molecular
Chemical
Reaction
Optimization
Simulated
Annealing
Hybrid algorithms
+ almost endless
variations

30. “NO FREE LUNCH” THEOREM*
Essentially states that if a search based algorithm is
better on some problems, it will be worse on other
problems (as compared to other search based
algorithms)
Explains the existence of the sheer variety of
evolutionary computing algorithms
Different search strategies are needed to tackle different
problems
* Wolpert & Macready (1997)

31. CULTURAL ALGORITHM
FRAMEWORK*
INSPIRED BY THE PROBLEM SOLVING ABILITIES OF
CULTURE
Population Space
Belief Space
accept()
influence()
Knowledge Sources
Historical/Topological/Domain/Normative/S
ituational
*see Wikipedia page for introduction and further references
Knowledg
e
Distributi
on

32. CULTURAL ALGORITHM TYPES–
F# (ALMOST)
type CA =
{
Population : Population
Network : Network
KnowlegeDistribution : KnowledgeDistribution
BeliefSpace : BeliefSpace
AcceptanceFunction : Acceptance
InfluenceFunction : Influence
UpdateFunction : Update
Fitness : Fitness
}
where:
Topology = the network topology type of the population e.g. square, ring, global, etc.
Knowledge = Situational | Historical | Normative | Topographical | Domain
Individual = {Id:Id; Parms:Parm array; Fitness:float; KS:Knowledge}
Id = int
Parm = numeric value such as int, float, long, etc.
Population = Individual array
Network = Population  Id  Individual array
Fitness = Parm array  float
BeliefSpace = KnowledgeSource Tree
Acceptance = BeliefSpace  Population  Individual array
Influence = BeliefSpace  Population  Population
Update = BeliefSpace  Individual array  BeliefSpace
KnowledgeDistribution = Population  Network  Population
KnowledgeSource =
{
Type : Knowledge
Accept : Individual array  Individual array * KnowledgeSource
Influence : Individual  Individual
}

33. THANK YOU contact: fwaris /
wayne.edu