How to classify time series data using deep learning techniques and TensorFlow.

1. Using Deep Learning and TensorFlow
to classify Time Series
Andreas Pawlik
Data Scientist at NorCom IT AG (Munich)
PhD in astrophysics (galaxy formation simulations)
TensorFlow Meetup, July 20, 2016, Munich

2. Data Scientists
Developers
jobs@norcom.de

3. Outline
• Time Series Classification & Use Case
• Deep Learning
• Why Tensorflow
• Deep Learning for Time Series: Paper Review
• Application to Use Case
• Summary

4. Classification for Time Series
• Time Series: Sequence of Events
• Examples:
– Light Curves in Astrophysics
– Skull (blood cell, butterfly, …) Shapes
– Electrocardiograms in Medicine
– Protein Sequences in Genetics
– Intruder Activity Logs in IT Security
– Sensor data in Industry
– …
• Use cases: Indexing, anomaly detection, …
Keogh et al (2006),
http://www.cs.ucr.edu/~eamonn/shape/shape.htm

5. Automotive example: oscillations in the engine
speed during gear shifts (“Schaltruckler”)
• Problem: gear shifts are
sometimes followed by
bucking
• Goal: root cause analysis
• Challenge: Tbs of data,
buckings are rare
• Approach:
– Use Big Data Technology
(Hadoop, Spark) to extract
individual gear shift events
– Use Machine Learning to
classify the engine speed
patterns during these
events
artificial toy data,
not to scale

6. Conventional classification techniques
(for a review see, e.g., Bagnall et al., arXiv:1602.01711)
• Distance based
– kNN search + majority vote
– Euclidean Distance,
Dynamic Time Warping
– Often difficult to beat in
accuracy
– Computationally expensive
• Feature based
– Machine Learning (logistic
regression, SVM, …)
– Usually less expensive
– Accuracy depends on
quality of features
(distributional properties,
spectral coefficients, …)
– Features are handcrafted

7. Deep Learning:
Feature Learning not Engineering
(e.g., Schmidhuber, arXiv:1404.7828; Goodfellow et al., www.deeplearningbook.org)
• Extremely successfully applied in
Speech Recognition, Natural
Language Processing, Image
Classification
• Prototype Architecture:
Multilayer Perceptron (fully
connected)
• Nonlinearities, for complexity
• Deep = many hidden layers, for
abstraction
Input
Output
Hidden

8. • CNNs very successfully applied to image classification,
beating classical feature based methods in accuracy
• Biologically inspired variant of Multilayer Perceptrons
(animal visual cortex, Hubel & Wiesel 1968)
• Building blocks:
– Local convolution
– Shared weights
– Subsampling
Convolutional neural nets (CNNs)
Image credit: http://deeplearning.net/tutorial/lenet.html

9. Convolutional neural nets
Feature learning Classification
Image credit: http://deeplearning.net/tutorial/lenet.html

10. Why TensorFlow
“In short, Theano invented the genre. It's the Ford
motors of compiling code for deep learning. But
TensorFlow appears well on its way to emerging as
the Tesla motors of the genre.”
-- Zachary Chase Lipton, UCSD
(http://www.kdnuggets.com/2015/12/tensor-flow-terrific-deep-learning-library.html)
• Python
prototyping
• Distributed
learning using
multiple devices
• Google (where
things move fast)
• Similar to Theano
(yes, it’s a plus)

11. Deep Learning for Time Series
• Like images, time series are often multiscale and locally correlated, and the
location of characteristic patterns is not fixed
– This suggest the use of convolutional neural nets
• How do we feed in the time series?
– as image: is there an appropriate representation?
(Wang & Oates, arXiv:1506.00327)
– raw: do we need to adapt feature learning?
(Cui, Chen, & Chen, arXiv:1603.06995; see also Zheng et al.,
http://staff.ustc.edu.cn/~cheneh/paper_pdf/2014/Yi-Zheng-WAIM2014.pdf)
Image credit: http://deeplearning.net/tutorial/lenet.html

12. Imaging Time-Series to Improve Classification
Wang & Oates, arXiv:1506.00327
• GAF is bijective
• GAF preserves
temporal
dependencies

13. Multi-Scale Convolutional Neural Networks for
Time Series Classification
Cui et al., arXiv:1603.06995
Code at http://www.cse.wustl.edu/~z.cui/projects/mcnn/

14. Both works report excellent performance
competitive with or better than other methods
Wang & Oates,
arXiv:1506.00327
Cui et al.,
arXiv:1603.06995

15. Application to bucking problem:
Artificial Data Set
• Class 1 (Normal): Logistic
Function (random steepness)
+ Gaussian random noise
(#time stamps: 100)
• Class 2: wavelet of random
width and height, added at
random location
(#time stamps: 100)

16. Classification Error Rates
(preliminary test results, no claim of correctness, use at your own risk)
Training
Size
100 1000 10000
kNN+DTW 17% 8 -
(too expensive)
Raw +
Log. Regr.
39 27 25
Wavelet +
Log. Regr.
11 2 1
GADF +
CNN
25 7 3
Raw +
MCNN
2 0 0 *I could not get it to work on time series of
length 100. Therefore, for this test, I used
length of 200.
*For this test, I used time series of length
28 to run with the standard parameters for
the Theano/LeNet CNN.

17. Summary
• Time Series Classification is a standard data
science problem, yet it remains a challenge
• Conventional approaches are computational
expensive (distance based), or their accuracy
depends strongly on the quality of the user input
(feature engineering)
• Deep Convolutional Nets are a promising
alternative that do not require to handcraft
features yet may reach very high accuracy while
being computationally efficient