Payal Mehta 9867746289, Escorts Service Near The Taj Mahal Palace Colaba
Pitfalls in benchmarking data stream classification and how to avoid them
1. Pitfalls in Benchmarking Data Stream
Classification and How to Avoid Them
Albert Bifet1, Jesse Read2, Indr˙e ˇZliobait˙e3
Bernhard Pfahringer4, Geoff Holmes4
1Yahoo! Research Barcelona
2Universidad Carlos III, Madrid, Spain
3Aalto University and Helsinki Institute for Information Technology (HIIT), Finland
4University of Waikato, Hamilton, New Zealand
ECML-PKDD 2013, 25 September 2013
2. Data Streams
Data Streams
Sequence is potentially infinite
High amount of data: sublinear space
High speed of arrival: sublinear time per example
Once an element from a data stream has been processed
it is discarded or archived
Big Data & Real Time
4. Electricity Dataset
Popular benchmark for testing adaptive classifiers
Collected from the Australian New South Wales Electricity
Market.
Contains 45,312 instances which record electricity prices
at 30 minute intervals.
The class label identifies the change of the price (UP or
DOWN) related to a moving average of the last 24 hours.
17. New Evaluation for Stream Classifiers
Kappa Statistic
p0: classifier’s prequential accuracy
pc: probability that a chance classifier makes a correct
prediction.
κ statistic
κ =
p0 − pc
1 − pc
κ = 1 if the classifier is always correct
κ = 0 if the predictions coincide with the correct ones as
often as those of the chance classifier
18. New Evaluation for Stream Classifiers
Kappa Plus Statistic
p0: classifier’s prequential accuracy
pe: no-change classifier’s prequential accuracy
κ+ statistic
κ+
=
p0 − pe
1 − pe
κ+ = 1 if the classifier is always correct
κ+ = 0 if the predictions coincide with the correct ones as
often as those of the no-change classifier
22. SWT: Temporally Augmented Classifier
SWT: meta strategy that builds meta instances by augmenting
the original input attributes with the values of recent class
labels from the past
Pr[class is c] ≡ h(xt
, ct−
, . . . , ct−1
)
for the t-th test instance, where is the size of the sliding
window over the most recent true labels.
26. Forest Cover Type Dataset
0 2 4
·105
60
80
100
Time, instances
Accuracy,%
No-Change HAT
Lev. Bagging
0 2 4
·105
0
20
40
60
80
100
Time, instances
KappaStatistic,% No-Change HAT
Lev. Bagging
0 2 4
·105
−300
−200
−100
0
100
Time, instances
KappaPlusStatistic,%
No-Change HAT
Lev. Bagging
0 2 4
·105
0
20
40
60
80
100
Time, instances
Accuracy,%
No-Change SWT HAT
SWT Lev. Bagging
0 2 4
·105
0
20
40
60
80
100
Time, instances
KappaStatistic,%
No-Change SWT HAT
SWT Lev. Bagging
0 2 4
·105
−300
−200
−100
0
100
Time, instances
KappaPlusStatistic,%
No-Change SWT HAT
SWT Lev. Bagging
27. Conclusions
Temporal dependence in data stream mining
new κ+ measure
a wrapper classifier SWT
Pitfalls in Benchmarking Data Stream
Classification and How to Avoid Them