Default Credit Card Prediction

Default of Credit Card Clients
Author:
Alexandre Pinto
Faculty of Sciences and Technology
Department of Informatics Engineering
University of Coimbra

Summary
Contents
1 Introduction
2 Feature Assessment/Visualization
3 Preprocessing
4 Feature Selection/Reduction
5 Classiﬁcation
6 Evaluation Metrics
7 Demo - Short Experiment
8 Conclusions
Default of Credit Card Clients Alexandre Pinto 2

Introduction
Problem Deﬁnition
Default Credit Card:
• Happens when clients fail to adhere to the credit card
agreement, by not paying the monthly bill
Main Goal:
• Development of a system capable of detecting clients that
will not be able to pay the next month

Introduction
Dataset Description
• 23 features: X1 - X23
• One predictive binary label (Default: Yes = 1, No = 0)
• X1: Amount of the given credit
X2: Gender
X3: Education
X4: Marital status
X5: Age
X6-X11: History of past payment (4/2005 to 9/2005)
X12-X17: Amount of bill statement (4/2005 to 9/2005)
X18–X23: Amount of previous payment (4/2005 to 9/2005)

Feature Assessment/Visualization
• Useful to visualize how the features are distributed, and
assess their discriminative capability

Tasks
Normalized Histogram
Distributions:
• Bins represent unique values
(Grouped by class)
• Heights represent relative
frequencies (Normalized)
Figure: Normalized Histogram

Tasks
Box Plots:
• Show distribution of features
for each class
• Representation of min,
median, max and outlier
values
Figure: Box Plot

Tasks
Pairwise Relationships:
• 2 × 2 grid
• Histograms in diagonal cells,
Scatter plots otherwise
• Useful to see how features
relate to each other
• Useful to see how two
features separate the pattern
classes
Figure: Box Plot

Tasks
Empirical Cumulative
Distribution Function :
• Check if feature distributions
are drawn from a normal
distribution
Figure: Cumulative Function vs Std
Normal Curve

Tasks
Pearson Correlations:
• Check highly correlated
features
ρ =
covariance(xi , xj )
σi × σj

Tasks
Two-Dimensional PCA:
• First 2 component axes with
highest variance
Figure: 2D PCA

Tasks
Two-Dimensional LDA:
• First 2 component axes with
class-separation
Figure: 2D LDA

Preprocessing
• Useful to improve the overall training data quality
• Turns data more suitable for the classiﬁcation models,
improving the ﬁnal accuracies

Preprocessing
Tasks
Standardization:
• Center the data by removing
the mean
• Scale the data by dividing by
the standard deviation
• Obtain normally distributed
data (ρxstd
= 0,σxstd
= 1)
xstd =
x − ρX
σX

Preprocessing
Tasks
Scaling features to a range:
• Scale features to lie between
a min and max value
x[0,1] =
x − minx
maxx − minx

Preprocessing
Tasks
Normalization:
• Scale samples/features
vectors to have unit norm
• Divide each element by the
euclidean norm of the vector
xnorm =
x
i xi
2

Preprocessing
Dataset Balancing
• Classes are not equally distributed:
Majority Class (’0’): 23364 (∼ 78%)
Minority Class (’1’): 6636 (∼ 22%)
• Biased classiﬁers tend to choose majority class

Preprocessing
Dataset Balancing
Two main methods:
• Oversampling the minority class
• Undersampling the majority class

Preprocessing
Undersampling
Random Majority Undersampling:
• Start with a set with samples from minority class
• Randomly add samples from the majority class until the
dataset is balanced

Preprocessing
Undersampling
NearMiss-1:
• Selects samples from majority class which are close to some of
the minority samples

Preprocessing
Undersampling
NearMiss-3:
• Selects samples from majority class which are the farthest
from the nearest minority samples

Preprocessing
Undersampling
Neighbor Cleaning Rule:
• Removes majority samples who are misclassiﬁed by its three
nearest neighbors
• Removes neighbor majority samples who misclassify minority
samples

Preprocessing
Oversampling
Random Minority Oversampling:
• Start with a set with samples from majority class
• Randomly add samples from the minority class until the
dataset is balanced

Preprocessing
Oversampling
SMOTE:
• Start with a set with
samples from majority class
• Add synthetic samples
created from the minority
class (interpolation) until
the dataset is balanced
Ssyntentic = Si + (Sk − Si )δ

Feature Selection/Reduction
Useful to:
• Remove irrelevant and redundant features
• Improve the prediction performance
• Reduce dimensionality and complexity
Approaches:
• Filter Methods: A subset of features is selected, without
considering the predictive model
• Wrapper Methods: The best subset of features is selected,
using the predictive model to rank the subset

Filter Methods
Information Gain:
• Rank each attribute
by its ability to
discriminate the
pattern classes
(decrease in entropy)
IG(S, A) = H(S)−
v∈values(A)
|Sv |
|S|
H(Sv )

Filter Methods
Information Gain Ratio:
• Information Gain
normalized with the
entropy of the attribute
• It takes into account the
number and size of branches
of a split
• Reduces bias (attributes
with high number of unique
values)
IGR(S, A) =
IG(S, A)
H(A)

Filter Methods
Kruskal-Wallis Test:
• Test whether class
groups are drawn
from the same
population and with
the same mean
• Assess the
class-separability of
the feature
H =
12
N(N + 1)
k
i=1
R2
i
ni
− 3(N + 1)

Filter Methods
Fisher Score:
• Select features with high
class-separability and low
class-variability
F(Xi ) =
|m1 − m2|2
s2
1 + s2
2

Filter Methods
Pearson Correlations:
• Select features highly
correlated with the target
class
• Select features with low
correlation between them
ρ =
covariance(xi , xj )
σi × σj

Filter Methods
mRMR:
• Select features with high
relevance (mutual
information) with the target
class
• Select features with low
redundancy between them
max D =
1
|S|
xi ∈S
I(xi , c)
min R =
1
|S|2
xi .xj ∈S
I(xi , xj )
max φ(D, R), φ = D − R

Filter Methods
Area Under the Curve:
• Select features that have good classiﬁcation performance

Wrapper Methods
Sequential Forward/Backward Selection:
• Start with an empty/full feature set
• Select the next best/worst feature that have the
largest-increase/smallest -decrease in feature importances

Wrapper Methods
Recursive Feature Elimination:
• Start with a full feature set and compute feature importances
• Recursively compute/remove features with low importances

Reduction methods
Linear Transformation Techniques:
• PCA: Find the directions (principal components) that
maximize the variance in the dataset
• LDA: Computes the directions (linear discriminants)
that maximize class-separability and minimizes the
variance within the class

Classiﬁcation
Predictive Models:
• Distance Based: Min. Distance Classiﬁer && k-NN
• Probabilistic: Naive Bayes
• Search: Decision Tree
• Optimization: Support Vector Machines (SVM)
• Ensemble: Random Forest

Evaluation Metrics
• Useful to assess the classiﬁer performance

Evaluation Metrics
Accuracy:
• Proportion of correctly
identiﬁed and rejected
instances
ACC =
TP + TN
TP + TN + FP + FN

Evaluation Metrics
Precision
Precision:
• Proportion of correct
answers from the positive
predictions
P =
TP
TP + FP

Evaluation Metrics
Recall
Recall:
• Proportion of correct
answers from the whole
positive part of a dataset
R =
TP
TP + FN

Evaluation Metrics
F1
F1:
• Harmonic mean of precision
and recall
F1 = 2 ×
P · R
P + R

Evaluation Metrics
Stratiﬁed K-fold Cross Validation
Stratiﬁed K-fold Cross Validation:
• Each fold is a good representative of the whole.

Evaluation Metrics
ROC Curves
ROC Curve:
• Show trade-oﬀs between
sensitivity(TPR) and
speciﬁcity(FPR)
Figure: ROC curves

Evaluation Metrics
Precision-Recall Curves
Precision-Recall Curve:
• Show trade-oﬀs between
precision and recall
Figure: PR curves

Demo
Short Experiment
• Features Standardized
• Data balanced with SMOTE
• Keep top 15 best features using mRMR
Precision Recall F1 AUC Avg Precision
Min. Distance 0.69 ± 0.02 0.64 ± 0.00 0.66 ± 0.01 0.65 ± 0.01 0.76 ± 0.01
kNN 0.76 ± 0.01 0.91 ± 0.04 0.82 ± 0.02 0.79 ± 0.02 0.86 ± 0.01
Naive Bayes 0.57 ± 0.01 0.93 ± 0.01 0.71 ± 0.01 0.56 ± 0.01 0.77 ± 0.00
Linear SVM 0.70 ± 0.02 0.65 ± 0.01 0.67 ± 0.01 0.67 ± 0.02 0.77 ± 0.01
Decision Tree 0.79 ± 0.02 0.82 ± 0.17 0.79 ± 0.11 0.79 ± 0.07 0.85 ± 0.05
Random Forest 0.88 ± 0.01 0.82 ± 0.16 0.84 ± 0.10 0.85 ± 0.07 0.90 ± 0.04
Table: Results

Demo
Short Experiment

Conclusions
• Feature Engineering(Data Transformation, Feature Selection)
is probably the most important step
• Exploratory Data Analysis is important to better get a sense
of the distribution of the data
• Feature selection helps reduce training times and keep only
the most relevant and non-redundant features
• PR is an iterative process

References I
[RF] A Gentle Introduction to Random Forests, Ensembles, and
Performance Metrics in a Commercial System.
https://citizennet.com/blog/2012/11/10/
random-forests-ensembles-and-performance-metrics/.
Accessed: 2016-06-06.
[Kag] General Tips for participating Kaggle Competitions.
http://www.slideshare.net/markpeng/
general-tips-for-participating-kaggle-competitions.
Accessed: 2016-06-06.
[knn] KNN classiﬁcation.
https://www.researchgate.net/figure/260397165_fig7_
Pseudocode-for-KNN-classification. Accessed:
2016-06-06.

References II
[PCA] Linear Discriminant Analysis - Bit by Bit. http://
sebastianraschka.com/Articles/2014_python_lda.html.
Accessed: 2016-06-06.
[NB] Naive Bayes. http://www.slideshare.net/
chowdhury343/naive-bayes-presentation. Accessed:
2016-06-06.
[6] Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer,
W. P. (2002). SMOTE: Synthetic Minority Over-Sampling
Technique. Journal of artificial intelligence research, pages
321–357.
[7] Laurikkala, J. (2001). Improving Identification of Difficult
Small Classes by Balancing Class Distribution. Springer.

References III
[8] Mani, I. and Zhang, I. (2003). kNN approach to unbalanced
data distributions: a case study involving information extraction.
In Proceedings of workshop on learning from imbalanced
datasets.
[9] Peng, H., Long, F., and Ding, C. (2005). Feature selection
based on mutual information criteria of max-dependency,
max-relevance, and min-redundancy. Pattern Analysis and
Machine Intelligence, IEEE Transactions on, 27(8):1226–1238.
[10] Prati, R. C., Batista, G. E., and Monard, M. C. (2009). Data
Mining with Imbalanced Class Distributions: Concepts and
Methods. In IICAI, pages 359–376.
[11] Yeh, I.-C. and Lien, C.-h. (2009). The comparisons of data
mining techniques for the predictive accuracy of probability of
default of credit card clients. Expert Systems with Applications,
36(2):2473–2480.

Default Credit Card Prediction

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Default Credit Card Prediction

Ähnlich wie Default Credit Card Prediction (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Default Credit Card Prediction