Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data
1. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Secure Multi-Party Computation Based Privacy
Preserving Extreme Learning Machine Algorithm
Over Vertically Distributed Data
Ferhat ¨Ozg¨ur C¸atak
ozgur.catak@tubitak.gov.tr
T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute
Kocaeli, Turkey
Nov. 10 2015
The 2015 International Data Mining and Cybersecurity Workshop
With ICONIP2015
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
2. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Table of Contents
1 Introduction
Secure Multiparty Computation
Contributions
2 Preliminaries
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
3 Privacy-preserving ELM over vertically partitioned data
Privacy-preserving ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
3. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Secure Multiparty Computation
Contributions
Table of Contents
1 Introduction
Secure Multiparty Computation
Contributions
2 Preliminaries
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
3 Privacy-preserving ELM over vertically partitioned data
Privacy-preserving ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
4. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Secure Multiparty Computation
Contributions
Secure Multiparty Computation (SMC)
Definition
SMC : The goal of creating methods for parties to jointly compute
a function over their inputs while keeping those inputs private.
In an MPC,
a given number of participants, p1, p2, · · · , pN
each has private data, respectively d1, d2, · · · , dN
Participants want to compute the value of a public function on
that private data:
F(d1, d2, ..., dN ) while keeping their own inputs secret.
Challenges :
To protect each participant’s private data, and intermediate results
The computation/communication cost introduced to each
participant shall be affordable
Training data is arbitrarily partitioned
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
5. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Secure Multiparty Computation
Contributions
Contributions
Main contributions of the work
Privacy-preserving Extreme Learning Machine (ELM) training
model
Global ELM classification model from the distributed data sets in
multiple parties
Training data set is vertically partitioned among the parties
The final distributed model is constructed at an independent party
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
6. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Table of Contents
1 Introduction
Secure Multiparty Computation
Contributions
2 Preliminaries
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
3 Privacy-preserving ELM over vertically partitioned data
Privacy-preserving ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
7. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Extreme Learning Machine (ELM)
Learning Without Iterative Tuning
All hidden node parameters can be randomly generated without the
knowledge of the training data. That is, for any continuous target
function f and any randomly generated sequence
limL→∞ ||f (x) − fL(x)|| = limL→∞ ||f (x) −
L
i=1 βi G(ai , bi , x)|| = 0
holds with probability one if βi is chosen to minimize
||f (x) − fL(x)||, ∀i
G.-B. Huang, et al., ”Universal approximation using incremental constructive
feedforward networks with random hidden nodes,” IEEE Transactions on Neural
Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., ”Convex incremental extreme learning machine,”
Neurocomputing, vol. 70, pp. 3056-3062, 2007.
G.-B. Huang, et al., ”Enhanced random search based incremental extreme learning
machine,” Neurocomputing, vol. 71, pp. 3460-3468, 2008.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
8. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Extreme Learning Machine (ELM)
Figure: Generalized SLFN
Mathematical Model
For N arbitrary distinct samples
(xi , ti ) ∈ Rn × Rm, SLFNs with L
hidden nodes each with output
function, G(ai , bi , x) are
mathematically modeled as
L
i=1
βi G(ai , bi , xj ) = tj , j = 1, ..., N
(1)
(ai , bi ): hidden node parameters.
βi : the weight vector connecting the
ith hidden node and the output node.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
9. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Extreme Learning Machine (ELM)
Mathematical Model
L
i=1 βi G(ai , bi , xj ) = tj , j = 1, ..., N is equivalent to Hβ = T where,
h(x1)
...
h(xN)
=
G(a1, b1, x1) · · · G(aL, bL, x1)
...
...
...
G(a1, b1, xN) · · · G(aL, bL, xN)
N×L
(2)
β =
βT
1
...
βT
L
L×m
ve T =
tT
1
...
tT
N
N×m
(3)
H, is called the hidden layer output matrix of the neural network, the ith
column of H is the output of the ith hidden node with respect to inputs ,
x1, ..., xn
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
10. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Extreme Learning Machine (ELM)
Three-Step Learning Model
Given a training set D = {(xi , ti )|xi ∈ Rn
, ti ∈ Rm
, i = 1, · · · , N}, hidden
node output function G(a, b, x), and the number of hidden nodes L,
1 Assign randomly hidden node parameters (ai , bi ), i = 1, · · · , L.
2 Calculate the hidden layer output matrix H.
3 Calculate the output weight β = Ht
T.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
11. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Secure Multi Party Computation
Secure Multi-Party Computation
The problem of n players to compute an agreed function f of their
inputs
In vertically partitioned data, each party holds different attributes of
same data set.
The partition strategy is shown in Figure 2.
x1,1 · · · x1,t−1 · · · x1,t · · · x1,k
x2,1 · · · x2,t−1 · · · x2,t · · · x2,k
...
...
...
...
...
...
...
xm,1 · · · xm,t−1 · · · xm,t · · · xm,k
P1 · · · Pk
D =
Figure: Vertically partitioned data set D.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
12. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Secure Multi-Party Addition I
Secure Multi-Party Addition
Assume k ≥ 3 parties, P0, · · · , Pk−1, with party Pi holding value vi
Aim is to compute v =
k−1
i=0
vi
Sum of vi lies in F
P0 randomly chooses a number R ∈ F
P0 adds R to its local value v0,
Sends v0 = R + v0mod|F| to P1
The remaining parties Pi , i = 1, · · · , k − 1
Pi receives V = R +
i−1
j=0
vj mod|F|
Pi sends R + j=1
ivj mod|F| = (vi + V )mod|F|
Finally, P0, subtracts R from the final message to compute actual
results.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
13. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Secure Multi-Party Addition II
Party0
x0 , R ∈ F
Party1
x1
Party2
x2
Partyk
xk
V = R + x0mod|F|
V = R +
i−1
j=0 xjmod|F|
(xi + V )mod|F|
Figure: Secure Multi-Party Addition.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
14. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
Secure Multi-Party Addition III
Algorithm 1: Secure multi-party addition
1: procedure SMA(P)
2: P0 : R ← rand(F) P0 randomly chooses a number R
3: V ← R + x0 mod F
4: P0 sends V to node P1
5: for i = 1, · · · , k − 1 do
6: Pi receives V = R +
i−1
j=0
xj mod F
7: Pi computes V = R +
i
j=1
xj mod F = ((xi + V ) mod F)
8: Pi sends V to node Pi+1
9: P0 : V ← (V − R) = (V − R mod F) Actual addition result
10: end procedure
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
15. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Privacy-preserving ELM
Table of Contents
1 Introduction
Secure Multiparty Computation
Contributions
2 Preliminaries
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
3 Privacy-preserving ELM over vertically partitioned data
Privacy-preserving ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
16. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Privacy-preserving ELM
Privacy-preserving ELM I
1 Master party creates weight matrix, W ∈ RL×N
2 Master party distributes partition W with same feature size for each
parties.
3 Party P0 creates a random matrix,
R =
rand1,1(F) · · · rand1,L(F)
...
...
...
randN,1(F) · · · randN,L(F)
N×L
4 Party P0 creates perturbated output,
V = R +
x0
1 · w0
1 + b0
1 · · · x0
1 · w0
L + b0
L
...
...
...
x0
N · w0
1 + b0
1 · · · x0
N · w0
L + b0
L
5 for i = 1, · · · , k − 1
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
17. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Privacy-preserving ELM
Privacy-preserving ELM II
Pi computes V = V +
xi
1 · wi
1 + bi
1 · · · xi
1 · wi
L + bi
L
...
...
...
xi
N · wi
1 + bi
1 · · · xi
L · wi
N + bi
N
Pi sends V to Pi+1
6 P0 subtracts random matrix, R, from the received matrix V.
H = (V − R) mod F
7 Hidden layer node weight vector, β, is calculated. β = H†
· T
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
18. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Privacy-preserving ELM
Privacy-preserving ELM III
P arty0
x0
, w, H0
,R ∈ F,
Party1
x1
, H1
, w
Party2
x2
, H2
, w
Partyk
xk−1
,
Hk−1
, w
V0 = R +
H0
mod|F|
V1 = V0+H1
mod|F|
V2 = V1 + H2
mod|F|
Vk = Vk−1 + Hk
mod|F|
Hi
=
G(w1, b1, xi
1) · · · G(wL, bL, xi
1)
.
.
.
...
.
.
.
G(w1, b1, xi
N ) · · · G(wL, bL, xi
N )
N×L
R =
rand1,1(F) · · · rand1,L(F)
.
.
.
...
.
.
.
randN,1(F) · · · randN,L(F)
N×L
H = (V − R) mod F
Figure: Overview of privacy-preserving ELM learning.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
19. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Experimental setup
Simulation Results
Table of Contents
1 Introduction
Secure Multiparty Computation
Contributions
2 Preliminaries
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
3 Privacy-preserving ELM over vertically partitioned data
Privacy-preserving ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
20. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Experimental setup
Simulation Results
Experimental setup I
Our approach is applied to six different data sets to verify its model
effectivity and efficiency.
Table: Description of the testing data sets used in the experiments.
Data set #Train #Classes #Attributes
australian 690 2 14
colon-cancer 62 2 2,000
diabetes 768 2 8
duke breast cancer 44 2 7,129
heart 270 2 13
ionosphere 351 2 34
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
21. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Experimental setup
Simulation Results
Simulation Results I
The number of party size, k: from 3 to number of feature n, of the
data set
For instance, k = 3, and n = 14, then the first two party have 5
attributes, and last party has 4 attributes.
The accuracy of secure multi-party computation based ELM is
exactly same for the traditional ELM training algorithm.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
22. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Experimental setup
Simulation Results
Simulation Results II
Figure shows results of our simulations.
Time scale becomes its steady state position when number of parties
k, moves closer to number of attributes, n.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
10
−1
10
0
10
1
10
2
10
3
AttributeS ize(n)
P artyS ize(k)
Timescale
australian colon−cancer diabetes duke heart ionosphere
Figure: Vertically partitioned data set D.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
23. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Conclusions
Table of Contents
1 Introduction
Secure Multiparty Computation
Contributions
2 Preliminaries
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
3 Privacy-preserving ELM over vertically partitioned data
Privacy-preserving ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
24. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Conclusions
Conclusions
Summary
ELM learning algorithm, a new method.
ELM outperforms traditional Single Layer Feed-forward Neural-networks and
Support Vector Machines for Big Data 1
In all fields that ELM is applied (i.e. medical records, business, government),
privacy is a major concern.
Conclusions
Privacy-preserving learning model is proposed in vertically partitioned data
In multi-party partitioning without sharing the data of each site to
the others.
Master party divides weight vector, and each party calculates the activation
function result with its data and weight vector.
1Cambria, Erik, et al. ”Extreme learning machines [trends & controversies].”
Intelligent Systems, IEEE 28.6 (2013): 30-59.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
25. Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Conclusions
Thank You
Ferhat ¨Ozg¨ur C¸atak
ozgur.catak@tubitak.gov.tr
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin