Data doping solution for the physics price in "Flavours of Physics Kaggle competition

1. DATA DOPING
Solution for "Flavour of Physics" challenge
Dr. Vicens Gaitan
Grupo AIA
Heavy Flavour Data mining Workshop
18-20 February 2016
University of Zurich

2. AGENDA
• “Flavour of Physics” Kaggle Challenge
• Why is so hard to “discover” the invariant mass?
• How to win the challenge: leasons learned
• Breaking the rules: Data Doping
• Machine Learning in HEP
• Conclusions

3. “FLAVOUR OF PHYSICS” KAGGLE
CHALLENGE

4. WHY IS SO HARD TO “DISCOVER” THE
INVARIANT MASS?
• Fact 1: The tau invariant mass can be reconstructed with high accuracy from
kinematic variables (p0,pt,eta) and/or lifetime & time of flight
ptau<-function(d){
# muon mass in MeV/c^2
mmu = 105.6583715
# calculate tau energy
tau_e = sqrt(d$p0_p ** 2 + mmu**2) + sqrt(d$p1_p ** 2 + mmu**2) +sqrt(d$p2_p ** 2 + mmu**2)
# calculate pz of tau candidate
tau_pz = d$p0_pt * sinh(d$p0_eta) + d$p1_pt * sinh(d$p1_eta)+d$p2_pt * sinh(d$p2_eta)
# calculate momentum of tau candidate
tau_p =sqrt(d$pt ** 2 + tau_pz ** 2)
# calculate eta of tau candidate
tau_eta = asinh(tau_pz / d$pt)
# calculate mass of tau candidate
tau_m2=tau_e ** 2 - tau_p ** 2
tau_m2[tau_m2<0]=0
tau_m_k=sqrt(tau_m2)
#M = tau_p*LifeTime*c/FlightDistance
#c Speed of light
c= 299.792458
tau_m_t=tau_p*d$LifeTime*c/d$FlightDistance
return(list(tau_e,tau_pz,tau_p,tau_eta,tau_m))
}

5. WHY IS SO HARD TO “DISCOVER” THE
INVARIANT MASS?
• Fact 2: The reconstructed tau mass separates nicely signal and background
because the background spectrum has a "hole" for decays coming from a true tau
(Real data (background) in this window can contain “signal”)
Train & Agreement Test

6. WHY IS SO HARD TO “DISCOVER” THE
INVARIANT MASS?
• Toy models ( not taking into account agreement & correlation):
• XGBoost
• 3-fold CV
• Par("max_depth"=5,"eta"=.1)
1. Using true mass : wAUC = 0.999999999999999(89) +/- 8.4e-16 (!!)
2. Using mass_k: wAUC = 0.997(51) +/- 0.00038
3. Using mass_t: wAUC = 0.996(81) +/- 0.00021
4. Using both wAUC = 0.999(83) +/- 0.00012
Reconstructed Mass is THE Golden Feature

7. WHY IS SO HARD TO “DISCOVER” THE
INVARIANT MASS?
• BUT using as variables:
c("p0_p","p1_p","p2_p","p0_pt","p1_pt","p2_pt","p0_eta","p1_eta","p2_eta","pt","LifeTime","FlightDistance")
c("tau_e","tau_p","LifeTime","FlightDistance")
wAUC = 0.85(15) +/- 0.0032 ??????
And trying to fit the mass with XGBoost : we obtain:

8. WHY IS SO HARD TO “DISCOVER” THE
INVARIANT MASS?
The reason: Highly correlated variables: mass is an effect of 1 over 2500
• Solution: uncorrelate variables with PCA
• wAUC = 0.947(73) +/- 0.00099
• Gradient Boosting Trees are not able build a representation of Invariant Mass
• Maybe Deep Learning can do it?

9. HOW TO WIN THE CHALLENGE:
LEASONS LEARNED
Recipe to win the challenge
1. Add the reconstructed tau mass ( don’t bother about mass correlation test)
2. Use all available variables ( profit from bad simulated MC variables to separate signal from real
background)
AUCw=0.9999920 CVM=0.0848 K-S=0.2226
3. Hack the Correlation and Agreement Test ;)

10. HOW TO WIN THE CHALLENGE:
LEASONS LEARNED
Define p’ = .99 * p ^5000 + .01 * RND and calculate the CVM
CVM= 0.85
Correlation Test : Correlation between classifier output p and mass over a rolling window:
CVM= 0.85
• CVM= 0.0012 !!
• p’ has very similar AUC as p
• p’ for agreement sample ->0 because p’ agreement <<1
• Useless for physics : Just exploiting the background mass gap
AUCw= 0.9999921 CVM=0.0014 K-S=0.0088

11. BREAKING THE RULES: DATA DOPING
• Recipe to build a physically sound classifier:
1. Not to use reconstructed mass, nor features allowing easy mass reconstruction
2. Try to not use variable regions for which the Monte Carlo simulation doesn't agree
with real data
In order to fullfill 2 we have to break the rules and take a look to the control channel
Goal: train a classifier able to separate A
from B, but not C from D
Max(wAUC(A,B)) with KS(C,D)<epsilon
Hypothesis: Control Channel & Analysis
channel share the same MC “defects”
MC Real Data
Control Channel
Analysis Channel
Known
Physics
Known
Physics
New
Physics
Background
A B
C D

12. BREAKING THE RULES: DATA DOPING
• The idea is to "dope" (in the semiconductor meaning) the training set with a small number of Monte
Carlo events from the control channel , but labeled as background.
This disallow the classifier to pick features discriminating data and Monte Carlo.
There are two parameters that regularize the learning:
• The number of "doping" events
• the complexity of the classifier (for instance number of trees)
MC Real Data
Control Channel
Analysis Channel
Known
Physics
Known
Physics
New
Physics
Background
A B
C D

13. BREAKING THE RULES: DATA DOPING
Free classifier Doping events: 2000 Doping events: 3000
Grid search over Classifier complexity (n_ trees) and Number (weight) of doping events
Dammit! A new hyperparameter….

14. BREAKING THE RULES: DATA DOPING
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
0.11
0.12
0.13
0.14
0.15
0.986 0.987 0.988 0.989 0.99 0.991 0.992 0.993 0.994
2500
3000
3100
3200
3500
3900
5000
4000
AUCw= 0.9893 CVM=0.0011 K-S=0.087
CV wAUC
Private LB
KS
Optimal Doping Events: 3960

15. BREAKING THE RULES: DATA DOPING
Analysis Channel Control channel Correlation Test
• Good discriminating power in analysis channel
• No separation for the control channel
• Classifier not correlated with mass
• Probably good for physics….

16. MACHINE LEARNING IN HEP
Today we have
• the right tools
• data availability
• the computer power
• But new physics is difficult to discover unless you know what
are you looking for…
• A complementary approach can be to use unsupervised
learning (only real data driven, we have lots of them)

17. MACHINE LEARNING IN HEP
Example: exploring tau decay at LEP (ALEPH 1993)
(yes, e+ e- physics is cleaner…)
Feeding an autoencoder with “elaborated” detector data
we are able to “discover” different decay modes looking at the
compressed representation without a physics model (MC)
Today is possible to do “end to end” autoencoding from raw
detector data

18. MACHINE LEARNING IN HEP
Example: t-sne with the challenge data: Look at the fine structure….
MC:
Control Channel
Signal
Real Data (all you see is real!)
Control Channel
Background
Test

19. CONCLUSIONS
• Machine learning algorithms alone can fail to discover tiny effects in the data (1777
MeV is only 1.e-4 of the energy at the center of mass)
• Use your knowledge: Try to reduce your data using fundamental simmetries, like
Lorentz invariance, detector geometry...
• Be aware of the test you are using to assure the classifier validity:
• If a test can be “hacked” an enough powerfull machine learning algorithm will find the
way
• If it is possible, try to use non supervised methods( without MC) to gain insight in
your data