SlideShare a Scribd company logo

2 Receiver Operating Characteristic (ROC)

•
•
Dmytro Fishman
Dmytro Fishman

The second lecture from the Machine Learning course series of lectures. This lecture discusses ROC metric for evaluating machine learning model's performance. In particular, two ways of building ROC are discussed. A link to my github (https://github.com/skyfallen/MachineLearningPracticals) with practicals that I have designed for this course in both R and Python. I can share keynote files, contact me via e-mail: dmytro.fishman@ut.ee.

Education
1 of 55
Download to read offline
Introduction to Machine Learning (ROC) Dmytro Fishman (dmytro@ut.ee)
ROCK
ROCK
Receiver Operating Characteristic
(1, 0, 1, 0 ,1) True labels
(1, 0, 1, 0 ,1) True labels (0.6,0.2,0.7,0.5,0.4) Classifier predicts
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) There are as many marks on y-axis as there are 1’s in our true labels
(1, 1, 0, 1, 0) True labels There are as many marks on x-axis as there are 0’s in our true labels (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels Go through true labels one by one, if 1 go up, if 0 go right (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) 1 Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is called Receiver Operating Characteristic (ROC) (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is square has sides of length 1 and 1 (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curve (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curveAUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 Here is another way to do it (not always you can count labels yourself)AUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/N TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/(FP + TN) TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) TPR = TP/P FPR = FP/(FP + TN) TPR FPR
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR We would like to evaluate different strictness levels of our classifier (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = ? FPR = ?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = 1/3 FPR = 0/(0 + 2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 Let’s plot this point on a graph 1/3 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 We shall do this procedure for all possible thresholds 1/3 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = ? FPR = ? How about TPR and FPR?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ? Oops, this is a false positive!
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 And so on…
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
AUC = 0.83 (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 AUC is considered to be more adequate performance measure than accuracy
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 0.5 AUC of 0.5 means random guess
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification overfitting 🙄
References • Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine- learning) • Introduction to Machine Learning by Pascal Vincent given at Deep Learning Summer School, Montreal 2015 (http://videolectures.net/ deeplearning2015_vincent_machine_learning/) • Welcome to Machine Learning by Konstantin Tretyakov delivered at AACIMP Summer School 2015 (http://kt.era.ee/lectures/aacimp2015/1-intro.pdf) • Stanford CS class: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy (http://cs231n.github.io/) • Data Mining Course by Jaak Vilo at University of Tartu (https://courses.cs.ut.ee/ MTAT.03.183/2017_spring/uploads/Main/DM_05_Clustering.pdf) • Machine Learning Essential Conepts by Ilya Kuzovkin (https:// www.slideshare.net/iljakuzovkin) • From the brain to deep learning and back by Raul Vicente Zafra and Ilya Kuzovkin (http://www.uttv.ee/naita?id=23585&keel=eng)
www.biit.cs.ut.ee www.ut.ee www.quretec.ee
You, guys, rock!

Recommended

AI and ML Training for non technical
AI and ML Training for non technicalAI and ML Training for non technical
AI and ML Training for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Universitat Politècnica de Catalunya
 
AI and ML for non technical
AI and ML for non technicalAI and ML for non technical
AI and ML for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la modaDeep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la moda
Javier Abadía
 
Deep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial IntelligenceDeep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial Intelligence
Lukas Masuch
 
DOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biologyDOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biology
Dmytro Fishman
 
Tips for effective presentations
Tips for effective presentationsTips for effective presentations
Tips for effective presentations
Dmytro Fishman
 
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WPAutonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Dmytro Fishman
 

Recommended

AI and ML Training for non technical
AI and ML Training for non technicalAI and ML Training for non technical
AI and ML Training for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Universitat Politècnica de Catalunya
 
AI and ML for non technical
AI and ML for non technicalAI and ML for non technical
AI and ML for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la modaDeep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la moda
Javier Abadía
 
Deep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial IntelligenceDeep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial Intelligence
Lukas Masuch
 
DOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biologyDOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biology
Dmytro Fishman
 
Tips for effective presentations
Tips for effective presentationsTips for effective presentations
Tips for effective presentations
Dmytro Fishman
 
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WPAutonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Dmytro Fishman
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
Dmytro Fishman
 
Introduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/BoltIntroduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/Bolt
Dmytro Fishman
 
Introduction to Gaussian Processes
Introduction to Gaussian ProcessesIntroduction to Gaussian Processes
Introduction to Gaussian Processes
Dmytro Fishman
 
Biit group 2018
Biit group 2018Biit group 2018
Biit group 2018
Dmytro Fishman
 
Detecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep LearningDetecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep Learning
Dmytro Fishman
 
Deep Learning in Healthcare
Deep Learning in HealthcareDeep Learning in Healthcare
Deep Learning in Healthcare
Dmytro Fishman
 
5 Introduction to neural networks
5 Introduction to neural networks5 Introduction to neural networks
5 Introduction to neural networks
Dmytro Fishman
 
4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)
Dmytro Fishman
 
3 Unsupervised learning
3 Unsupervised learning3 Unsupervised learning
3 Unsupervised learning
Dmytro Fishman
 
1 Supervised learning
1 Supervised learning1 Supervised learning
1 Supervised learning
Dmytro Fishman
 
What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?
Dmytro Fishman
 
Machine Learning in Bioinformatics
Machine Learning in BioinformaticsMachine Learning in Bioinformatics
Machine Learning in Bioinformatics
Dmytro Fishman
 
ICT role in 21st century education and it's challenges.
ICT role in 21st century education and it's challenges.ICT role in 21st century education and it's challenges.
ICT role in 21st century education and it's challenges.
MaryamAhmad92
 
Unit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptxUnit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptx
VishalSingh1417
 
psychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docxpsychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docx
PoojaSen20
 
ICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptxICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptx
AreebaZafar22
 
How to Give a Domain for a Field in Odoo 17
How to Give a Domain for a Field in Odoo 17How to Give a Domain for a Field in Odoo 17
How to Give a Domain for a Field in Odoo 17
Celine George
 
Beyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactBeyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global Impact
PECB
 
Class 11th Physics NEET formula sheet pdf
Class 11th Physics NEET formula sheet pdfClass 11th Physics NEET formula sheet pdf
Class 11th Physics NEET formula sheet pdf
AyushMahapatra5
 
Food Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-II
Food Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-IIFood Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-II
Food Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-II
Shubhangi Sonawane
 
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptxBasic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
Denish Jangid
 
Asian American Pacific Islander Month DDSD 2024.pptx
Asian American Pacific Islander Month DDSD 2024.pptxAsian American Pacific Islander Month DDSD 2024.pptx
Asian American Pacific Islander Month DDSD 2024.pptx
David Douglas School District
 

More Related Content

More from Dmytro Fishman

More from Dmytro Fishman (12)

Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
 
Introduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/BoltIntroduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/Bolt
 
Introduction to Gaussian Processes
Introduction to Gaussian ProcessesIntroduction to Gaussian Processes
Introduction to Gaussian Processes
 
Biit group 2018
Biit group 2018Biit group 2018
Biit group 2018
 
Detecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep LearningDetecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep Learning
 
Deep Learning in Healthcare
Deep Learning in HealthcareDeep Learning in Healthcare
Deep Learning in Healthcare
 
5 Introduction to neural networks
5 Introduction to neural networks5 Introduction to neural networks
5 Introduction to neural networks
 
4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)
 
3 Unsupervised learning
3 Unsupervised learning3 Unsupervised learning
3 Unsupervised learning
 
1 Supervised learning
1 Supervised learning1 Supervised learning
1 Supervised learning
 
What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?
 
Machine Learning in Bioinformatics
Machine Learning in BioinformaticsMachine Learning in Bioinformatics
Machine Learning in Bioinformatics
 

Recently uploaded

Beyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactBeyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global Impact
PECB
 
Activity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdfActivity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdf
ciinovamais
 
1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf
QucHHunhnh
 
The basics of sentences session 3pptx.pptx
The basics of sentences session 3pptx.pptxThe basics of sentences session 3pptx.pptx
The basics of sentences session 3pptx.pptx
heathfieldcps1
 

Recently uploaded (20)

ICT role in 21st century education and it's challenges.
ICT role in 21st century education and it's challenges.ICT role in 21st century education and it's challenges.
ICT role in 21st century education and it's challenges.
 
Unit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptxUnit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptx
 
psychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docxpsychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docx
 
ICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptxICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptx
 
How to Give a Domain for a Field in Odoo 17
How to Give a Domain for a Field in Odoo 17How to Give a Domain for a Field in Odoo 17
How to Give a Domain for a Field in Odoo 17
 
Beyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactBeyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global Impact
 
Class 11th Physics NEET formula sheet pdf
Class 11th Physics NEET formula sheet pdfClass 11th Physics NEET formula sheet pdf
Class 11th Physics NEET formula sheet pdf
 
Food Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-II
Food Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-IIFood Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-II
Food Chain and Food Web (Ecosystem) EVS, B. Pharmacy 1st Year, Sem-II
 
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptxBasic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
 
Asian American Pacific Islander Month DDSD 2024.pptx
Asian American Pacific Islander Month DDSD 2024.pptxAsian American Pacific Islander Month DDSD 2024.pptx
Asian American Pacific Islander Month DDSD 2024.pptx
 
Activity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdfActivity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdf
 
Micro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdfMicro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdf
 
Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17
 
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
 
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
 
Ecological Succession. ( ECOSYSTEM, B. Pharmacy, 1st Year, Sem-II, Environmen...
Ecological Succession. ( ECOSYSTEM, B. Pharmacy, 1st Year, Sem-II, Environmen...Ecological Succession. ( ECOSYSTEM, B. Pharmacy, 1st Year, Sem-II, Environmen...
Ecological Succession. ( ECOSYSTEM, B. Pharmacy, 1st Year, Sem-II, Environmen...
 
2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx
2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx
2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx
 
1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy Consulting
 
The basics of sentences session 3pptx.pptx
The basics of sentences session 3pptx.pptxThe basics of sentences session 3pptx.pptx
The basics of sentences session 3pptx.pptx
 

2 Receiver Operating Characteristic (ROC)

  • 5. (1, 0, 1, 0 ,1) True labels
  • 6. (1, 0, 1, 0 ,1) True labels (0.6,0.2,0.7,0.5,0.4) Classifier predicts
  • 7. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2)
  • 8. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) There are as many marks on y-axis as there are 1’s in our true labels
  • 9. (1, 1, 0, 1, 0) True labels There are as many marks on x-axis as there are 0’s in our true labels (0.7,0.6,0.5,0.4,0.2)
  • 10. (1, 1, 0, 1, 0) True labels Go through true labels one by one, if 1 go up, if 0 go right (0.7,0.6,0.5,0.4,0.2)
  • 11. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 12. (1, 1, 0, 1, 0) True labels 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 13. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) 1 Go through true labels one by one, if 1 go up, if 0 go right
  • 14. (1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 15. (1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 16. (1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 17. (1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 18. (1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 19. (1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 20. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 21. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is called Receiver Operating Characteristic (ROC) (0.7,0.6,0.5,0.4,0.2)
  • 22. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is square has sides of length 1 and 1 (0.7,0.6,0.5,0.4,0.2)
  • 23. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curve (0.7,0.6,0.5,0.4,0.2)
  • 24. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curveAUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
  • 25. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 Here is another way to do it (not always you can count labels yourself)AUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
  • 26. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2)
  • 27. (1, 1, 0, 1, 0) True labelsTPR FPR TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
  • 28. (1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/N TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
  • 29. (1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/(FP + TN) TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
  • 30. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) TPR = TP/P FPR = FP/(FP + TN) TPR FPR
  • 31. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR We would like to evaluate different strictness levels of our classifier (0.7,0.6,0.5,0.4,0.2)
  • 32. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
  • 33. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
  • 34. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case?
  • 35. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = ? FPR = ?
  • 36. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = 1/3 FPR = 0/(0 + 2)
  • 37. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0
  • 38. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 Let’s plot this point on a graph 1/3 0
  • 39. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 We shall do this procedure for all possible thresholds 1/3 0
  • 40. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = ? FPR = ? How about TPR and FPR?
  • 41. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0
  • 42. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ?
  • 43. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ? Oops, this is a false positive!
  • 44. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
  • 45. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 And so on…
  • 46. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
  • 47. AUC = 0.83 (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 AUC is considered to be more adequate performance measure than accuracy
  • 48. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 0.5 AUC of 0.5 means random guess
  • 49. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification
  • 50. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification overfitting 🙄
  • 51.
  • 52. References • Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine- learning) • Introduction to Machine Learning by Pascal Vincent given at Deep Learning Summer School, Montreal 2015 (http://videolectures.net/ deeplearning2015_vincent_machine_learning/) • Welcome to Machine Learning by Konstantin Tretyakov delivered at AACIMP Summer School 2015 (http://kt.era.ee/lectures/aacimp2015/1-intro.pdf) • Stanford CS class: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy (http://cs231n.github.io/) • Data Mining Course by Jaak Vilo at University of Tartu (https://courses.cs.ut.ee/ MTAT.03.183/2017_spring/uploads/Main/DM_05_Clustering.pdf) • Machine Learning Essential Conepts by Ilya Kuzovkin (https:// www.slideshare.net/iljakuzovkin) • From the brain to deep learning and back by Raul Vicente Zafra and Ilya Kuzovkin (http://www.uttv.ee/naita?id=23585&keel=eng)
  • 54.