slides for 5-minute summary.

1. MARS: Motion-Augmented RGB Stream for Action
Recognition (CVPR 2019)
MARS: Motion-Augmented RGB Stream for Action
Recognition (CVPR 2019)
Influence-Balanced Loss for
Imbalanced Visual Classification
ASRI, Dept. of Electrical and Computer Engineering
Seoul National University
Seulki Park Jongin Lim Younghan Jeon Jin Young Choi

2. Imbalanced Visual Classification
2
Introduction Proposed Method Experiment Conclusion
Many real-world data often exhibit long-tailed distribution.
✓ The model trained on such imbalanced data tends to overfit the majority classes.
✓ That is, the model performs poorly on minority classes.
Problem Definition:
● Input: Long-tailed (imbalanced) training data & uniform-distributed (balanced) test data.
● Goal: To make a robust model that can generalize well on balanced test data.
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)
Faces (Zhang et al., 2017) Places (Wang et al., 2017) Species (Van Horn et al., 2018) Actions (Zhang et al., 2019)
* Images by authors.

3. Previous Methods
Introduction Proposed Method Experiment Conclusion
1. Data-level approach
◦ Directly balance the training data distributions by re-sampling or generating synthetic samples
(Chawla’02, Mullick’ 19, Hulse’ 07).
◦ Under-sampling majority classes can lose some valuable information.
◦ Over-sampling or data generation is susceptible to overfitting to certain repetitive samples.
2. Meta-learning approach
◦ Recent meta-learning methods have shown promising results (Shu’ 19, Liu’20, Ren’20).
◦ However, these methods are difficult to implement in practice.
▪ Additional unbiased data are required (Shu’19), Meta-sampler is computationally expensive (Ren’20).
3. Re-weighting approach
◦ Assign different weights to each sample according to its importance.
◦ However, they have focused on only global class-level distribution and assign the same weight to all
samples belonging to the same class (CB Loss [Cui’19], LDAM [Cao’19]).
→ “Not all samples in a dataset play an equal role in determining the model parameters. (Cook, 1982)”
3
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)

4. Motivation
Introduction Proposed Method Experiment Conclusion
Q. How can we appropriately re-weight each training samples
while preventing the model from overfitting to majority classes?
- Focal Loss [Lin’17] assigns more weights to hard examples, but most hard examples
belong to majority classes as training progresses.
A. Let’s re-weight samples by their direct influences on making the overfitted model!
→ To measure the influence of a sample, we focus on Influence function (Cook, 1982).
4
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)

5. Motivation
Introduction Proposed Method Experiment Conclusion
Key observations:
✓ Influence of the majority class is much greater than that of the minor class!
Key idea:
✓ Down-weight the samples that have large influence on the overfitted decision
boundary to make a smoother decision boundary!
5
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)
(a) Comparison of Influences
between balanced and imbalanced dataset.
(b) key concept of our approach.

6. Proposed Method
Introduction Proposed Method Experiment Conclusion
0. Recap: Influence Functions on DNNs (Koh and Liang, ICML, 2017)
Let 𝑓(𝑥, 𝜔) a model, 𝐿(𝑦, 𝑓(𝑥, 𝜔)) a loss for a training point (𝑥, 𝑦).
By definition, Influence of (𝑥, 𝑦) on parameters of model (𝜔) is given by:
𝛪 𝑥; 𝜔 = −𝐻−1
∇𝝎𝐿 𝑦, 𝑓 𝑥, 𝜔 ,
where 𝐻 ≝
1
𝑛
σ𝑖=1
𝑛
∇𝜔
2 𝐿 𝑦𝑖, 𝑓 𝑥𝑖, 𝜔
1. Influence-balanced (IB) weighting factor
From 𝛪 𝑥; 𝜔 , we design the IB weighting factor as follows:
IB 𝑥; 𝜔 = ∇𝝎𝐿 𝑦, 𝑓 𝑥, 𝜔 1
6
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)
(1)
(2)

7. Proposed Method
Introduction Proposed Method Experiment Conclusion
2. Influence-balanced (IB) Loss
When using the softmax cross-entropy loss, IB weighting factor can be further simplified:
IB 𝑥; 𝜔 = ∇𝝎𝐿 𝑦, 𝑓 𝑥, 𝜔 1
= 𝑓 𝑥, 𝜔 − 𝑦 1 ⋅ ℎ 1
where ℎ is a hidden feature vector.
Finally, the influence-balanced loss is given by
𝐿𝐼𝐵(𝑦, 𝑓 𝑥, 𝜔 ) =
𝐿(𝑦, 𝑓 𝑥, 𝜔 )
𝑓 𝑥, 𝜔 − 𝑦 1 ⋅ ℎ 1
→ The proposed influence-balanced term constrains the decision boundary to
not overfit to influential majority samples.
7
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)
(3)
(4)

8. Proposed Method
Introduction Proposed Method Experiment Conclusion
3. Influence-balanced (IB) Class-wise Re-weighting
Finally, we add a class-wise re-weighting term 𝜆𝑘 to the IB-loss in 2. as:
𝐿𝐼𝐵(𝜔) =
1
𝑚
෍
𝑥,𝑦 ∈𝐷𝑚
𝜆𝑘
𝐿(𝑦, 𝑓 𝑥, 𝜔 )
𝑓 𝑥, 𝜔 − 𝑦 1 ⋅ ℎ 1
where 𝜆𝑘 = 𝛼
𝑛𝑘
−1
σ𝑘′
𝐾
𝑛𝑘′
−1. (𝑛𝑘: the number of samples in the 𝑘-th class)
→ The class-wise re-weighting can further control the influences depending on the classes.
8
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)
(5)

9. Proposed Method
Introduction Proposed Method Experiment Conclusion
4. Influence-balanced Training Scheme
The influence-balanced training process comprises two phases:
1) normal training and 2) fine-tuning for balance.
9
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)

10. Experimental Results
Introduction Proposed Method Experiment Conclusion
1. Datasets
◦Synthetic data:
▪CIFAR10/100 (10/100 classes), Tiny ImageNet (200 classes)
▪Long-tailed imbalance: the number of samples of 𝑘-th class is set to 𝑛𝑘𝜇𝑘, (𝜇 ∈ 0,1 ).
▪Step-imbalance: the classes are divided into two groups (majority, minority).
◦ Real-world data:
▪iNaturalist 2018: 437,513 images from 8,142 classes (imbalance factor: 500)
※ imbalance factor: the ratio between the most frequent class and the least frequent class.
2. Baselines
We compare our method with the following cost-sensitive loss methods:
◦CE: uses standard cross-entropy loss.
◦Focal Loss [Lin et al., ICCV17]: down-weights well-classified samples and up-weights hard samples.
◦CB Loss [Cui et al., CVPR19]: re-weights the loss inversely proportional to the effective number of samples.
◦LDAM [Cao et al., NeurIPS19]: regularizes the minority classes to have larger margins.
10
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)

11. Comparison with the state-of-the-arts
Introduction Proposed Method Experiment Conclusion
Our method achieves the state-of-the-art results on benchmark datasets with various
imbalance factors.
11
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)
Classification Accuracy (%) of ResNet-32 on imbalanced CIFAR-10 and CIFAR-100 Classification Accuracy (%) of ResNet-18 on
imbalanced Tiny ImageNet.
Classification Accuracy (%) of ResNet-50 on
iNaturalist2018.

12. Class-wise classification accuracy
Introduction Proposed Method Experiment Conclusion
Our proposed method shows that the significant performance improvement has resulted
from the minority classes, not from the majority classes!
12
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)

13. Conclusion
Introduction Proposed Method Experiment Conclusion
✓Discovered that the existing loss-based loss methods can lead a decision boundary of DNNs to
eventually overfit to the majority classes.
✓Designed a novel influence-balanced loss function to re-weight samples more effectively in a
way to alleviate overfitting of decision boundary.
✓Experimentally demonstrated that IB-loss can improve the generalization performance on
imbalanced data.
✓Our method is easy to be implemented and integrated into existing methods.
13
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)

14. Conclusion
Introduction Proposed Method Experiment Conclusion
✓Discovered that the existing loss-based loss methods can lead a decision boundary of DNNs to
eventually overfit to the majority classes.
✓Designed a novel influence-balanced loss function to re-weight samples more effectively in a
way to alleviate overfitting of decision boundary.
✓Experimentally demonstrated that IB-loss can improve the generalization performance on
imbalanced data.
✓Our method is easy to be implemented and integrated into existing methods.
14
Influence-Balanced Loss for Imbalanced Visual Classification (ICCV 2021)
Contact: seulki.park@snu.ac.kr
Code: https://github.com/pseulki/IB-Loss