Tool Competition Introduction NLP-based approaches and tools have been proposed to improve the efficiency of software engineers, processes, and products, by automatically processing natural language artifacts (issues, emails, commits, etc.). We believe that the availability of accurate tools is becoming increasingly necessary to improve Software Engineering (SE) processes. One important process is issue management and prioritization where developers have to understand, classify, prioritize, assign, etc. incoming issues reported by end-users and developers. This year, we are pleased to announce the first edition of the NLBSE’22 tool competition on issue report classification, an important task in issue management and prioritization. For the competition, we provide a dataset encompassing more than 800k labeled issue reports (as bugs, enhancements, and questions) extracted from real open-source projects. You are invited to leverage this dataset for evaluating your classification approaches and compare the achieved results against a proposed baseline approach (based on FastText). Competition overview We created a Colab notebook with detailed information about the competition (provided data, baseline approach, paper submission, paper format, etc.). If you want to participate, you must: Train and tune a multi-label multi-class classifier using the provided training set. The classifier should assign one label to an issue. Evaluate your classifier on the provided test set Write a paper (4 pages max.) describing: The architecture and details of the classifier The procedure used to pre-process the data The procedure used to tune the classifier on the training set The results of your classifier on the test set Additional info.: provide a link to your code/tool with proper documentation on how to run it Submit the paper by emailing the tool competition organizers (see below) Submissions will be evaluated and accepted based on correctness and reproducibility, defined by the following criteria: Clarity and detail of the paper content Availability of the code/tool, released as open-source Correct training/tuning/evaluation of your code/tool on the provided data Clarity of the code documentation The accepted submissions will be published at the workshop proceedings. The submissions will be ranked based on the F1 score achieved by the proposed classifiers on the test set, as indicated in the papers. The submission with the highest F1 score will be the winner of the competition. How to participate? Email your paper to Oscar Chaparro (oscarch@wm.edu) and Rafael Kallis (rk@rafaelkallis.com) by the submission deadline.

1. @NLBSE_workshop nlbse2022.github.io/tools
NLBSE’22: Tool Competition
Oscar Chaparro
Rafael Kallis
College of
William & Mary
USA
Rafael Kallis
Consulting
Switzerland

2. @NLBSE_workshop nlbse2022.github.io/tools
The competition at a glance
Goal: develop more accurate models for issue classification
Baseline model: TicketTagger
Dataset: 800k+ issue reports from 127k+ GitHub projects
Competitors: 5 teams

3. @NLBSE_workshop nlbse2022.github.io/tools
Issue report classification
Issue
report
Classification
model
Bug
Enhancement
Question
Important task in issue management and prioritization
Extensive research in the field that uses NLP/ML

4. @NLBSE_workshop nlbse2022.github.io/tools
Baseline model: TicketTagger
Rafael Kallis et al., Ticket tagger: Machine learning driven issue classification, ICSME’19
Rafael Kallis et al., Predicting issue types on GitHub, Science of Computer Programming, 2021
Bug
Enhancement
Question
Issue title &
description

5. @NLBSE_workshop nlbse2022.github.io/tools
Benchmark dataset
800k+ issues from 127k+ GitHub projects
Closed issues with any of the 3 labels using Google BigQuery
• Label (aka issue type)
• Title and description
• URL (issue and repository), timestamp
• Author type (owner, contributor, etc.)

6. @NLBSE_workshop nlbse2022.github.io/tools
Benchmark dataset
50.0%
41.4%
8.6%
% of issues
Bug Enhancement Question
Training set: 90% (723k)
Testing set: 10% (80.5k)

7. @NLBSE_workshop nlbse2022.github.io/tools
Infrastructure

8. @NLBSE_workshop nlbse2022.github.io/tools
Competition rules
Training and fine-tuning on the training set
Model classification accuracy on the testing set
Preprocessing and manipulation on the training set
Feature engineering, data balancing, validation set, etc.
No balancing of the testing set

9. @NLBSE_workshop nlbse2022.github.io/tools
Metrics
Precision, recall, and
F1-score for each label
Micro average F1-score to
declare the winner

10. @NLBSE_workshop nlbse2022.github.io/tools
Competitors
Team 4
(Colavito et al.)
Team 3
(Trautsch & Herbold)
Team 2
(Bharadwaj & Kadam)
Team 5
(Izadi)
Team 1
(Siddiq & Santos)

11. @NLBSE_workshop nlbse2022.github.io/tools
Submitted tools at a glance
BERT*
XLNet
MLP
Log. Regression
Title
Description
Repository
Timestamp
Author
Text
normalization
Duplicate
removal
…

12. @NLBSE_workshop nlbse2022.github.io/tools
Classification results
0.872
0.865
0.859 0.858 0.857
0.818
0.79
0.8
0.81
0.82
0.83
0.84
0.85
0.86
0.87
0.88
Micro avg. F1-score
Team A Team B Team C Team D Team E Baseline

13. @NLBSE_workshop nlbse2022.github.io/tools
Results for the best model
0.84 0.874
0.72
0.897 0.885 0.879
0.72
0.664 0.691
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision Recall F1-score
Bug Enhancement Question

14. @NLBSE_workshop nlbse2022.github.io/tools
Competition ranking
Team 4
(Colavito et al.)
Team 3
(Trautsch & Herbold)
Team 1
(Siddiq & Santos)
3

15. @NLBSE_workshop nlbse2022.github.io/tools
Competition ranking
Team 2
(Bharadwaj & Kadam)
Team 5
(Izadi)
1
2

16. @NLBSE_workshop nlbse2022.github.io/tools
Tool presentations
Motivation for choosing models
Challenges during model training/evaluation
Features that most contributed to performance
Preprocessing pipeline and their effect on performance
Examples of failing and successful predicted cases
Ideas on a customized model to improve performance

17. @NLBSE_workshop nlbse2022.github.io/tools
Discussion panel
ADDITIONAL Q&A FEEDBACK ON
COMPETITION
IDEAS FOR THE
NEXT EDITION