Mobile applications have become a popular software development domain in recent years due in part to a large user base, capable hardware, and accessible platforms. However, mobile developers also face unique challenges, including pressure for frequent releases to keep pace with rapid platform evolution, hardware iteration, and user feedback. Due to this rapid pace of evolution, developers need automated support for documenting the changes made to their apps in order to aid in program comprehension. One of the more challenging types of changes to document in mobile apps are those made to the graphical user interface (GUI) due to its abstract, pixel-based representation. In this paper, we present a fully automated approach, called Gcat, for detecting and summarizing GUI changes during the evolution of mobile apps. Gcat leverages computer vision techniques and natural language generation to accurately and concisely summarize changes made to the GUI of a mobile app between successive commits or releases. We evaluate the performance of our approach in terms of its precision and recall in detecting GUI changes compared to developer specified changes, and investigate the utility of the generated change reports in a controlled user study. Our results indicate that Gcat is capable of accurately detecting and classifying GUI changes – outperforming developers – while providing useful documentation.

1. Kevin Moran,
Cody Watson,
John Hoskins,
George Purnell, &
Denys Poshyvanyk
Detecting and Summarizing GUI
Changes in Evolving Mobile Apps
33rd International Conference
on Automated Software Engineering (ASE’18)
Thursday, September 6th, 2018

3. !3

4. !3

5. COMPREHENDING MOBILE GUIS

6. COMPREHENDING MOBILE GUIS

7. COMPREHENDING MOBILE GUIS
<PreferenceScreen
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:key="main"
android:title="@string/account_settings_title_fmt">
<PreferenceScreen
android:key="account_settings"
android:title="@string/account_settings_general_title">
<com.takisoft.fix.support.v7.preference.AutoSummaryEditTextPreference
android:dialogTitle="@string/account_settings_description_label"
android:key="account_description"
android:title="@string/account_settings_description_label" />
<CheckBoxPreference
android:key="account_default"
android:summary="@string/account_settings_default_summary"
android:title="@string/account_settings_default_label" />
<com.fsck.k9.ui.settings.account.HoloColorPickerPreference
android:key="chip_color"
android:summary="@string/account_settings_color_summary"
android:title="@string/account_settings_color_label" />
</PreferenceScreen>

8. COMPREHENDING MOBILE GUIS
<PreferenceScreen
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:key="main"
android:title="@string/account_settings_title_fmt">
<PreferenceScreen
android:key="account_settings"
android:title="@string/account_settings_general_title">
<com.takisoft.fix.support.v7.preference.AutoSummaryEditTextPreference
android:dialogTitle="@string/account_settings_description_label"
android:key="account_description"
android:title="@string/account_settings_description_label" />
<CheckBoxPreference
android:key="account_default"
android:summary="@string/account_settings_default_summary"
android:title="@string/account_settings_default_label" />
<com.fsck.k9.ui.settings.account.HoloColorPickerPreference
android:key="chip_color"
android:summary="@string/account_settings_color_summary"
android:title="@string/account_settings_color_label" />
</PreferenceScreen>

9. COMPREHENDING MOBILE GUIS

10. COMPREHENDING MOBILE GUIS

11. class GeneralSettingsActivity : K9Activity(), OnPreferenceStartScreenCallback {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.general_settings)
initializeActionBar()
if (savedInstanceState == null) {
fragmentTransaction {
add(R.id.generalSettingsContainer, GeneralSettingsFragment.create())
}
}
}
private fun initializeActionBar() {
val actionBar = supportActionBar ?: throw
RuntimeException("getSupportActionBar() == null”)
actionBar.setDisplayHomeAsUpEnabled(true)
}
COMPREHENDING MOBILE GUIS

12. COMPREHENDING MOBILE GUIS

13. COMPREHENDING MOBILE GUIS

16. Video Source: Google

17. Video Source: Google

18. COMPLICATIONS OF FRAGMENTATION
https://opensignal.com/reports/2015/08/android-fragmentation/

20. Developers need support in
identifying and reasoning about GUI
related changes in mobile apps

21. EXISTING WORK

22. EXISTING WORK
Detecting
Presentation
Failures in
Web Apps

23. EXISTING WORK
Detecting
Presentation
Failures in
Web Apps
Automated
Source Code
Summarization

24. EXISTING WORK
Detecting
Presentation
Failures in
Web Apps
GUI
Differencing
Automated
Source Code
Summarization

25. EXISTING WORK
Detecting
Presentation
Failures in
Web Apps
GUI
Differencing
Automated
Source Code
Summarization
Cross
Browser
Testing

26. EXISTING WORK
Detecting
Presentation
Failures in
Web Apps
Detecting Design
Violations in
Mobile Apps
GUI
Differencing
Automated
Source Code
Summarization
Cross
Browser
Testing

27. CURRENT LIMITATIONS

28. CURRENT LIMITATIONS
• No approach combines:

29. CURRENT LIMITATIONS
• No approach combines:
• Automated dynamic analysis

30. CURRENT LIMITATIONS
• No approach combines:
• Automated dynamic analysis
• CV-based GUI analysis

31. CURRENT LIMITATIONS
• No approach combines:
• Automated dynamic analysis
• CV-based GUI analysis
• Automated summarization

32. PRELIMINARY INDUSTRIAL EMPIRICAL STUDY1
71Screens with
DesignViolations
1Moran, K., Li, B., Bernal-Cárdenas, C., Jelf, D., and Poshyvanyk, D., “Automated Reporting of GUI Design Violations for Mobile Apps”, (ICSE’18)

33. PRELIMINARY INDUSTRIAL EMPIRICAL STUDY1
71Screens with
DesignViolations
Authors
1Moran, K., Li, B., Bernal-Cárdenas, C., Jelf, D., and Poshyvanyk, D., “Automated Reporting of GUI Design Violations for Mobile Apps”, (ICSE’18)

34. PRELIMINARY INDUSTRIAL EMPIRICAL STUDY1
71Screens with
DesignViolations
Authors Open Coding
1Moran, K., Li, B., Bernal-Cárdenas, C., Jelf, D., and Poshyvanyk, D., “Automated Reporting of GUI Design Violations for Mobile Apps”, (ICSE’18)

35. PRELIMINARY INDUSTRIAL EMPIRICAL STUDY1
71Screens with
DesignViolations
Authors Open Coding
1Moran, K., Li, B., Bernal-Cárdenas, C., Jelf, D., and Poshyvanyk, D., “Automated Reporting of GUI Design Violations for Mobile Apps”, (ICSE’18)

36. PRELIMINARY INDUSTRIAL EMPIRICAL STUDY1
71Screens with
DesignViolations
Authors Open Coding
GUI-Change
Taxonomy
25%
1Moran, K., Li, B., Bernal-Cárdenas, C., Jelf, D., and Poshyvanyk, D., “Automated Reporting of GUI Design Violations for Mobile Apps”, (ICSE’18)

37. GUI-CHANGE TAXONOMY
Change Category Specific Change
Text Changes
Text Content Change
Font Change
Font Color Change
Layout Changes
VerticalTranslation
HorizontalTranslation
Horizontal Size
Vertical Size
Resource Changes
Image Color Change
Removed Component
Added Component
Image Change
ComponentType Change

38. GUI-CHANGE TAXONOMY
Change Category Specific Change
Text Changes
Text Content Change
Font Change
Font Color Change
Layout Changes
VerticalTranslation
HorizontalTranslation
Horizontal Size
Vertical Size
Resource Changes
Image Color Change
Removed Component
Added Component
Image Change
ComponentType Change

39. GUI-CHANGE TAXONOMY
Change Category Specific Change
Text Changes
Text Content Change
Font Change
Font Color Change
Layout Changes
VerticalTranslation
HorizontalTranslation
Horizontal Size
Vertical Size
Resource Changes
Image Color Change
Removed Component
Added Component
Image Change
ComponentType Change

40. GUI-CHANGE TAXONOMY
Change Category Specific Change
Text Changes
Text Content Change
Font Change
Font Color Change
Layout Changes
VerticalTranslation
HorizontalTranslation
Horizontal Size
Vertical Size
Resource Changes
Image Color Change
Removed Component
Added Component
Image Change
ComponentType Change

41. GUI-CHANGE TAXONOMY
Change Category Specific Change
Text Changes
Text Content Change
Font Change
Font Color Change
Layout Changes
VerticalTranslation
HorizontalTranslation
Horizontal Size
Vertical Size
Resource Changes
Image Color Change
Removed Component
Added Component
Image Change
ComponentType Change

42. GUI-CHANGE TAXONOMY
Change Category Specific Change
Text Changes
Text Content Change
Font Change
Font Color Change
Layout Changes
VerticalTranslation
HorizontalTranslation
Horizontal Size
Vertical Size
Resource Changes
Image Color Change
Removed Component
Added Component
Image Change
ComponentType Change

43. GUI-CHANGE TAXONOMY
Change Category Specific Change
Text Changes
Text Content Change
Font Change
Font Color Change
Layout Changes
VerticalTranslation
HorizontalTranslation
Horizontal Size
Vertical Size
Resource Changes
Image Color Change
Removed Component
Added Component
Image Change
ComponentType Change

44. GUI CHANGE ANALYSIS TOOL (GCAT)
Git/CI
Integration
Project History
Virtual or Physical Device Pool
N1
N2
N3
N4
N5T7
T3
T17
T35
T12
T25
T42
T59
T29
T21
N6
CrashScope
Automated
Execution
Engine
Screenshot & Metadata
Pairs - Commit i
Screenshot & Metadata
Pairs - Commit i+n
Screen Matching
& Filtering
Activity/
Window
Comparator
Bipartite
Image
Comparison
GUI-Hierarchy for
Commit i
GUI-Hierarchy for
Commit i+n
Leaf Node Component
Detection and Matching
GUI Change Resolution
Perceptual Image
Differencing (PID)
GUI-Change Report
Layout Change
Detection
Text Change
Detection
Resource Change
Detection
NL Summary Generator
Checks:
<x>,<y>
Checks:
<width>,<height>
Normalized
String
Comparison
Image
Comparison
Analysis
Commit i
Commit i+1
Commit i+2
Commit i+n

45. GUI CHANGE ANALYSIS TOOL (GCAT)
1) CI/CD Integration
Git/CI
Integration
Project History
Virtual or Physical Device Pool
N1
N2
N3
N4
N5T7
T3
T17
T35
T12
T25
T42
T59
T29
T21
N6
CrashScope
Automated
Execution
Engine
Screenshot & Metadata
Pairs - Commit i
Screenshot & Metadata
Pairs - Commit i+n
Screen Matching
& Filtering
Activity/
Window
Comparator
Bipartite
Image
Comparison
GUI-Hierarchy for
Commit i
GUI-Hierarchy for
Commit i+n
Leaf Node Component
Detection and Matching
GUI Change Resolution
Perceptual Image
Differencing (PID)
GUI-Change Report
Layout Change
Detection
Text Change
Detection
Resource Change
Detection
NL Summary Generator
Checks:
<x>,<y>
Checks:
<width>,<height>
Normalized
String
Comparison
Image
Comparison
Analysis
Commit i
Commit i+1
Commit i+2
Commit i+n

46. GUI CHANGE ANALYSIS TOOL (GCAT)
1) CI/CD Integration
2) Automated GUI-based
Exploration
Git/CI
Integration
Project History
Virtual or Physical Device Pool
N1
N2
N3
N4
N5T7
T3
T17
T35
T12
T25
T42
T59
T29
T21
N6
CrashScope
Automated
Execution
Engine
Screenshot & Metadata
Pairs - Commit i
Screenshot & Metadata
Pairs - Commit i+n
Screen Matching
& Filtering
Activity/
Window
Comparator
Bipartite
Image
Comparison
GUI-Hierarchy for
Commit i
GUI-Hierarchy for
Commit i+n
Leaf Node Component
Detection and Matching
GUI Change Resolution
Perceptual Image
Differencing (PID)
GUI-Change Report
Layout Change
Detection
Text Change
Detection
Resource Change
Detection
NL Summary Generator
Checks:
<x>,<y>
Checks:
<width>,<height>
Normalized
String
Comparison
Image
Comparison
Analysis
Commit i
Commit i+1
Commit i+2
Commit i+n

47. GUI CHANGE ANALYSIS TOOL (GCAT)
1) CI/CD Integration
2) Automated GUI-based
Exploration
3) GUI Interpretation &
Change Detection
Git/CI
Integration
Project History
Virtual or Physical Device Pool
N1
N2
N3
N4
N5T7
T3
T17
T35
T12
T25
T42
T59
T29
T21
N6
CrashScope
Automated
Execution
Engine
Screenshot & Metadata
Pairs - Commit i
Screenshot & Metadata
Pairs - Commit i+n
Screen Matching
& Filtering
Activity/
Window
Comparator
Bipartite
Image
Comparison
GUI-Hierarchy for
Commit i
GUI-Hierarchy for
Commit i+n
Leaf Node Component
Detection and Matching
GUI Change Resolution
Perceptual Image
Differencing (PID)
GUI-Change Report
Layout Change
Detection
Text Change
Detection
Resource Change
Detection
NL Summary Generator
Checks:
<x>,<y>
Checks:
<width>,<height>
Normalized
String
Comparison
Image
Comparison
Analysis
Commit i
Commit i+1
Commit i+2
Commit i+n

48. STAGE 1: CI/CD INTEGRATION
Git/CI
Integration
Project History
Virtual or Physical Device Po
N1
N2
N3
N4
T7
T3
T17
T35
T12
T25
T42
T
T29
T21
N6
CrashScope
Automated
Execution
Engine
Commit i
Commit i+1
Commit i+2
Commit i+n

49. STAGE 1: CI/CD INTEGRATION
Git/CI
Integration
Project History
Virtual or Physical Device Po
N1
N2
N3
N4
T7
T3
T17
T35
T12
T25
T42
T
T29
T21
N6
CrashScope
Automated
Execution
Engine
Commit i
Commit i+1
Commit i+2
Commit i+n

50. STAGE 1: CI/CD INTEGRATION
Git/CI
Integration
Project History
Virtual or Physical Device Po
N1
N2
N3
N4
T7
T3
T17
T35
T12
T25
T42
T
T29
T21
N6
CrashScope
Automated
Execution
Engine
Commit i
Commit i+1
Commit i+2
Commit i+n

51. STAGE 2:AUTOMATED DYNAMIC ANALYSIS

52. STAGE 2:AUTOMATED DYNAMIC ANALYSIS
CrashScope1
Execution
Engine
1Moran, K., Linares-Vásquez, M., Bernal-Cárdenas, C., Vendome, C., and
Poshyvanyk, D., “Automatically Discovering, Reporting and Reproducing
Android Application Crashes”, ICST’16

53. STAGE 2:AUTOMATED DYNAMIC ANALYSIS
CrashScope1
Execution
Engine
Execution Engine
1Moran, K., Linares-Vásquez, M., Bernal-Cárdenas, C., Vendome, C., and
Poshyvanyk, D., “Automatically Discovering, Reporting and Reproducing
Android Application Crashes”, ICST’16

54. STAGE 2:AUTOMATED DYNAMIC ANALYSIS
CrashScope1
Execution
Engine
Execution Engine
1Moran, K., Linares-Vásquez, M., Bernal-Cárdenas, C., Vendome, C., and
Poshyvanyk, D., “Automatically Discovering, Reporting and Reproducing
Android Application Crashes”, ICST’16

55. STAGE 2:AUTOMATED DYNAMIC ANALYSIS
CrashScope1
Execution
Engine
Execution Engine
1Moran, K., Linares-Vásquez, M., Bernal-Cárdenas, C., Vendome, C., and
Poshyvanyk, D., “Automatically Discovering, Reporting and Reproducing
Android Application Crashes”, ICST’16

56. STAGE 2: SCREEN MATCHING AND FILTERING
Older Commit Newer Commit

57. STAGE 2: SCREEN MATCHING AND FILTERING
GUI Color
Histogram Analysis
Spatial GUI
Component
Analysis
Older Commit Newer Commit
Discard Non-Unique
Activity-Window Pairs

58. STAGE 2: SCREEN MATCHING AND FILTERING
GUI Color
Histogram Analysis
Spatial GUI
Component
Analysis
Older Commit Newer Commit
Discard Non-Unique
Activity-Window Pairs

59. STAGE 2: SCREEN MATCHING AND FILTERING
GUI Color
Histogram Analysis
Spatial GUI
Component
Analysis
Older Commit Newer Commit Matched Screen Pairs
≈
≈
≈
Discard Non-Unique
Activity-Window Pairs

60. STAGE3: GUI COMPREHENSION
Old Commit New Commit

61. STAGE3: GUI COMPREHENSION
Old Commit New CommitSpatial
Component
Matching

62. = (|xm xr| + |ym yr| + |wm wr| + |hm hr|)<latexit sha1_base64="Ht7153Xmi/4IY79Asy9xCp7Yq18=">AAACKHicdVDLSgMxFM3UV62vqks3wSIo4pCpr3YhFN24VLBaaMuQSdNOMJkZkox2aPs7bvwVNwoqbv0S03EEFb0QOI97ubnHizhTGqE3KzcxOTU9k58tzM0vLC4Vl1cuVRhLQusk5KFseFhRzgJa10xz2ogkxcLj9Mq7Phn7VzdUKhYGFzqJaFvgXsC6jGBtJLdYa/WwEBgewc1h3xVwB/ZdOYTbcJikLMnYbcpuM+anzDdsyy2WkF2uOPsHDkT2LkKHCBlQrh44lTJ0bJRWCWR15hafWp2QxIIGmnCsVNNBkW4PsNSMcDoqtGJFI0yucY82DQywoKo9SC8dwQ2jdGA3lOYFGqbq94kBFkolwjOdAmtf/fbG4l9eM9bdSnvAgijWNCCfi7oxhzqE49hgh0lKNE8MwEQy81dIfCwx0Sbcggnh61L4P6iX7aqNzvdKteMsjTxYA+tgEzjgENTAKTgDdUDAHXgAz+DFurcerVfr7bM1Z2Uzq+BHWe8fdYKjUQ==</latexit><latexit sha1_base64="Ht7153Xmi/4IY79Asy9xCp7Yq18=">AAACKHicdVDLSgMxFM3UV62vqks3wSIo4pCpr3YhFN24VLBaaMuQSdNOMJkZkox2aPs7bvwVNwoqbv0S03EEFb0QOI97ubnHizhTGqE3KzcxOTU9k58tzM0vLC4Vl1cuVRhLQusk5KFseFhRzgJa10xz2ogkxcLj9Mq7Phn7VzdUKhYGFzqJaFvgXsC6jGBtJLdYa/WwEBgewc1h3xVwB/ZdOYTbcJikLMnYbcpuM+anzDdsyy2WkF2uOPsHDkT2LkKHCBlQrh44lTJ0bJRWCWR15hafWp2QxIIGmnCsVNNBkW4PsNSMcDoqtGJFI0yucY82DQywoKo9SC8dwQ2jdGA3lOYFGqbq94kBFkolwjOdAmtf/fbG4l9eM9bdSnvAgijWNCCfi7oxhzqE49hgh0lKNE8MwEQy81dIfCwx0Sbcggnh61L4P6iX7aqNzvdKteMsjTxYA+tgEzjgENTAKTgDdUDAHXgAz+DFurcerVfr7bM1Z2Uzq+BHWe8fdYKjUQ==</latexit><latexit sha1_base64="Ht7153Xmi/4IY79Asy9xCp7Yq18=">AAACKHicdVDLSgMxFM3UV62vqks3wSIo4pCpr3YhFN24VLBaaMuQSdNOMJkZkox2aPs7bvwVNwoqbv0S03EEFb0QOI97ubnHizhTGqE3KzcxOTU9k58tzM0vLC4Vl1cuVRhLQusk5KFseFhRzgJa10xz2ogkxcLj9Mq7Phn7VzdUKhYGFzqJaFvgXsC6jGBtJLdYa/WwEBgewc1h3xVwB/ZdOYTbcJikLMnYbcpuM+anzDdsyy2WkF2uOPsHDkT2LkKHCBlQrh44lTJ0bJRWCWR15hafWp2QxIIGmnCsVNNBkW4PsNSMcDoqtGJFI0yucY82DQywoKo9SC8dwQ2jdGA3lOYFGqbq94kBFkolwjOdAmtf/fbG4l9eM9bdSnvAgijWNCCfi7oxhzqE49hgh0lKNE8MwEQy81dIfCwx0Sbcggnh61L4P6iX7aqNzvdKteMsjTxYA+tgEzjgENTAKTgDdUDAHXgAz+DFurcerVfr7bM1Z2Uzq+BHWe8fdYKjUQ==</latexit>
≊
≊
≊
≊
STAGE3: GUI COMPREHENSION
Old Commit New CommitSpatial
Component
Matching

63. Perceptual
Image
Differencing
(PID)
STAGE 3: GUI-CHANGE DETECTION

64. Perceptual
Image
Differencing
(PID)
STAGE 3: GUI-CHANGE DETECTION

65. Perceptual
Image
Differencing
(PID)
STAGE 3: GUI-CHANGE DETECTION

66. ≊
≊
≊
≊
≊
STAGE 3: GUI-CHANGE DETECTION

67. ≊
≊
≊
≊
≊
STAGE 3: GUI-CHANGE DETECTION

68. ≊
≊
≊
≊
≊
STAGE 3: GUI-CHANGE DETECTION

69. List of Potential
GUI Changes
STAGE 3: GUI CHANGE RESOLUTION

70. List of Potential
GUI Changes
STAGE 3: GUI CHANGE RESOLUTION

71. List of Potential
GUI Changes
Layout Change Resolver
• <x,y>
• <height,width>
Text Change Resolver
• <image>
• <text> CHA + CQ
Resource Change Resolver
• <image>
• Missing/
Extra CHA + CQ
STAGE 3: GUI CHANGE RESOLUTION

72. GCAT REPORT

73. GVT REPORT

74. GVT REPORT

75. EMPIRICAL STUDY

76. EMPIRICAL STUDY
• RQ1: Performance of Screen Matching and Filtering?

77. EMPIRICAL STUDY
• RQ1: Performance of Screen Matching and Filtering?
• RQ2: GCAT Detecting and Classifying GUI-Changes?

78. EMPIRICAL STUDY
• RQ1: Performance of Screen Matching and Filtering?
• RQ2: GCAT Detecting and Classifying GUI-Changes?

79. EMPIRICAL STUDY
• RQ1: Performance of Screen Matching and Filtering?
• RQ2: GCAT Detecting and Classifying GUI-Changes?
• RQ3: Developers Detecting and Classifying GUI-Changes?

80. EMPIRICAL STUDY
• RQ1: Performance of Screen Matching and Filtering?
• RQ2: GCAT Detecting and Classifying GUI-Changes?
• RQ3: Developers Detecting and Classifying GUI-Changes?
• RQ4: Developer Utility?

81. STUDY CONTEXT
FDroid
31 Open Source Apps (62 Apks)
Screen Collection Methodology

82. STUDY CONTEXT
FDroid
• Compilable Subsequent
Commits?
31 Open Source Apps (62 Apks)
Screen Collection Methodology

83. STUDY CONTEXT
FDroid
• Compilable Subsequent
Commits?
• Compiled Commits run
on target Nexus 7 device?
31 Open Source Apps (62 Apks)
Screen Collection Methodology

84. STUDY CONTEXT
FDroid
• Compilable Subsequent
Commits?
• Compiled Commits run
on target Nexus 7 device?
• GUI Changes exist on at
least 2 screens?
31 Open Source Apps (62 Apks)
Screen Collection Methodology

85. RQ1 & RQ2: EFFECTIVENESS AND PERFORMANCE
31 Open Source Apps (62 Apks)

86. RQ1 & RQ2: EFFECTIVENESS AND PERFORMANCE
31 Open Source Apps (62 Apks)
CrashScope
Execution
Engine

87. RQ1 & RQ2: EFFECTIVENESS AND PERFORMANCE
31 Open Source Apps (62 Apks)
CrashScope
Execution
Engine
3,854
Extracted
Screens

88. RQ1 & RQ2: EFFECTIVENESS AND PERFORMANCE
3,854
Extracted
Screens

89. RQ1 & RQ2: EFFECTIVENESS AND PERFORMANCE
3,854
Extracted
Screens
GUI Color
Histogram Analysis
Spatial Analysis
Screen Matching & Filtering
Discard Non-Unique
Activity-Window Pairs

90. RQ1 & RQ2: EFFECTIVENESS AND PERFORMANCE
3,854
Extracted
Screens
Matched Screens
• High # of Changes
• Medium # of Changes
• Low # of Changes
GUI Color
Histogram Analysis
Spatial Analysis
Screen Matching & Filtering
Discard Non-Unique
Activity-Window Pairs

91. RQ1 & RQ2: RESULTS
% Filtered Screens Matching Precision
(3,538 screens)
91.8%
(134/151 pairs)
84.4%
Screen Filtering & Matching

92. RQ1 & RQ2: RESULTS
% Filtered Screens Matching Precision
(3,538 screens)
91.8%
(134/151 pairs)
84.4%
Screen Filtering & Matching
R
CP
DP
0 25 50 75 100
GUI Change Detection and Classification
Detection Precision (DP)
Classification Precision (CP)
Recall (R)
*Random
Sample
of 18
Screen Pairs

93. RQ2 & RQ3: DEVELOPER PERFORMANCE & UTILITY
Layout Changes
TextChanges
Resource Changes
20 Academic and
Industrial Developers
25%
CP, DP, R
Usability and
User Preference
Questions

94. RQ2 & RQ3: EFFECTIVENESS AND PERFORMANCE
31 Open Source Apps (62 Apks)
Manual
Author
Screen
Analysis
50 Screen Paris
Split Among High,
Medium and Low
# of changes

95. RQ2 & RQ3: RESULTS
R
CP
DP
0 25 50 75 100
Detection Precision (DP)
Classification Precision (CP)
Recall (R)
Developer Performance

96. RQ2 & RQ3: RESULTS
R
CP
DP
0 25 50 75 100
Detection Precision (DP)
Classification Precision (CP)
Recall (R)
Developer Performance
Easy to understand?
Reports were Useful?
Would Ike to use often?
Hard to Read?
Too Complex?
0 1.25 2.5 3.75 5
AgreeDisagree
Developer Utility

97. RQ2 & RQ3: RESULTS

98. “The detailed changes [are best], since they show the status of the
elements before/after the changes. ”
RQ2 & RQ3: RESULTS

99. “The detailed changes [are best], since they show the status of the
elements before/after the changes. ”
“Seeing how components moved with the highlighted red
boxes [was most helpful to me].”
RQ2 & RQ3: RESULTS

100. “The detailed changes [are best], since they show the status of the
elements before/after the changes. ”
“Seeing how components moved with the highlighted red
boxes [was most helpful to me].”
“The spanning tree, I like the way it describes
changes in tree structure to make it clear and
straightforward. ”
RQ2 & RQ3: RESULTS

101. “The detailed changes [are best], since they show the status of the
elements before/after the changes. ”
“A way to group the changes or a potential importance
(e.g., a new component may be more important as
compared to a 2px change).”
“Seeing how components moved with the highlighted red
boxes [was most helpful to me].”
“The spanning tree, I like the way it describes
changes in tree structure to make it clear and
straightforward. ”
RQ2 & RQ3: RESULTS

102. AVENUES FOR FUTURE WORK

103. • Develop Static AnalysisTechniques for integrating GUI code
information into change reports
AVENUES FOR FUTURE WORK

104. • Develop Static AnalysisTechniques for integrating GUI code
information into change reports
• Integrate GUI change information into IDEs to aid in program
comprehension
AVENUES FOR FUTURE WORK

105. • Develop Static AnalysisTechniques for integrating GUI code
information into change reports
• Integrate GUI change information into IDEs to aid in program
comprehension
• DevelopTechniques for effectively handling dynamic screen content
AVENUES FOR FUTURE WORK

106. • Develop Static AnalysisTechniques for integrating GUI code
information into change reports
• Integrate GUI change information into IDEs to aid in program
comprehension
• DevelopTechniques for effectively handling dynamic screen content
• Prioritizing the Importance of changes for more effective
summarization
AVENUES FOR FUTURE WORK

107. https://www.android-dev-tools.com/gcat/

113. Any Questions?
Thank you!
Kevin Moran
Post-Doctoral Fellow
College of William & Mary
@kevpmo
kpmoran@cs.wm.edu
https://www.kpmoran.com

114. ADDITIONAL SLIDES

115. GUI CHANGE AMBIGUITIES
a) Previous application menu bar
b) New application menu bar
The was replaced by the , the was replaced by the
and the was added to the GUI.
The and were shifted to the left, and the was added to the GUI.