TOWARDS AN IDE-BASED CONTEXT-AWARE META SEARCH ENGINE FOR EXCEPTION RECOMMENDATION

TOWARDS A CONTEXT-AWARE META SEARCH
ENGINE FOR IDE-BASED RECOMMENDATION
ABOUT PROGRAMMING ERRORS &
EXCEPTIONS
Mohammad Masudur Rahman, Shamima Yeasmin, and
Chanchal K. Roy
Department of Computer Science
University of Saskatchewan
CSMR-18/WCRE-21 Software Evolution Week (SEW
2014), Antwerp, Belgium

SOFTWARE MAINTENANCE, BUGS &
EXCEPTIONS
2
SoftwareResearchLab,UofS

EXCEPTION HANDLING: IDE SUPPORT
1
2
3

EXCEPTION SEARCH QUERY
4

EXCEPTION HANDLING: WEB SEARCH
Traditional web search
•No ties between IDE and web
browsers
•Does not consider problem-context
•Environment-switching is distracting
& Time consuming
•Often not much productive (trial &
error approach)
5

IDE-BASED WEB SEARCH
 About 80% effort on Software Maintenance,(Ponzanelli
et al, ICSE 2013)
 Bug fixation– error and exception handling
 Developers spend about 19% of time in web
search, (Brandt et al, SIGCHI, 2009)
o IDE-Based context-aware web search is the right
choice
6

EXISTING RELATED WORKS
 Rahman et al. (WCRE 2013)
 ERA version of this paper
 Outlines basic idea, limited experiments
 Cordeiro et al. (RSSE 2012)
 Based on StackOverflow data dump
 Subject to the availability of the dump, not easily updatable
 Uses limited context, only stack trace
 Very limited experiments
 Ponzanelli et al. (ICSE 2013)
 Based on StackOverflow data dump
 Uses limited context, only context-code
 Not specialized for exception handling
7

EXISTING RELATED WORKS
 Poshyvanyk et al. (IWICSS 2007)
 Integrates Google Desktop in the IDE
 Not context-aware
 Brandt et al. (SIGCHI 2010)
 Integrates Google web search into IDE
 Not context-aware
 Focused on usability analysis
8

MOTIVATION EXPERIMENTS
Search
Query
Common for
All
Google
Unique
Yahoo
Unique
Bing
Unique
Content Only 32 09 16 18
Content and
Context
47 09 11 10
 75 Exceptions (details later)
 Individual engine can provide solutions for 58
exceptions at most.
 Maximizing total solutions
9

THE KEY IDEA !! META SEARCH ENGINE
10

PROPOSED IDE-BASED META SEARCH MODEL
Start
search
Results
Web page
11

PROPOSED IDE-BASED META SEARCH
MODEL
 Distinguished Features (5)
 IDE-Based solution
 Web search, search result and web browsing all from IDE
 No context-switching needed
 Meta search engine
 Captures data from multiple search engines
 Also applies custom ranking techniques
 Context-Aware search
 Uses stack traces information
 Uses context-code (surroundings of exception locations)
 Software As A Service (SAAS)
 Search is provided as a web service, and can be leveraged by
an IDE. http://srlabg53-2.usask.ca/wssurfclipse/ 12

PROPOSED IDE-BASED META SEARCH
MODEL
 Two Working Modes
 Proactive Mode
 Auto-detects the occurrence of an exception
 Initiates search for exception by client itself
 Aligned with Cordeiro et al. (RSSE’ 2012) & Ponzanelli et al.
(ICSE 2013)
 Interactive Mode
 Developer starts search using context menu
 Also facilitates keyword-based search
 Aligned with traditional web search within the IDE

SEARCH QUERY GENERATION
 Search Query required to collect results from the
Search Engine APIs and to develop the corpus.
 Query generation
 Uses stack trace and Context code
 Collects 5 tokens of top-most degree of interests from
stack trace.
 Collects 5 most frequently invoked methods in the
context-code.
 Combined both token list to form the recommended
keywords for the context.
14

RESULT RANKING ASPECTS (4)
 Content-Relevance
 Considers page title, body content against search query
 Context-Relevance
 Considers stack traces from webpage against target stack trace
 Considers code snippets against context-code extracted from IDE
 Link Popularity
 Considers the Alexa & Compete site rank
 Estimates a normalized score from those ranks
 Search Engine Confidence
 Heuristic measure of confidence for the result
 Considers the frequency of occurrence
 Considers the weight of each search engine
15

PROPOSED METRICS & SCORES
 Content Matching Score (Scms)
 Cosine similarity based measurement
 Stack trace Matching Score (Sstm)
 Structural and lexical similarity measurement of stack
traces
 Code context Matching Score (Sccx)
 Code snippet similarity (code clones)
 StackOverflow Vote Score (Sso)
 Total votes for all posts in the SO result link
16

PROPOSED METRICS & SCORES
 Site Traffic Rank Score (Sstr)-- Alexa and Compete
Rank of each link
 Search Engine weight (Ssew)---Relative reliability
or importance of each search engine. Experiments
with 75 programming queries against the search
engines.
 Heuristic weights of the metrics are determined
through controlled experiments.
17

EXPERIMENT OVERVIEW
 75 Exceptions collected from Eclipse IDE
workspaces of grad-students of SR Lab, U of S,
and different online sources (StackOverflow,
pastebin)
 Related to Eclipse plug-in framework and Java
Application Development
 Solutions chosen from exhaustive web search with
cross validations by peers
 Recommended results manually validated.
 Results compared against existing approaches and
search engines.
18

PERFORMANCE METRICS
 Mean Precision (MP)
 Recall (R)
 Mean First False Positive Position (MFFP)
 Mean Reciprocal Rank (MRR)
19

RESULTS FOR SCORE COMPONENTS
Score Components Metrics Proactive Mode
(Top 30)
Interactive Mode
(Top 30)
Content MP
TEF
R
0.0371
56 (75)
74.66%
0.0481
65 (75)
86.66%
Content +Context MP
TEF
R
0.0376
55 (75)
73.33%
0.0514
66 (75)
88.00%
Content + Context +
Popularity
MP
TEF
R
0.0381
56 (75)
74.66%
0.0519
66 (75)
88.00%
Content +Context +
Popularity
+Confidence
MP
TEF
R
0.0380
56 (75)
74.66%
0.0538
68 (75)
90.66%
[ MP = Mean Precision, R = Recall,
TEF= Total Exceptions Fixed]
20

RESULTS OF EXISTING APPROACHES
Recommender Metrics Top 10 Top 20 Top 30
Cordeiro et al.
(only stack traces)
MP
TEF
R
0.0202
15 (75)
20.00%
0.0128
18 (75)
24.00%
0.0085
18 (75)
24.00%
Proposed Method
(Proactive Mode)
MP
TEF
R
0.0886
51 (75)
68.00%
0.0529
55 (75)
73.33%
0.0380
56 (75)
74.66%
Ponzanelli et al.
(only context-code)
MP
TEF
R
0.0243
7 (37)
18.92%
0.0135
7 (37)
18.92%
0.0099
7 (37)
18.92%
Proposed Method
(Proactive Mode)
MP
TEF
R
0.1000
30 (37)
81.08%
0.0621
32 (37)
86.48%
0.0450
32 (37)
86.48%
[ MP = Mean Precision, R = Recall,
TEF= Total Exceptions Fixed]
21

RESULTS OF SEARCH ENGINES
Search Engine Metrics Top 10 Top 20 Top 30
Google MP
TEF
R
0.1571
57 (75)
76.00%
0.0864
57 (75)
76.00%
0.0580
57 (75)
76.00%
Bing MP
TEF
R
0.1013
55 (75)
73.33%
0.0533
58 (75)
77.33%
0.0364
58 (75)
77.33%
Yahoo MP
TEF
R
0.0986
54 (75)
72.00%
0.0539
57 (75)
76.00%
0.0369
57 (75)
76.00%
StackOverflow
Search
MP
TEF
R
0.0226
14 (75)
18.66%
0.0140
17 (75)
22.66%
0.0097
17 (75)
22.66%
Proposed Method
(Interactive mode)
MP
TEF
R
0.1229
59 (75)
78.66%
0.0736
64 (75)
85.33%
0.0538
68 (75)
90.66%
22

THREATS TO VALIDITY
 Search not real time yet, generally takes about 20-
25 seconds per search. Multithreading used,
extensive parallel processing needed.
 Search engines constantly evolving, same results
may not be produced at later time.
 Experimented with common exceptions, which are
widely discussed and available in the web.
23

LATEST UPDATES
 More extensive experiments with 150 exceptions.
Achieved 92% accuracy.
 Eclipse plugin release
(https://marketplace.eclipse.org/content/surfclipse)
 Context-aware Keyword search with automatic
query completion feature.
 Visual Studio 2012 Plugin under development.
 Extensive User Study ongoing.
24

SURFCLIPSE TOOL DEMONSTRATION
 Tool Demo video:
https://www.youtube.com/watch?v=hGbyF4YveaI
25

THANK YOU !!!
26

REFERENCES
[1] M.M. Rahman, S.Y. Mukta, and C.K. Roy. An IDE-Based Context-Aware Meta
Search Engine. In Proc. WCRE, pages 467–471, 2013.
[2] J. Cordeiro, B. Antunes, and P. Gomes. Context-based Recommendation to
Support Problem Solving in Software Development. In Proc. RSSE, pages 85
–89, June 2012.
[3] L. Ponzanelli, A. Bacchelli, and M. Lanza. Seahawk: StackOverflow in the
IDE. In Proc. ICSE, pages 1295–1298, 2013.
[4] D. Poshyvanyk, M. Petrenko, and A. Marcus. Integrating COTS Search
Engines into Eclipse: Google Desktop Case Study. In Proc. IWICSS, pages
6–, 2007.
[5] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer. Two
Studies of Opportunistic Programming: Interleaving Web Foraging, Learning,
and Writing Code. In Proc. SIGCHI, pages 1589–1598, 2009.
27

SAMPLE STACK TRACE
28
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at test.SockTest.main(SockTest.java:13)

SAMPLE CONTEXT CODE
29
try {
Socket client = new Socket("localhost", 4321);
ObjectOutputStream out = new ObjectOutputStream(
client.getOutputStream());
out.flush();
ObjectInputStream in = new ObjectInputStream(
client.getInputStream());
System.out.println("Buffer size: " + client.getSendBufferSize());
for (int i = 0; i < 10; i++) {
if (i == 3) {
Thread.currentThread().interrupt();
System.out.println("Interrupted.");
}
out.writeObject("From Client: Hellow." + i);
out.flush();
System.out.println(in.readObject());
}
} catch (Exception e) {
e.printStackTrace();
}

SEARCH QUERY FOR CORPUS DEVELOPMENT
java.net.ConnectException Connection refused
connect currentThread
30

ITEMS USED FOR RELEVANCE CHECKING
31
java.net.ConnectException Connection refused
connect currentThread
+
Sample Stack Trace
+
Sample Context Code

SAMPLE STACK TRACE (2)
32
java.lang.ClassNotFoundException: org.sqlite.JDBC
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at core.ANotherTest.main(ANotherTest.java:18)

CONTEXT CODE (2)
33
try
{
//code for making connection with a sqlite database
Class.forName("org.sqlite.JDBC");
Connection connection=null;
connection=DriverManager.getConnection("jdbc:sqlite:"+"/"+"test.db");
Statement statement=connection.createStatement();
String create_query="create table History ( LinkID INTEGER primary
key, Title TEXT not null, LinkURL TEXT not null);";
boolean created=statement.execute(create_query);
System.out.println("Succeeded");
}catch(Exception exc){
exc.printStackTrace();
}

SEARCH QUERY FOR CORPUS DEVELOPMENT
java.lang.ClassNotFoundException
org.sqlite.JDBC db ClassLoader execute
34

ITEMS USED FOR RELEVANCE CHECKING
35
java.lang.ClassNotFoundException org.sqlite.JDBC
db ClassLoader execute
+
Sample Stack Trace
+
Sample Context Code

TOWARDS AN IDE-BASED CONTEXT-AWARE META SEARCH ENGINE FOR EXCEPTION RECOMMENDATION

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (15)

Andere mochten auch

Andere mochten auch (7)

Ähnlich wie TOWARDS AN IDE-BASED CONTEXT-AWARE META SEARCH ENGINE FOR EXCEPTION RECOMMENDATION

Ähnlich wie TOWARDS AN IDE-BASED CONTEXT-AWARE META SEARCH ENGINE FOR EXCEPTION RECOMMENDATION (20)

Mehr von Masud Rahman

Mehr von Masud Rahman (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

TOWARDS AN IDE-BASED CONTEXT-AWARE META SEARCH ENGINE FOR EXCEPTION RECOMMENDATION

Hinweis der Redaktion