Information Needs for Software Development Analytics

Ray Buse

Tom Zimmermann

Ray Buse • Information Needs for Software Development Analytics
6/20/2012 22

6/20/2012 33

Analytics

Measurements
Measurements
Metrics What?

How much?
Exploratory Analysis

Quantitative Analysis
Analytics Why?
Qualitative Analysis

Experiments
What if?

Insights Goal
Insights

6/20/2012 55

Where is Software Eng. Today?

Few Insights Many Insights

Ray Buse • Information Needs for Software Development Analytics 6

Dependencies Code Clones Telemetry
Velocity

Change Type

Readability
Architecture

Expertise

Data Rich
Ownership
Churn
Failure Documentation
Models

Test
Defect
Complexity Coverage
Density

Data is plentiful

Researcher Project Manager

Acting on data is hard
Readability
is 4.8 So what?
Researcher
Complexity
is 9.6

…


Acting on data is hard
Has to learn about
Knows how to analysis to understand
analyze data and act on results
Researcher Makes
assumptions Knows project
about project very well


Analytics Tools
Draws on
Meets specific
powerful analyses
information needs

Researcher Product Manager

Existing Tools

Analytics: Focus on data collection and developer
awareness rather than information needs


But what information is needed?

6/20/2012 1313

But what information is needed?
… and how should it be presented?

6/20/2012 1414

This Talk
What is Software Analytics? Guidelines

Survey at Microsoft


This Talk
Guidelines

Survey at Microsoft

6/20/2012 1616

Survey of Microsoft Engineers
110 responses. 53 developers and 57 managers.
30

25

20

15
Developers
10 Managers

5

0
Entertainment Microsoft Online Server & Tools Windows and
& Devices Business Services Division Windows Live
Division Division Division Division

6/20/2012 1717

Survey of Microsoft Engineers
110 responses. 53 developers and 57 managers.
30

25

20

15
Developers
10 Managers

5

0
Entertainment Microsoft Online Server & Tools Windows and
& Devices Business Services Division Windows Live
Division Division Division Division

6/20/2012 1818

What factors most influence your
decisions?

6/20/2012 1919

Important Decision Factors

6/20/2012 2020

Important Decision Factors
Managers rely
more on data

Developers rely
on experience

6/20/2012 2121

What questions are important or
difficult to answer?

6/20/2012 2222

6/20/2012 2323

6/20/2012 2424

6/20/2012 2525

Which artifacts are
important to measure?

6/20/2012 2626

Important Artifacts to Report

6/20/2012 2727

Important Artifacts to Report
Important to tie data
to deliverable artifacts

Developers more concerned
with code artifacts

6/20/2012 2828

What indicators do you currently use?
What would you like to use?

6/20/2012 2929

Fraction of Developers Fraction of Developers
that currently use this that WOULD use this

Indicator

Fraction of Managers Fraction of Managers
that WOULD use this
that currently use this

6/20/2012 3030

6/20/2012 3131

Desired AND Highly Available

Desired AND
Highly Available

6/20/2012 3232

Desired but NOT
Highly Available

6/20/2012 3333

All useful to at
least half of
respondents

6/20/2012 3434

What decisions could
analytics help with?

6/20/2012 3535

Targeting Testing

“Targeting testing of a product needs information
on the code that changed from build to build
and [the] result of bug fixes so we could more
easily map out what features and what other
code requires re-examination.”

6/20/2012 3636

Targeting Refactoring

“The number of bug reports for a certain feature
area helps us decide whether that feature area is
mature for a refactoring.”

“Telemetry allows us to prioritize investment for
code cleanup and bug fixing in a way that has
substantial customer impact.”

6/20/2012 3737

Understanding Customers

“Analytics helps us understand how a user is
using our product. Are they performing tasks we
expect? Performing tasks we didn’t anticipate?
We can determine effectiveness of features, as
well.”

6/20/2012 3838

Targeting Inspection
“If I had decent metrics for software readability,
dependency changes, component level changes,
etc., it could help drive decisions on what levels
of code review we would force. I’d also like to
make comparisons between the results of the
analytics and the actual number of issues found
during code review (in order to tune the
analytics).”

6/20/2012 3939

Analytics Tools

6/20/2012 4040

Recommendations
• Be easy to use.
• Be fast and produce concise or
summary output.


Recommendations
• Measure many artifacts using
many indicators.
• Map indicators to features and
dates to milestones.


Recommendations
• Be current and interactive.
• Focus on characterizing the past
and present over predicting the
future.
• Focus on information relevant to
the target audience.


Tool Spectrum
Past Present Future

Exploration Trends Alerts Forecasting

Analysis Summarization Overlays Goals

Experimentation Modeling Benchmarking Simulation

6/20/2012 4444

Tool Spectrum
Past Present Future




6/20/2012 4545

Tool Spectrum
Past Present Future




6/20/2012 4646

Tool Spectrum
Past Present Future




6/20/2012 4747

Make Data Actionable and Accessible

Data collection Data quality

User needs Education

Privacy User experience

Thank you!

Ray Buse
http://arrestedcomputing.com

Tom Zimmermann
http://thomas-zimmermann.com

6/20/2012 5656

Information Needs for Software Development Analytics

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (11)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Information Needs for Software Development Analytics

Ähnlich wie Information Needs for Software Development Analytics (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Information Needs for Software Development Analytics