The document discusses using user sentiment analysis from app reviews and feedback to determine app quality. It describes how Applause analyzes app reviews to classify sentiment, extract key attributes mentioned, identify trends, and cluster similar reviews to provide insights. This helps app developers focus on the right issues to improve their apps based on what users are saying.

1. 1
User Sentiment to Determine App Quality
Heidi Young
VP Engineering, Applause

2. User Sentiment to Determine App Quality
Heidi Young
VP of Engineering
2

3. Getting to Know Each Other
3

4. Who am I?
• Roles/Titles: Engineer, Data Scientist, Software Junkie
• Worked at/for:
– Applause (App Quality)
– Bing (Microsoft, Search)
– Rotella Capital Management (Financial Industry)
– SAIC (Defense Contracting)
– Other smaller startups
• Passionate about:
– Search
– Information Retrieval
– Optimizing and Quantifying Large Data Sets
• Member of the Apigee Technical Advisory Board
4©2015 Apigee. All Rights Reserved.

5. Applause: Who we are and what we do
5©2015 Apigee. All Rights Reserved.

6. Level Setting
6

7. Building an App is Easy
7
Step 1: Write code Step 2: Ship to App
Store
©2015 Apigee. All Rights Reserved.

8. Building a Good App is Hard
• Device Fragmentation
• Multiple App Stores
• Native vs Hybrid vs Web
• Server/Client Tradeoffs
• International, globalization, localization, etc.
• Security
8©2015 Apigee. All Rights Reserved.
? ? ?

9. Building a High Quality App is Harder
• No crashes
• High star ratings
• Lots of downloads
• Happy customers
9©2015 Apigee. All Rights Reserved.

10. Different Needs at Different Times
10©2015 Apigee. All Rights Reserved.
Ideation
Design
Development
TestingBeta
Production
Maintenance

11. Ideation Design Development Testing Beta Production Maintenance
We think about things like this
11©2015 Apigee. All Rights Reserved.
• Use humans to touch your apps in the wild, Who better than an actual user?
• Use software to collect data and measure quality of your your apps, Too much data, too
little time
• Use data to analyze where to focus, Actionable Metrics are the only way to get things
done
Crowd Testers
SD
K
User
Sentiment

12. User Sentiment
12

13. User Reviews and Feedback are Important
1. They are very public
2. They end up making headlines
3. They inform user download decisions
13©2015 Apigee. All Rights Reserved.

14. Store Star Rating vs Review Star Rating
• In a lot of cases, what you can extract from review text is very different than the
store star rating
• Don’t let rating inflation fool you!
14©2015 Apigee. All Rights Reserved.
Store Store Star Rating App Reviews (only) Rating Delta
Google Play (US) 3.97 3.83 -0.14
Apple iTunes 3.76 3.81 +0.05
App Store Star Rating App Reviews (only) Rating Delta
HBO Go (Android) 3.89 2.64 -1.25
Netflix (iOS) 3.43 2.67 -0.76
Bank of America 4.16 2.94 -1.22

15. A Systematic Approach to Measuring App Quality
15©2015 Apigee. All Rights Reserved.

16. Extracting something out of the noise
16©2015 Apigee. All Rights Reserved.
• Step 1: Classify Every Review
• Step 2: Use ratings to assign a score/importance to put a
measurement stake in the ground
Max Score
Mean for Category
Your app score

17. Using Signals to Help with Classification
17©2015 Apigee. All Rights Reserved.

18. Applause Analytics Scoring and Sentiment
Analysis
18©2015 Apigee. All Rights Reserved.
• Histogram of Attribute Scores Across App
Stores
• Mean Attribute Scores Across App Stores

19. Applause Analytics Scoring and Sentiment
Analysis
19©2015 Apigee. All Rights Reserved.
• Histogram of Attribute Scores Across App
Stores
• Mean Attribute Scores Across App Stores
People talk
about these
things a LOT

20. Applause Analytics Scoring and Sentiment
Analysis
20©2015 Apigee. All Rights Reserved.
• Histogram of Attribute Scores Across App
Stores
• Mean Attribute Scores Across App Stores
People talk
about these
things a LOT
People talk more
positively about
these things
People talk more
negatively about
these things

21. Opens Up Competitive Benchmarking
21©2015 Apigee. All Rights Reserved.

22. Applause Analytics Clustering and Ranking
22©2015 Apigee. All Rights Reserved.
The How:
• Modified Agglomerative Clustering Algo (Patent Pending)
• Represent reviews per app as term review vectors and
calculate cosine similarities
• Cluster based on size of review corpus, vocabulary for app, etc.
• Rank based on magnitude, date, avg sentiment, and rarity
The What:
• Grouped, related app reviews
• Most actionable reviews
Example:
Can’t turn off
notifications

23. So… what does this help us do?
23©2015 Apigee. All Rights Reserved.

24. Smarter, More Context to Alert You to the Right Issues
24©2015 Apigee. All Rights Reserved.
Step 1: Use ratings and review count trends
for detection
Step 2: Use clusters for smart
recommendation of focus areas
Security issues with UK accounts
Bidding isn’t sticking
Items are no longer copyable
Search is broken on HTC devices
Auto-detected,
pushed to your
inbox, or app
notification
Trend over N timeunits, then pop alert if >90th%
over N-1

25. Food for thought: Pre- vs Post- Release Issues
25©2015 Apigee. All Rights Reserved.
Top Pre Release Bug Issues
– Creating/Modifying Account
information
– Navigation flow breaks
– Order forms
– Video playback (typically on
mobile due to rotated screens)
– Search experiences
– Broken pictures/photos
Top Post Release User Issues
- Latest update is regression in
functionality
- Bad graphics
- Music or video playback is poor
- Slow to load, freezes on load
- Login broken
- Upsell is annoying

26. Food for thought: Pre- vs Post- Release Issues
26©2015 Apigee. All Rights Reserved.
Top Pre Release Bug Issues
– Creating/Modifying Account
information
– Navigation flow breaks
– Order forms
– Video playback (typically on
mobile due to rotated screens)
– Search experiences
– Broken pictures/photos
Top Post Release User Issues
- Latest update is regression in
functionality
- Bad graphics
- Music or video playback is poor
- Slow to load, freezes on load
- Login broken
- Upsell is annoying
What’s common before and after release?
So that perhaps we can tell you…

27. Credit Where Credit is Due
• Everyone at Applause!
• Especially the people doing all of the actual work behind Applause Analytics:
– Jason Stredwick
– Matthew Johnson
– Scott Ferguson
– Yashas Mavinakere
– Lin Zhong
– Rob Reid
– Josh Wilcox
– Christina Green
– Peter Bell
27©2015 Apigee. All Rights Reserved.