The document introduces MobileBench, a benchmarking consortium and suite of tools for mobile devices. It aims to provide more credible, consistent, and user-experience based benchmarking than existing solutions. MobileBench measures overall system and individual component performance, as well as web and UI behaviors. The consortium plans to hold its first general assembly in September 2013 to launch the 2013 version of the MobileBench suite and establish goals for the 2014 version.

1. Copyright© MobileBench Consortium, 2013
MobileBenchMobileBench ConsortiumConsortium
MobileBench
Standard performance metric for Mobile
2013.07

2. Copyright© MobileBench Consortium, 2013
Lack of credible Benchmark tools
• The biggest motivation of developing a new benchmark tool was the lack
of credibility, fairness, and accuracy among conventional benchmark tools
in the market.
 No detailed information for deviceNo detailed information for device
comparisoncomparison
 Small Test items and coverageSmall Test items and coverage
 HardHard to get feedbackto get feedback
 Difficult to interpret the resultsDifficult to interpret the results
 Wide fluctuationWide fluctuation
 User experiences reflected insufficientlyUser experiences reflected insufficiently
1 2 3 4

3. Copyright© MobileBench Consortium, 2013
Goal for MobileBench consortium
 FairnessFairness
Ensure the fairness through opening source-code
 ConsistencyConsistency
Minimize the standard deviation of results
+ User Experience+ User Experience
the performance evaluation based-on user scenario
+ Flexibility+ Flexibility
Support various evaluation functions for Analysis
 DistinctionDistinction
Maximize device performance

4. Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (1/5)
Avg: 0.7
Avg: 4.4
Avg: 12.0
Avg: 0.6
 Consistency
Minimize the standard deviation of results

5. Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (2/5)
 Distinction
Maximize device performance
108%
73%
17%
12%
Spec.

6. Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (3/5)
 Flexibility
Support various evaluation functions for analysis

7. Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (4/5)
MobileBenchMobileBenchTMTM
ResultResult
 User Experience
the performance evaluation based-on user scenario
Other ToolOther Tool
ResultResult
Difference of
storage
performance
MobileBenchMobileBenchTMTM --UXUX
ResultResult

8. Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (5/5)
 System Monitoring
Monitor the behavior of each device in the background
Processor Usage & Freq.
Memory Usage
Storage I/O
MFLOPS/MIPS/Memory/StorageMFLOPS/MIPS/Memory/Storage Java-scriptJava-script Web-loading / HTML5Web-loading / HTML5 UI-stressUI-stress
* Time resolution 1sec* Time resolution 1sec

9. Copyright© MobileBench Consortium, 2013
MobileBench Suite
MobileBench MobileBench-UX
MobileBenchMobileBench SuiteSuite
 Mobile ApplicationMobile Application
 Device Benchmark onDevice Benchmark on PlatformPlatform
 Device Benchmark onDevice Benchmark on KernelKernel
 Synthetic WorkloadSynthetic Workload
 System MonitorSystem Monitor
 Mobile App. +Mobile App. + PC AgentPC Agent
 Mobile Device Control via USBMobile Device Control via USB
 Real Workload/ApplicationReal Workload/Application
 ScenarioScenario--basedbased
 System MonitorSystem Monitor

10. Copyright© MobileBench Consortium, 2013
MobileBench Features
• MobileBench is a total benchmark solution for mobile devices. It can
measure the performance of each H/W component as well as web-
related and UI performance.
1. System Performance:
- CPU/Memory/Storage
2. Web browsing:
- Javascript, Web Loading,
HTML5 compatibility
3. User Experience:
- U/I Stress, Battery Eater
4. Special Feature:
- System Monitor, Email
Reporting
MobileBenchTM Features

11. Copyright© MobileBench Consortium, 2013
MobileBench Features
Features AP Memory Storage Function in Advanced Mode
Processor Core #, Time control
Memory Thread #, Request size
Storage Chunk size, Pre-condition, Native access
Java Script None
Web Loading None
HTML5 Compatible None
Graphic 3D Graphic (TBD)
MobileBenchTM

12. Copyright© MobileBench Consortium, 2013
MobileBench-UX Features
• MobileBench-UX is a PC software to measure UX performance controlling
mobile devices with PC via USB connection. It can simulate all the
different user scenario, changing the combination of scenario items.
Since all the test items are what users actually do every day, this tool can
actually represent and score the real user experience.
One Click!
UX based bench
Multitasking bench
Multimedia bench
Automatic reporting
CameraVideo
Playback
Video
Creating
Application
Installation
Application
Loading
Media
Scanning
I/O
Operation
Boot Test

13. Copyright© MobileBench Consortium, 2013
MobileBench-UX Features
Category Item AP Memory Storage Function
Utility
 Reboot Rebooting time
 Media Scanning Android media scan service time
 External File I/O PC to phone file copy time
 MTP Test Android media transfer protocol
 App. Install Application install time
Multi-
media
 Video Playback-S
Movie frame drop/delay time
- Background storage file copy
 Video Playback-M
Movie frame drop/delay time
- Background memory access
 Video Creating MPEG-1 video encoding time
 Video Trans-coding Movie, MPEG-1 video encoding time
 Camera Taking pictures and saving time
Multi-
tasking
 App. Switching 20-dummy application switching time
 Multi-Popup Dual-movie play
 Web Browsing Web page loading & zoom in-out
MobileBenchTM-UX

14. Copyright© MobileBench Consortium, 2013
MobileBench Consortium
• Organization Structure
Board of Director
(Chair & Directors)
Special Committees Work Committees
CFO
Secretary
President
Tool Maintainer
Board of
Director
Work
Committee
Spec. release
Approval
Spec.
Approval
Specification
& Tools
Developer Yes Yes Yes Yes Yes
Adopter No Yes No Yes Yes
Honorary* No Yes No No Yes
Non-member No No No No
No
(Lite version only)
* Non-Profit organization, educational institutes may be participated as “Honorary” membership with limited right for FREE
• Membership types

15. Copyright© MobileBench Consortium, 2013
Consortium Roadmap
• First General Assembly is set for 26th of September 2013 at Shenzhen, China
• Open for anyone for 1st General Assembly and launching event of
MobileBench suit 2013 for FREE
2013 2014
2Q 3Q 4Q 1Q
Jun.
Consortium
Development
Sep. Mar.
Milestone
 2nd Consortium
Meeting
 Milestone setup for
MobileBench 2014
Sep.Jun. Mar.
MobileBench 2013 rev.1.0
1st MobileBench
General Assembly
Release of
MobileBench suit 2013
1st BoD @
Beijing
Release of beta
MobileBench Suit

16. Copyright© MobileBench Consortium, 2013
MobileBench Consoritium
Register in advance to secure your spot for
1st MobileBench Consoritum General Assembly for FREE
September 26th 2013, Shenzhen, China
(25th for Press & Board of Director meeting)
Hotels and more information will be informed later

17. Copyright© MobileBench Consortium, 2013
Thank youThank you