"There’s An App for That" -- but how do we actually develop them? While smartphones and tablets are even getting increasingly more popular and their application scenarios are growing, we still develop them using only a standard integrated development environment. As context-based services and apps do, next to network connectivity, require lots of sensor data, the tools for providing realistic sensor data during development are still immature.
Developing, testing, debugging and evaluating those next-generation context-based apps require sensor data for the mobile device -- acceleration, motion, light, sound, camera and many more sensors are available. Though, the existing development tools do seriously limit application developers by not providing the data at all or only on a very limited scale. Especially for indoor environment with applications such as indoor navigation, seamless interaction between public and private displays and activity recognition and monitoring, realistic sensor data are needed and simulation support during the development phase is essential.
In this paper, we present our work towards a holistic approach for mobile application development in intelligent environments, leveraging the existing development tool chain, facilitating more effective and realistic means for mobile application development at the example of the Android mobile device platform.
Towards a Holistic Approach for Mobile Application Development in Intelligent Environments
1. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Towards a Holistic Approach for
Mobile Application Development
in Intelligent Environments
Stefan Diewald, Luis Roalter, Andreas Möller, and Matthias Kranz
Technische Universität München
Beijing, December 9, 2011
10th International Conference on Mobile and Ubiquitous Multimedia
(MUM 2011)
2. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
General Trends for Mobile Computing
over 512,000 apps
“There’s an approx. 18 billion D/Ls
app for that.”
over 340,000 apps
over 10 billion D/Ls
§ “Always connected, personal and contextual” (finally Ubiquitous Computing!!!)
§ “Rapid development”: more and more apps being developed,
but less and less time for it
§ “Rapid innovation”: apps are getting more and more complex and
having more and more features
à better tools are needed to support better development in less time
ü Source: Appcelerator and IDC: Mobile Developer Report, January 2011 (05.12.2011)
ü Market numbers: http://appshopper.com/ and http://www.appbrain.com/stats/number-of-android-apps (08.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 2
3. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Motivation & Contribution
§ Improvement of the technical development process by extended
debugging capabilities
§ Improvement of the application development process by extended
prototyping capabilities
§ Improvement of the algorithm development process by extended
simulation capabilities
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 3
4. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Outline & Structure
§ Introductory example: “MagicPhone” (Wu et al., 2010) as an example for a
mobile application for indoor interaction in intelligent environments
§ Introduction to ROS (http://www.ros.org - not our contribution, just our tool!)
and its key concepts – a robotics middleware, used also for intelligent
environments (c.f. Roalter et al., 2011)
§ Current state of the art sensor simulation & tools for mobile application
development – focus: sensor-data driven and event-driven applications and
context-aware applications in indoor environments
§ Technical details of our approach
§ Examples for mobile interaction supported by the proposed approach
§ Outlook
ü Jiahui Wu, Gang Pan, Daqing Zhang, Shijian Li, and Zhaohui Wu. MagicPhone: Pointing & Interacting. In Proceedings of the12th
ACM International Conference Adjunct Papers on Ubiquitous Computing (Ubicomp '10). ACM, New York, NY, USA, 451-452.
ü Luis Roalter, Andreas Möller, Stefan Diewald, and Matthias Kranz. Developing Intelligent Environments: A Development Tool
Chain for Creation, Testing and Kranz - MUM of Smart and Intelligent Environments. In Proceedings of the 7th International
09.12.2011 Prof. Dr. Matthias Simulation 2011 4
Conference on Intelligent Environments (IE '11). pp. 214-221, Nottingham, UK, July 2011
5. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Modern Applications need Rich Sensor Data
§ Physical ubiquitous interaction § Indoor localization / navigation
§ Context-based § ...
Esp. indoor apps are heavily dependent on local and environment sensor data.
ü Image Source: MagicPhone: Pointing & Interacting: http://www.youtube.com/watch?v=6aVnSmY81KA (06.12.2011)
ü Jiahui Wu, Gang Pan, Daqing Zhang, Shijian Li, and Zhaohui Wu. MagicPhone: Pointing & Interacting. In Proceedings of the12th
ACM International Conference Adjunct Papers on Ubiquitous Computing (Ubicomp '10). ACM, New York, NY, USA, 451-452.
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 5
6. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Testing & Development of Mobile Applications
§ Context-aware applications depend on
multiple sensors’ data at the same time
(e.g. accelerometer, gyroscope, location, …)
§ Current limitations on multi-sensor simulation
possibilities require that currently:
• (much) testing is carried out on real devices
• the tester has to repeat the physical actions
(incl. moving around, pointing, …) over and
over again in the real world
§ Often functions are tested separately
(e.g. only one sensor input at a time) which does
not allow verification of the complete solution
§ Testing in artificial lab environments can only cover
a very limited set of scenarios
Virtual Prototyping as potential approach for
mobile application development in UbiComp
ü Image Source: MagicPhone: Pointing & Interacting: http://www.youtube.com/watch?v=6aVnSmY81KA (06.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 6
7. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
ROS as Middleware for
Distributed Sensor-Actuator Systems
§ ROS = Robot Operating System, originates from the robotics domain, but it is
also used for intelligent and ambient assisted living environments
(intelligent environment as “ImmoBot”, static distributed sensor-actuator system)
http://www.ros.org/wiki/vmi-ros-pkg ß Our complete stacks avail. online
§ Provides hardware and software abstraction, device drivers, message passing,
visualization and simulation tools (2D and 3D), (…)
Message Passing system (additionally also a service-oriented comm. system)
§ Event-based message generation & handling (somewhat similar to Android!)
§ Basic message types e.g. for strings, integers or floats
§ Special message types e.g. for images, point clouds or object poses
§ Publisher / Subscriber model allows fast interconnection of components
§ All kinds of messages can be recorded and replayed.
Subscriber 1 Publisher Function Subscriber
Publisher msg
09.12.2011 Subscriber n
Prof. Dr. Matthias Kranz - MUM 2011 e.g. add noise, rectify, … 7
8. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Realization of Realistic Simulation with ROS (IE 2011 Paper)
Gazebo 3D Simulator (as part of ROS)
§ Goal: (more) realistic 3D environments
with full texture and lighting support
§ Environments can be created by
using a modified interior design
application (‘SweetGazebo’)
§ Objects can be created with
common 3D modeling tools such as
Blender, Cinema4D or 3ds Max
§ Physics simulation: realistic behavior by defining
mass, inertia and friction values for objects
§ Virtual sensors and actuators can be integrated in ROS via plugins and
attached to any object in the simulation.
§ ROS connection allows reading out and setting objects’ states
ü Luis Roalter, Andreas Möller, Stefan Diewald, and Matthias Kranz. Developing Intelligent Environments: A Development Tool
Chain for Creation, Testing and Kranz - MUM of Smart and Intelligent Environments. In Proceedings of the 7th International
09.12.2011 Prof. Dr. Matthias Simulation 2011 8
Conference on Intelligent Environments (IE '11). pp. 214-221, Nottingham, UK, July 2011
9. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Current Situation w.r.t. the Development Support
Development Support for Sensor Data at the
Example of Android-based Devices (fastest growing OS):
§ Position: input of GPS data via e.g. the NMEA protocol
goal: make it easier to input (esp. for indoor env.)
§ Camera: test image or test video inclusion via a
Client/Server solution – but:
- How long would the user actually point a device in a certain direction?
- How does this additional code influence the app during development, and
when is it removed later? (not transparent to the application)
à goal: make it part of the simulator/emulator!
§ (Acceleration/Pose/…) Sensor data simulation/generation:
artificially generated with an input device with mismatching dimensions
(2D mouse vs. 3D orientation), some support on replay/recording of data sets
goal: make it more realistic
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 9
10. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Current Situation w.r.t. the Development Support
ü Left: Samsung Sensor Simulator, source: http://innovator.samsungmobile.com/image.view.do?serviceId=cmsCnts
&attachmentId=11105 (05.12.2011)
ü Right: Live Camera Previews in Android, source: http://www.tomgibara.com/android/camera-source (05.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 10
11. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Current Situation – OpenIntents Sensor Simulator
ü Image Source: OpenIntents Sensor Simulator: http://code.google.com/p/openintents/wiki/SensorSimulator (05.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 11
12. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Issues and Problems with Current Development Support
§ Physicality of the human as user and of the device in space is often neglected,
e.g. movement in an environment, pointing of a device at an object,
scanning/touching of a QR or NFC marker, co-location of objects, relation of
objects to each other, interrupts during application runtime as standard, not
exception, …
§ Imperfection and variability of the real world are not really taken into account:
à temporal and spatial instability/variability (of sensor data) “ignored”
(“spice” of real world challenges – c.f. impressive UbiChallenge presentations!)
Examples of what is often neglected/not investigated during mobile app development:
• WLAN connection is ‘always bad’ at a given location (that is, the connection
breaks, like in the real world, bursty location updates/jumpy positioning/
suffering localization accuracy indoors; c.f. Kranz et al, 2010),
• places change frequently (like posters at the walls that influence tracking), …
ü M. Kranz, C. people/objectsComparativethe line of sight of a camera, …Localization. 8th IEEE International
• Fisher, A. Schmidt. A walk in Study of DECT and WLAN Signals for Indoor
Conference on Pervasive Computing and Communications (PerCom2010). pp. 235-243, Mannheim, Germany, March 2010
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 12
13. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Room for Improvement in the Development Process
§ Debugging & testing of interactive systems already during development phase
(and not at the end) à not really available as of now (c.f. reviewed simulators)
§ Still no (little) realistic statements can be made on usage/performance of apps in
real(istic) environments as they are (mostly) not part of the development
process!
à not really (quickly) testing possible in multiple environments before
deployment/extensive real world testing
§ Complex interactions such as touching objects or grabbing an image of a
display can currently not directly be investigated in the development tools
à not really available as of now
Challenges:
§ How to provide (better) data (or at least initial data at all)?
§ How to more efficiently develop those complex interactive systems (and not only
mobile phone software)?
§ How to more efficiently prototype multimodal and multimedia systems faster?
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 13
14. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Proposed Idea: Extended IDE and Development Tool Chain
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 14
15. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Example: Virtual Camera and Virtual Display Functionality
§ Linked ROS sensors and Android sensors in the example: camera and
accelerometer.
§ Display: Android output that is pushed back to the simulated 3D environment
all simulation no longer part of the application, but the emulator/simulator
(question: how do we move/steer/control the device in the simulation? à later!)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 15
16. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Supported Sensors and Actuators
Sensors
§ Accelerometer* and Gyroscope
§ Camera* and Depth Image Sensor
§ Light Sensor
§ Proximity Sensor
§ GPS
§ NFC Reader
§ Battery Status A virtual Android device in a
realistic 3d simulation of an office.
§ …
Actuators
§ Light Emitting Diodes and Displays*
§ Projectors
§ Speakers * Currently fully supported by the connected Android emulator,
all others are already supported by ROS as well
§ …
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 16
17. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Example 1: Indoor Localization and Physical Interaction
using NFC tags and QR codes
Real Simulated
§ Connect device, install changed § Install changed version of app
version of app, disconnect device § Perform test: replay recorded data set
§ Perform test: walk several meters in (even faster than real-time), variation
corridor, take picture or touch NFC can be added automatically,
tag e.g. position offset, …
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 17
18. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Ex 2: Indoor Localization/Tracking using Visual Information
Instead of taking various fixed and prerecorded videos, daylight, obstacles, and
other properties are simulated in an extended IDE/tool chain.
ü PTAM (Parallel Tracking and Mapping), source: http://www.robots.ox.ac.uk/~gk/PTAM/ (06.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 18
(markerless, feature-based tracking, no a-priori maps, no inertial measurement unit; test implementation)
19. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Example – Manual Testing of Mobile Applications in the
Virtual Environment
§ 3d input devices like the 3Dconnexion SpaceNavigator
or joysticks can be used for 6 degrees-of-freedom
controlling
§ Mobile device’s accelerometer,
gyroscope and magnetometer for 3D
+
orientation + classic keyboard for 2D
navigation (walking through the virtual
scene by holding a real Android device
for the pose information and steering by
the keyboard for the 2D location info.)
ü Image Sources: top: 3Dconnexion – SpaceNavigator: http://www.3dconnexion.de/products/spacenavigator.html (06.12.2011),
bottom left: http://alwaysinternet.de/wp-content/uploads/2010/12/google-nexus-s-samsung.jpg (06.12.2011),
bottom right: http://www.logitech.com (06.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 19
20. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Example – (human-inspired robotic) Path Planner and
Navigation Stack for Automated (Randomized) Testing
§ ROS offers several path planning and
navigation stacks (spice for getting slightly
different paths on each walk from A to B)
§ Autonomic ROS running robots (like the PR2)
with laser scanner and/or depth cameras can
be used for generating a map
§ Random paths can be created by imposing
different constraints for the route
§ Mobile device could be steered along path
e.g. having a fixed pose similar to holding a
mobile phone in front user
§ Test how applications can adapt to
disturbances, variations, …
ü Image Source: ROS package - base planner: http://www.ros.org/wiki/base_planner_cu (07.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 20
21. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Example – Testing with different Light Conditions
§ Light conditions can be varied in the 3d
simulation.
§ Light direction, color and intensity can be
adjusted.
§ Shadows and reflections are supported.
§ Additional distortion can be applied to the
virtual camera image.
Can be used for testing image/vision-based
applications.
Example: Checking under which conditions or
distances a designed marker is still readable.
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 21
22. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Summary of the Potentials of the Proposed Approach
§ Multi-sensor simulation in a (more) realistic & easy-to-create 3d
environment.
§ Sensor data can either be generated automatically, or by a human moving
the mobile device in the intuitive 3D simulated virtual environment.
§ ‘Randomized’ automatic test cases can make use of available components
(e.g. route planning algorithms) and data sets (biasing of positions and other
sensor values)
§ Recording and replaying of all sensors and states (via ROS messages)
allows detailed debugging.
§ Connection to a hardware emulator allows debugging end-user versions
of mobile applications (no additional source code necessary, that is, no added
code that changes the app’s behavior)
§ Pushing back the mobile’s displays’ content enables realistic usage of
mobile applications in the simulated environment
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 22
23. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Outlook – To Dos for the Future
§ Verification and analysis of the tool chain / approach:
• Survey and test with professional developers is planned
• Compare development time within the scope of our research-oriented
teaching: students will develop various indoor localizations systems with
and without tool chain support
• Analyze and compare the types and amount of errors made with normal
and proposed approach
§ Evaluation of realism of the simulated sensors: metrics need to be defined
§ Exploration of new interaction potentials: fast prototyping of mobile interaction
with e.g. robots and intelligent environments (home automation systems),
possible due to their ROS support
§ Extension for other (physical) interaction approaches, esp. Tangible User
Interfaces
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 23
24. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Outlook – Further possible Application Areas
§ Public and private displays
§ 2d and 3d touch interfaces
§ Tangible User Interfaces (I like cubes)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 24
25. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Outlook – 2D Tabletop Interactions with a Mobile Device
Mixed physical-digital interactions:
§ Displays: either external or mobile device as pocket table
§ Physical Objects (e.g. Tangible User Interfaces)
§ NFC (Near Field Communication) tagged objects
ü Image Source: reacTable: http://www.reactable.com/ (07.12.2011)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 25
26. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Questions?
? ?
? ??
09.12.2011
Thank you- MUM 2011 much for your kind attention!
Prof. Dr. Matthias Kranz
very 26
27. Prof. Dr. Matthias Kranz www.vmi.ei.tum.de
Fachgebiet Verteilte Multimodale Informationsverarbeitung
Technische Universität München
28. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
m.de
i.ei.tu m.de
w ww.vm nz@tu
atthia s.kra
m
09.12.2011
Thank you- MUM 2011 much for your kind attention!
Prof. Dr. Matthias Kranz
very 28
29. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Combination of the Real and Simulated World at the
Example of Indoor Localization Systems
Environment
Real Simulated
Real world test, most
Ground truth, testing of
Real effort and time
controlled effects
consuming
Mobile Device
Real effects, but no Fast evaluation of
Simulated working prototpye algorithms, everything
necessary controllable
ROS can be used to connect real and simulated components.
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 29
30. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Thank you for your attention!
Questions?
?
?
matthias.kranz@tum.de
www.vmi.ei.tum.de/team/matthias-kranz.html
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 30
31. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
Paper Reference
• Please find the associated paper at:
https://vmi.lmt.ei.tum.de/publications/2011/MUM11_preprint_v2.pdf
• Please cite this work as follows:
• Stefan Diewald, Luis Roalter, Andreas Möller, Matthias Kranz. 2011.
Towards a Holistic Approach for Mobile Application Development in Intelligent
Environments.
In: Proceedings of the 10th International Conference on Mobile and
Ubiquitous Multimedia (MUM 2011), Beijing, China, pp. 73-80, December
2011
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 31
32. Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München
If you use BibTex, please use the following entry
to cite this work:
@InProceedings{MobileSimulation11,
author = {Stefan Diewald and Luis Roalter and Andreas M{"o}ller and Matthias Kranz},
title = {{Towards a Holistic Approach for Mobile Application Development in Intelligent Environments}},
booktitle = {Proceedings of the 10th International Conference on Mobile and Ubiquitous Multimedia},
series = {MUM '11},
location = {Beijing, China},
publisher = {ACM},
pages = {73--80},
numpages = {8},
month = dec,
year = {2011},
address = {New York, NY, USA},
ISBN = {978-1-4503-1096-3},
url = {http://doi.acm.org/10.1145/2107596.2107604},
doi = {10.1145/2107596.2107604},
acmid = {2107604},
keywords = {Android, ROS, intelligent environment, middleware, mobile devices,
simulation, smart phones, virtual environment},
}
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 32