Augmented reality (AR) is a promising technology for building applications in an Internet of Things (IoT) environment, utilized for visualizing information provided by IoT devices. In this paper, we enable Web-based mobile AR applications with mobile agents in a resource-oriented IoT system architecture. We present an adaptable mobile agent composition that contains the data representation logic and mappings between AR applications and system resources. Thus, mobile agents and AR application-specific data structures are exposed as global system resources. System resource linkages are considered between real-world objects and their virtual representations for mobile agent-based AR applications. The agent composition also complies with the REST principles for resource access and control system-wide. This allows dynamic runtime adaptation and addressing the device and resource heterogeneity, thus eliminating the need for application-specific communication protocols. Moreover, we utilize a Web-based mobile AR application framework, running completely in a Web browser, which facilitates straightforward AR application development. Lastly, a proof of concept mobile AR application is implemented, where a coffee maker with a visual tag is connected to a low-power resource-constrained wireless sensor network node as an IoT device. A mobile agent is injected into the IoT environment to expose the state changes of the coffee maker. Through the visual tag, AR applications are able to visualize the state changes of the coffee maker in their user interface.

1. Augmented Reality Web
Applications with Mobile Agents
in the Internet of Things
Leppänen, T., Heikkinen, A., Karhu, A., Harjula, E., Riekki, J. and Koskela, T. “Augmented Reality Web
Applications with Mobile Agents in the Internet of Things,” In: 8th International Conference on Next
Generation Mobile Applications, Services, and Technologies (NGMAST2014), pp. 54-59, September
10-12, Oxford, UK, 2014. http://dx.doi.org/10.1109/NGMAST.2014.24
Teemu Leppänen1, Arto Heikkinen2, Antti Karhu2, Erkki Harjula2,
Jukka Riekki1 and Timo Koskela2
1) Department of Computer Science and Engineering, University of Oulu, Finland
2) Center for Internet Excellence, University of Oulu, Finland

2. Why mobile agents in IoT for AR?
Augmented reality technologies provide an enhanced user interface to
monitor and control smart environments
Mobile agents enable injecting application-specific computations into the
system
– Create agent-based services
– Re-usable in different applications and services
– Platform and programming language independent
– Adapts to fluctuating IoT system environment
Transfer code between AR application and system devices
– Application-specific data processing and UI code for different platforms
Mobile agents live in IoT system without explicit coupling to AR applications
– Facilitates loose coupling between AR application and system resources
– AR tag mapped to a POI, which is in turn mapped to a system resource, i.e. agent-
based service or computation

3. Integration to the Web
Resource-oriented architecture
– Resouce is the main abstraction, exposed with unique URI and state
– Enables various representations of the resource state for human-machine
interactions
Resources include:
– System devices, their physical components (sensors and actuators) and data
– Mobile agents as representation of the application-specific data processing
– In AR: POI entities
Unified and standardized communication for resource access and control
– Promotes simplicity and re-usability in the application design
– HTTP methods: GET, POST, DELETE
• Create new resources and functionality, retrieve application-specific
representations and control the resources
Facilitates seamless information retrieval and control for both physical and
virtual objects with unified interface for AR applications

4. Web-based mobile AR Framework
Enables implementing mobile AR (MAR)
applications running plugin-free in a Web
browser
Relies on common Web technologies
– HTML5, WebRTC and XML3D
– Modular design
Compatible with all mobile platforms and
operating systems
Components
– Sensor-based registration and tracking component detects the device’s position
and orientation
– Vision-based registration and tracking component detects AR tags from camera
video stream
– 3D scene manipulation component converts above data for XML3D
– XML3D provides representations of the content for HTML
– Web service-interface provides communication

5. IoT system architecture
Based on IETF CoRE WG’s framework
Mobile device hosts the AR application
in browser
WSN based on 8-bit MCU nodes
System components
– Resource directory stores the system resource descriptions
• Name, URI, current address, content-type, semantic description, …
– Proxy component translates protocols over disparate networks
• HTTP (Wi-Fi atop 2.4Ghz) <-> CoAP (6LoWPAN atop 868Mhz)
– Repositories
• POI Database hosts POI entity descriptions with corresponding AR tags and linked
resources
• Other repositories host more system resources: e.g. mobile agent code

6. Proof-of-concept application (1/2)
MAR application running in a
Web browser in a tablet device
Shows the ”freshness” of coffee
in a coffee maker
1. Detects the AR tag from video
feed
2. Retrieves the POI description
3. Retrieves the state of the
corresponding linked resource,
i.e. mobile agent
4. Displays the state representation
in MAR application in the Web
browser

7. Proof-of-concept application (2/2)
Coffeemaker
– Connected to a WSN node, which monitors
its power consumption
– Attached AR tag, with POI description linked
to a mobile agent
Mobile agent injected to the system to
calculate time since last state change of the
coffeemaker
– Thresholding the power consumption for two
states: ON and OFF
Mobile agent composition
POI data structure

8. Conclusions
Augmented reality is a natural way of visualizing information provided by
different IoT devices
Web browser provides a uniform application development platform for AR
applications
IoT systems and applications become Web-enabled with the resource-
oriented architectural approach
Mobile agents enable injecting dynamic application-specific data
processing and information visualization into the system
The presented prototype application was straightforward to implement
with the presented IoT system architecture

9. Thank you!
Questions?
For further information, please contact:
teemu.leppanen@ee.oulu.fi
arto.heikkinen@cie.fi