This document describes experiences from implementing a cross-domain use case that combines a "Music follows user" use case with a "Read aloud message" use case. It does this by combining services operating on a Service Oriented Architecture (SOA) with agents executing on a smart space architecture. Lessons learned suggest defining simple ontological concepts and associated behaviors to implement similar use cases in a scalable, future-proof manner.

1. Experiences in Implementing a Cross-domain Use Case by Combining Semantic
and Service Level Platforms
Vesa Luukkala, Dirk-Jan Binnema, Mihaly Börzsei Alessio Corongiu
Nokia Research Center Centro Ricerche Fiat
Finland Orbassano, Italy
{vesa.luukkala,dirk- alessio.corongiu@crf.it
jan.binnema,mihaly.borzsei}@nokia.com
Pasi Hyttinen
Valtion Teknillinen Tutkimuskeskus
Kuopio, Finland
pasi.hyttinen@vtt.fi
Abstract— We describe experiences of implementing a mash- mechanisms which allow handling and representing semi-
up of two independent use cases, one operating in the structured information. At the lowest level, the Resource
embedded domain and the other in the internet messaging Description Framework [4] (RDF) is used to present the
domain. We combine these service implementations on an
existing Service Oriented Architecture (SOA) system with
information as a set of triples. The structures which are built
agents executing on a novel smart space architecture. Based on on top of RDF can be specified by knowledge representation
the lessons learned from this work, we sketch an approach for languages such as RDF-Schema [5] (RDFS) or a web
implementing similar use cases in a scalable and future proof ontology language [6] (OWL). These languages are used to
manner by defining simple ontological concepts and associated define ontologies which describe the shared vocabulary for
behavior. modeling a particular domain. Exhaustively defining and
Keywords: semantic interoperability, semantic web, smart agreeing ontologies for all or most domains and participants
spaces, RDF, ontologies, embedded systems is a similar effort to standardization, but the Semantic Web
approach gives the possibility of leaving information only
I. INTRODUCTION partially defined. Furthermore, the information described by
semantic web technologies is to a degree self-describing,
A smart space can be defined as an ecosystem of interacting
allowing participating entities to interpret the information at
computational objects. The physical manifestation of a
runtime. To minimize the standardization problem, we
smart space can consist of sensors, devices, and appliances
expect to canonically define the smallest building blocks
which communicate with each other by means provided by
(RDF, RDFS, OWL) as well as selected higher level
the smart space. To guarantee interoperability between the
concepts as core ontologies. Use of ontologies and even
participants, a common ground needs to be defined and this
core ontologies is not enforced; the participating nodes
is typically done by means of standardization. However, the
define their own, local interpretation of the used ontologies.
promise of smart space environments is that all kinds of
Eventually this may lead to emergence of de-facto standard
information can be shared and accessed via the smart space
ontologies.
and thus the number of participating devices, vendors,
The Artemis SOFIA project [7] aims to provide a solution
product domains and produced information is large. The
to the interoperability problem by creating an open
gamut runs from sensors measuring a physical property like
innovation platform which is a framework of middleware,
temperature, to mobile devices providing a list of favorite
methods and tools to create ecosystems of interacting
restaurants and their addresses. The downside of this variety
objects. The presented work is part of the SOFIA project
is that creating a common standard is hard, as is changing an
effort. We have used the Smart-M3 [8] middleware
existing standard [1]. The existing SOA-based [2]
produced by SOFIA to implement use cases in the SOFIA
approaches offer tools for defining service interfaces, but
“Personal Spaces” work package. “Personal Spaces” aims
they also suffer from the same problem of standardization.
to create a smart space around a moving person in order to
This slows down innovation which would otherwise benefit
enable this person to access and organize relevant services
from the available multitude of information. Semantic Web
and data around him. Here we use the term “service level”
[3] approaches offer one solution for the above problem by

2. to mean implementations based on SOA [2] principles (e.g. cases and we have selected two from these to be
Web-services, UPnP, NoTA), which typically consist of an implemented and then combined, even though they compete
interface description and possibly a way of advertising the for same resources.
services at a registry. By “semantic level” we mean the The driving force in our particular environment is that
information sharing occurring in a blackboard-like Smart- existing and future devices and embedded systems
M3 system by means of the above Semantic Web participating in the personal space have very different life
technologies. cycles. A car model has a life cycle of eight to twelve years,
We have selected two use cases: “Music follows user” whereas a consumer electronic device has a life cycle of
and “Read aloud message”, which we are using as trialing only a few years. Smart space ties these different
platforms for SOFIA results as well as parts of the “Personal interoperating systems and devices closer together. Smart
spaces” offerings for the user. Our specific target is to space interoperation has a net effect that products now
combine or mash-up the two use cases and be able to repeat having short life cycles can be longer – new features appear
the same process for future use cases. A more generic goal from smart space. Also product having now long life cycles
is to be able to execute the same use case in many can be shorter via smart space application. We expect that
environments using the best possible resources. For example similar situations arise elsewhere as well. The combination
in our case we want to reify the abstract notion of “music of on-board facilities and nomadic devices as well as
listening” to a semantic level service, which may be services here presented will be offered in the near future
instantiated to seamlessly use various available resources. will narrow this gap, allowing for a fast market growth for
We present a naive solution and, based on our these applications. The possibility of bidirectional
experiences from that work, we propose to introduce and communication among the vehicle and the personal devices
define ontologically a semantic level entity corresponding to enables the use of vehicle system and nomadic device
the semantic service and the resources it uses. Any service applications and controls in coordination for example,
implementations or agents which contribute to this semantic sharing displays and controls in both systems. Furthermore
service must adhere to a small set of commands given by the it enables the access to vehicle data via mobile device in a
semantic level. The managing of these semantic service way fully transparent to the user. The service and
instances must be implemented separately. application will not be coupled strictly to one device but
This approach resembles service composition or rather to the environment.
orchestration [9] and it can also be viewed as a simple and The mash-up of the two selected use cases aims to show
static implementation layer for a multi-stakeholder how a user can enjoy music from different devices while
distributed system [10]. moving around between different locations and given the
Section 2 describes the environment we are operating in, opportunity, use resources from the personal environment to
section 3 describes the architecture and the implementation, enhance the experience of playing music. User can start
Section 4 describes our findings, and solutions to solve the listening to music on the personal mobile device (MD),
observed problems and section 5 offers conclusions and our picking a song. User can control the playing from the
future plans. mobile device user interface and the music plays from its
loudspeakers. When entering a car with steering wheel
II. BACKGROUND controls and a rear seat entertainment system (RSE), the
For our “personal space” we expect to have the following user can switch the rendering between the mobile device
resources available: a Mobile Device (MD) with multiple and the Rear Seat Entertainment system at will, with a
connectivity options, an embedded car dashboard unit with dedicated user interface element. The user can also control
Bluetooth connectivity and a Rear Seat Entertainment the rendering both from the steering wheel as well from the
System (RSE) with WLAN connectivity. These devices mobile device user interface.
implement services on an embedded SOA system [11] and Note that the mobile device and the on-board telematic
use a common logical blackboard for sharing of semantic unit share the control of the playback by means of the
level information (the Smart-M3). Smart-M3 is a platform Steering Wheel Buttons, while audio handover happens
inspired by the tuple space paradigm [12]. It provides a between speakers of the mobile and the rear seat
single logical information store by which multiple entertainment system.
independent nodes interact with each other. Store and the The user is also subscribed to a particular twitter feed and
associated nodes can be thought to implement a “smart when a new message is received during music playback, the
space” [13]. We expect that later on new services and music is paused, the message is read out aloud and then the
information will be available for the “personal space”, for music rendering resumes. This is useful when the messages
example via the other SOFIA work packages. In order to describe for instance traffic conditions. While the use cases
show the value for the user we have defined several use are as such independent from each other, they both require

3. exclusive use of audio resources. We expect similar kind of smart applications to program using ontology concepts
of conflicts over same resources to arise more generally in relevant to the application area. This Ontology API may
other use cases. either be provided by a separate ontology library generated
Also, the components we are using as building blocks of at design-time, or by the Sofia Application Development Kit
the use cases are a suitably diverse set of service (ADK).
implementations, applications using the services and The service architecture is based on the NoTA service
semantic level agents. These give a good overview of based concept. In NoTA-based systems, the service
typical challenges in integrating semantic-level entities to implementations are called Service Nodes (SNs) and users
service-level and other functionality. of those services are called Application Nodes (ANs). A
At the time of this implementation, no core ontologies NoTA service implementation is typically accompanied by a
were available, so we used our own simple domain service description file, which enumerates the service
ontologies and tools as a starting point. messages that are used to communicate with that service.
The service description file is the canonical definition of
III. LOGICAL ARCHITECTURE AND IMPLEMENTATION the service interface, but not the service behavior. These
files can also be used to generate stub code to access the
A. Architecture
service. To enable run-time discovery, NoTA services can
The functional architecture of the use cases is illustrated either register themselves with a Resource Manager (RM)
in Figure 1. The two main elements at the semantic level are or, alternatively, count on pre-existing knowledge regarding
the Semantic Information Broker (SIB), an entity the service ID (SID). The resource manager can be then
performing triple governance in possible co-operation with queried by clients for service descriptions and connected to
other SIBs for one Smart Space and the Knowledge the service based on the results.
Processor (KP), an entity contributing to insert/remove)
and/or consuming (query/subscribe) content according to B. Service Level Implementations
ontology relevant to its defined functionality. Figure 2 depicts the service level architecture
implemented for the “Music Follows User” use case. The
remote music rendering use case is implemented directly
over the NoTA service-level and requires the NoTA
infrastructure to operate. The audio rendering service
implementations rely on the OpenMAX-IL-IL
standardOpenMAX-IL. The Steering Wheel service
implementation is basically a keypad service
implementation.
Figure 2 - The "Music follows user" service architecture
1) The player
The mediaplayer acts as the control point for the use case
“Music follows user”. The user can start playing an audio
file and control the playing with the usual player controls on
Figure 1 - Service level and semantic level architecture the mobile device. The player acts as a client to existing
The KPs can connect to the SIB with any suitable services (keypad, OpenMAX-ILRemote Audio Renderer,
transport using the Smart Space Access Protocol (SSAP). SIB as described below). The player itself is not a service in
The components handling different transports may either be the SOA sense as it does not offer a service interface. It still
included in the SIB or KPs at compile-time or be dynamic, implements the functionality for music rendering and is in
plugin-like components. Both SIB and KPs may have charge of the interoperability between the Remote Audio
multiple transports. Renderer OpenMAX-ILand Keypad services. Note that we
The architecture also contains the notion of an Ontology could (re-)build services using the same services as the
API, which is an ontology-specific API allowing developers

4. player uses, but this would mean unnecessary duplication of C. Semantic level implementation
work and in case of handover, fairly involved engineering After this discussion of the service level implementations,
work. The semantic part of the mediaplayer is that it we continue with the semantic level – that is, the knowledge
connects to the SIB service and then places a subscription processors (KPs). We have three KPs: the “tweet-
on the arrival of a message, which then causes pausing and notification”, “media-control” and “speech synthesizer”.
resuming. Firstly, the “tweet-notification” refers to the detection of
2) Steering Wheel new Twitter messages (“tweets”). It is backed by a specially
The Steering Wheel service implementation makes the crafted version of the Mauku Twitter-client, and the
button presses on the wheel available for remote clients. The “MessagingClient” NoTA service, which along with
steering wheel buttons are mapped to the usual player “speech synthesizer” allows for retrieving the arrived
controls. The transmitted key symbols are compatible with “tweets” and performing text-to-speech of their contents.
the X Window system keycodes. Whenever “tweet-notification” detects that a new tweet
3) ILProxy has been added to the Twitter feed, it then creates an
OpenMAX™ is a royalty-free, cross-platform API that instance of a Message (as defined in domain specific
provides comprehensive streaming media codec and messaging ontology) to the Smart Space. The message class
application portability. The OpenMAX IL (Integration definition is as follows:
Layer) API defines a standardized media component wp1:msg a owl:class;
interface [14]. ILProxy is a drop-in replacement for a local wp1:senderId xsd:string;
OpenMAX-IL library, which makes a remote OpenMAX-IL wp1:msgId xsd_string;
wp1:arrivalTime xsd:string;
library implementation available locally. In this case, the wp1:content xsd:string.
“service description” is the C header file for the library. The “tweet-notification” fills in all the properties, except the
Upon request, the client streams this music to the remote content property.
media playback device. In the demonstration setup, we use Secondly, the “speech synthesizer” has subscribed itself to
an NXP media rendering device for that, but the same role instances of type msg and for new messages it first obtains
could be played by any computer with the necessary
the message contents corresponding to the msgId from the
software. The client creates a distributed rendering pipeline
NoTA messaging service. Then it assumes that the playing
starting at the mobile device and ending in the remote
is stopped and initiates the reading of the message content.
playback service. The client feeds the music files into this
After the message has been read, it adds the
pipeline, and it allows for switching between local and
msg:hasBeenRead property with value “yes” to the
remote playback.
particular message instance. This is used to keep track of
4) NoTA Messaging service
messages which have been already processed.
The messaging service makes the content of received
Thirdly, the “media-control” has subscribed itself to
Twitter messages on a certain Twitter feed available.
However, the full service counts on the semantic level being instances of type msg and will be notified of appearance of
available. In this implementation, the notification of the message triggering pausing of playback and another
received message along with an ID is published via the SIB subscription described soon below. Messages that have the
service and the content is obtainable only through the NoTA msg:hasBeenRead with value “yes” property are
service level interface. The mapping of the ID to the ignored. Here we have in effect defined an extension (the
message content is internal data of this service. hasBeenRead property) to the messaging domain
5) SIB Service ontology, which is shared by the three entities. Any other
The SIB service implementation is part of the M3 entity which only knows about the message definition can
baseline for SOFIA IOP. The semantic activities of this ignore the extension and operate without knowledge of that.
demo are all handled through the SIB by means of nodes Also, using this representation we maintain part of the state
reading and writing information to SIB service. There is a of the whole use case (the messages which have been
single SIB implementation which exposes both a NoTA processed) on the semantic side.
service interface as well as an XML based interface over
IV. ANOTHER APPROACH
TCP/IP. The SIB registers itself to Resource Manager like
any other service. For interoperability between entities, we need to share
Any semantic level node needs to connect to this service well-defined information and agree on operational
implementation to be able to share semantic content. From semantics according to which the information is operated on
the SSAP protocol point of view the messages are lists of [15]. The implementation described in previous section is
RDF-triples. straightforward from the ontological perspective, but it has
some flaws:
1. the relation between new instance of a Message

5. appearing and the player pausing, is not obvious Activity. We have two devices, a mobile device and a
2. if we want to add another type of use case which dashboard device, both of which have the same capabilities
needs to pause the player, then we need another for audio output and keyboard input. Even though they are
ontology that must be understood by the player or contained in devices, in this example they have been
even worse, we must abuse the Message mechanism. published to the SIB as individual capabilities. An
This is detrimental for scalability. unspecified actor has inserted an Activity, which uses
In short, the operational semantics have been some of the capabilities published on the SIB and it has
purposefully left informally defined. To solve the problems published that Activity.
in a more general way requires more overhead: a description
of the behavior in addition to a description of the
information representation. A general solution is sketched in
[10], which suggests that each participating agent publishes
a description of its behavior. Other participants can interpret
this information and then deduce based on that how to hook
into the existing system. Alternatively, we could take the
other extreme, which is the standardization approach by
defining operational semantics for all information. This
approach is not realistic due to the open ended nature of the Figure 3: Example of an Activity
information and the amount of work required.
We propose an in-between solution which is more The activity instances are managed by arbiters, which are
limited, yet simpler mechanism. We define an ontology, dedicated to a single instance of an activity. An arbiter is a
which includes an Activity class that indicates the program which monitors instances of activities. The main
resources (which may include service implementations) purpose of an arbiter is to schedule different activities by
which are to be used by some functionality: setting the wp1:active property to True or False. Note that
wp1:Activity a rdfs:Class; even though an arbiter most likely resides in the device
wp1:requires rdfs:Class; which originates the Activity, it is a logical entity and
wp1:uses dcs:Capability;
wp1:active xsd:string; thus can reside in any node. Arbiters are able to detect on
wp1:conflicts owl:Thing; their own whether different activities are conflicting by
wp1:importance xsd:string; using the same resources or same types of resources. In this
wp1:additional owl:Thing.
case the arbiters can try to resolve the conflict by comparing
The activity may be active or not active, it may have a
preference information which has been related to the
free form importance tag added to it and it may also link to
activity instance by the additional property. At simplest
additional information. In addition to the indication of used
resource instances, it is also possible to require a class of conflict can be resolved by comparing the importance
resources. In our use cases both the “media-control” and property along with a preference list of values. On the other
“speech-synthesizer” KPs create activity instances which hand if an activity instance has not defined resources to be
require a resource of type AudioOutput and also point to used the arbiter may assign existing free resources for it.
known resources. This mechanism relies on the resource For the example in Figure 3, suppose that another actor
descriptions being available and described on the semantic would insert new Activity, which would use at least one
side. Any entity which publishes an Activity instance is of the same capabilities as the existing Activity. It would
expected to commit to some functionality and rules: then be up to the arbiter to detect that a conflict exists and
1. it observes changes to the Activity instance, secondly to resolve it based on the available information on
especially the active property the SIB. For decisions which cannot be resolved by just
2. it honors the changes to the active property, when looking at the Activity instances there may be a more
active has value “no”, it pauses its use of general arbiter, which is able to schedule the conflicting
resources so that they can be used by other entities, entities.
when active has value “yes”, it can use the resources As mentioned before this relies on the fact that the
3. it does not change the Activity instance, expect for resources which can be used are also maintained on the
semantic side faithfully, so that their existence implies
the uses properties
readiness to participate or be directed by semantic level
4. it is able to access the indicated resource instances
activities. This requires integration with the service level
by information in the its descriptor
resource manager. Like for the activities the resources may
Here we are separating and making explicit the
have their own arbiters which independently decide whether
coordination of the activity from the domain specific
they comply with requests from the activities or not. We
information. Figure 3 shows and example of use of an

6. expect that resource specific arbiters with an independent V. CONCLUSION AND FUTURE WORK
arbiter for multiple activities can make essentially local Based on our experiences on implementing a mash-up of
decisions without ending up in a heavy multiparty conflict two use cases, we have designed an ontology-based
resolution process. mechanism for abstracting away access to shared resources.
The activity and the behavior experienced by the user We expect to re-implement existing use cases using this
should reflect the desires and wishes of the user. In a mechanism to gain scalability and future-proofness of these
simplified setting these can be expressed as an ordered list use cases. We have trialed the mechanism on semantic-level
of preferences. Preference can be defined as a set of simulator. The activity abstraction itself is fairly simple, but
activities that does not produce resource conflicts. If the it can be parameterized by producing different kinds of
activities do not use the same resources they can be easily schedulers, which can take into account the preferences
put into the same preference set since there is never conflict. associated with activities and resources. We expect that this
If activities use the same resource then we need to notify the mechanism will be included in SOFIA core ontologies.
nature of the usage and define rules for using it. Exclusive
usage of the resource can be handled by using user defined ACKNOWLEDGMENT
priorities. Potentially conflicting activities in a preference This work was supported by SOFIA (Smart Objects For
are ordered. Activities may use several resources that Intelligent Applications) project and funded by Tekes (the
potentially are always conflicting. In this case the user Finnish Funding Agency for Technology and Innovation)
accepts only one activity. and the European Commission.
From the user point of view the preferred smart space
REFERENCES
behavior and therefore activity is also context and situation
[1] J. Farrell and G. Saloner, "Standardization, Compatibility, and
dependent. User will have different audio preferences for Innovation", The RAND Journal of Economics, Vol. 16, No. 1
day and night situations, for example. Full context aware (Spring, 1985), pp. 70-83
and situation conscious behavior is the ultimate target but [2] T. Erl, Service-Oriented Architecture, Prentice-Hall, Englewood
very difficult to achieve. Therefore we simplify again and Cliffs, 2005
define context as all information in the smart space and [3] Tim Berners-Lee, James Hendler, and Ora Lassila. The semantic web.
Scientific American, 2001.
reduce the situation to set of current activities. Smart space
[4] Resource description framework. http://www.w3.org/RDF/.
behavior is series of transitions where predefined user Referenced March 4th 2010
preference is changed to next preference. This is controlled [5] RDF vocabulary description language, http://www.w3.org/TR/rdf-
by preference scheduler that using the resource availability schema
information switches between preferences. [6] Web ontology language. http://www.w3.org/2004/OWL/. Referenced
March 4th 2010
We note that the simplest approach for combining the
[7] SOFIA project. http://www.sofia-project.eu/. Referenced March 4th
service and semantic level is to have the service 2010
implementations publishing information on the SIB where [8] OpenM3 demonstration release. http://sourceforge.net/projects/smart-
the information can then be freely operated on in any m3/files/OpenM3.tar.gz/download. Referenced March 4th 2010.
manner by other actors in the system. This write-only [9] D. Chakraborty and A. Joshi, "Dynamic Service Composition: State-
approach enables easy mash-ups as any entity operating on of-the-Art and Research Directions,"University Of Maryland,
Baltimore, December 19 2001.
the information will not affect the producers at all. In this
[10] R. J. Hall, "Open Modeling in Multi-stakeholder Distributed
way the interoperability can be achieved without definition Systems" presented at Proc. First Workshop on the State of the Art in
of operational semantics. However, it also sidesteps the Automated Software Engineering, 2003.
problem we discussed in this paper, and we think that the [11] Suoranta, R.: Modular service-oriented platform architecture - a key
enabler to soc design quality. In: 7th International Symposium on
dynamic environment where state needs to be updated on Quality of Electronic Design (ISQED 2006) (March 27-29, 2006),
the SIB requires a mechanism which is similar to ours. San Jose, CA, USA, pp. 11–13. IEEE Computer Society, Los
We also hope that the relative simplicity of the Activity Alamitos (2006)
mechanism would make it easy to modify existing service- [12] Gelernter, David. "Generative communication in Linda". ACM
Transactions on Programming Languages and Systems, volume 7,
level entities to participate. Also the limited amount of number 1, January 1985
requirements may be easier to fulfill, even when the [13] Ian Oliver and Jukka Honkola. Personal Semantic Web Through A
behavior of the service level is not fully understood. This is Space Based Computing Environment. In Second IEEE Interntional
a situation which typically arises when using an existing Conference on Semantic Computing, August 4, 2008, Santa Clara,
CA, USA. IEEE, 2008
piece of software as a component of a smart space.
[14] OpenMAX APIs.http://www.khronos.org/openmax/. Referenced
Finally, as the activity is an abstract entity, it can exist on March 4th 2010
its own, independently of the underlying resources. This [15] L. Brownsword, et al, "Current Perspectives on
allows the resources to be reassigned based on need. Interoperability",Software Engineering Institute, Carnegie Mellon
University, Pittsburgh, PA, 2004.