1. The Initiative Experiments to Make Virtual and Real Space Seamless
Jong-Hyoun Kim1
Department of Gameware
Kaywon School of Art & Design
125 Naeson-dong, Euwang, Republic of Korea
hyoun@kaywon.ac.kr
Abstract
Nearly all of current networked games still use old-fashioned interfaces - a keyboard, a mouse or a
joystick at the most - which are out of touch with our daily lives. These interfaces don’t comprehend
our metaphors, don’t reflect on our natural behavior aspects, disrupt immersion in a gaming, and
become a serious impediment to perceiving virtual space as another extended real one. It is very
challenging issue to utilize human-being’s behaviors or reactions as a way of interacting to virtual
world. I attempted a few experiments on new game interfaces for a networked dice game and a TCG
game. For a user to communicate with virtual game applications, the former uses dice artifacts with
standardized markers which a camera may distinguish the spot on a die well, and the latter uses real
TCG cards as it is instead of untouchable cards on a deck of virtual space. These new game interfaces
guarantee the consistency between the game-player’s behaviors and the gaming environment. This
consistency increases friendliness and immersion to a game as well as the ability of bodily sensation
for a game. These new interfaces will open the new field of digital interfaces for networked games
including mobile environment.
Keywords: new game interface, marker/marker-less, camera- based
1. Introduction
I am embarking on the exploration of developing interfaces that can regain missing intelligences
and can revitalize dull creativities. To do this, interfaces must understand our metaphors, must solicit
information on its own, must acquire experiences, must talk to a wide variety of people, must improve
over time, and must be intelligent in context. To mirror that captures the essences and agendas
represented by contemporary digital information and communications society, designing digital
creativity is a critical asset to built tactical forces that blur the boundary between people and
information in new computing and communication culture.
Currently so many online games are serving. Nearly all of these networked games still depend on
traditional input interfaces only like a keyboard or a mouse though. These interfaces can’t comprehend
our metaphors, can’t reflect on our natural behavior aspects, can’t make us immerse into a game, and
makes a barrier between virtual game space and our real behavior. As a trial to overcome these
shortcomings I did two experiments one is marker-based the other is marker-less. The former uses dice
artifacts attaching markers which a camera to distinguish the spot on a die, and the latter uses the real
TCG cards as it is.
The remainder of this paper is organized as follows. Section 2 compares my approaches with
existing related work. Section 3 introduces my experiments on maker-based interface for the
networked dice game. Section 4 represents another trial on the online TCG game using marker-less real
images. Finally, I conclude with a note on the current status of my project and future work.
2. Related work
A variety of research efforts have recently explored computationally augmented interfaces that
emphasize human interaction using the human being’s behaviors and senses.
1
This research is supported by Ministry of Culture, Sports and Tourism(MCST) and Korea Creative
Content Agency(KOCCA) in the Culture Technology(CT) Research & Development Program 2010

2. 2.1. Direct manipulation interface
The Tangible Media group at MIT is creating a repertoire of physical devices for Tangible User
Interface (TUI) [1]. TUI seeks to build upon the sophisticated skills that people have developed for
sensing and manipulating our environment. The Bricks "Graspable User Interface" project involves
putting one or more bricks onto some screen-based virtual object [2]. Bricks can then be used to
physically rotate, translate, or scale and deform the attached virtual entities by manipulating the brick
devices. Direct Manipulative Interfaces are computationally enhanced versions of traditional children's
toys. The goal is to provide children with tools that enable them to explore complex concepts through
direct manipulation of physical objects [3].
2.2. Position and Orientation Sensor interface
The ScrollPad is a mouse-mounted PDA allowing the visualization of large documents while
scrolling it on a table [4]. The problem of all these techniques is that the user’s viewpoint to the screen
changes permanently, losing the optimal visualization angle.
2.3. Camera-based interface
MagicMouse allows the user to operate within both 2D and 3D environments by simply moving and
rotating their fist. Position and rotation around the X, Y and Z-axes are supported, allowing full six
degree of freedom input. This is achieved by having the user wear a glove, to which is attached a
square marker. Translation and rotation of the hand is tracked by a camera attached to the computer,
using the ARToolKit software library [5]. Wagner and Schmalstieg use a PDA camera to recover the
position of specific markers positioned in the real environment [6]. Rohs uses visual codes for several
interaction tasks with camera-equipped cell phones [7]. His IPARLA Project designed a new 3-DOF
interface adapted to the characteristics of handheld computers. This interface tracks the movement of a
target that the user holds behind the screen by analyzing the video stream of the handheld computer
camera. The position of the target is directly inferred from the color codes that are printed on it using
an efficient algorithm [8].
2.4. Sensor-based interface
Affective Computing Group at MIT is making an effort for sensing, recognizing, understanding, and
synthesizing the human behavior patterns. This group discussed the use of biometric sensors with
wearable computers. Such sensors allow for new interactions between the wearable and the wearer,
which they based upon affect detection, prediction, and synthesis [9].
No one of above approaches has been applied to online game environment. I think my experiments
are the first trial to apply camera-based interface including marker and marker-less methods into
networked game applications. I am sure these experiments will give a refreshing jolt to the related
researches and will open the new field of digital interfaces for networked games including mobile
environment.
3. Marker-based new interface for dice game
3.1. Online dice games
Dice games are games that use or incorporate a die as their sole or central component, usually as a
random device. There are many online dice games; Backgammon, Dudo, Jacquet, Strat-O-Matic, et
cetera. All of them requires mouse click to roll a die. Instead of mouse interface, I introduced new
marker-based input interface using ARToolkit for online dice games, especially Backgammon.
Backgammon is a board game for two players in which the playing pieces are moved according to the
roll of dice. A player wins by removing all of his pieces from the board.

3. Figure 1. A die artifact and pip detection process
3.2. System flow
This system has been implemented under ARToolkit, Microsoft Visual Studio 2008, and Windows
XP. This system has been tested using the Logitech QuickCam camera.
Figure 2. A dice game system flow
The artifact and its pip detection are shown in figure 1 and here is the flow of new interface system
as shown in figure 2.
a) The player in a game rolls the die artifact on the real floor twice.
b) As soon as the interface recognizes a spot on a die, it represents the digit on the screen.
c) The interface sends the digits to Dice Client (e.g. Backgamman application) using Window
Message Passing method.

4. d) Dice Client sends the packet to Dice Server.
e) Dice Server forwards the packet to the other party's Dice Client.
f) Dice Client displays the other party's pips.
g) The player moves his pieces on the board.
h) Now the opponent’s turn.
i) Repeats from a) to h) until one of the players removes all of his pieces from the board.
3.3. Packet structure
Here is the structure of major packets transferred to/from Dice Client to/from Dice Server. The
contents of packets are very simple. toServerPacket consists of 3 elements and toClenetPacket has 4
elements.
Table 1. Packet structure
// Enumeration
enum PIP {ONE=1, TWO, THREE, FOUR, FIVE, SIX}
// Packet from client to server
struct toServerPacket {
int id; // Client Identification
PIP face; // the spot on a die
int bet; // bet
};
// Packet from server to client
struct toClientPacket {
int id; // Client Identification
bool result; // result, win or lose
int numOfDrawer; // number of drawer
int earn; // earn
};
4. Marker-less new interface for TCG game
4.1. Online TCG Game
Figure 3. Two TCG real cards used in experiment

5. Each TCG system has a fundamental set of rules that describes the players' objectives, the
categories of cards used in the game, and the basic rules by which the cards interact. Each card will
have additional text explaining that specific card's effect on the game. During a game, players usually
take turns playing cards and performing game-related actions. Restore - Make all in-play cards ready
for the upcoming turn.
Draw card - Necessary in order to circulate cards in players' hands.
Play card - Use the cards in hand to interact with the game.
Conflict - The primary method for victory in most games.
Discard card - Discard to a maximum hand size, or need to refresh for next turn.
In the online environment, instead of receiving physical cards, a player establishes a virtual
collection that exists only as a set of data stored on a server. Such cards can be purchased or traded
within this environment.
For applying real TCG card into online TCG game I used two real TCG cards shown in Figure 3 for
experimenting purpose. One is Albertosaurus and the other is Spectral Armor Majungadaurus card.
4.2. Real card recognition system
Figure 4. Recognizing any position of the planar image

6. The recognition system based on Image Hot Spot Extraction Algorithm can recognize any position
of planar image and can be calibrated to recognize the tilted image within 20 degree above and below
both, so the user does not have any difficulties or does not need to train it to make the card be
recognized by a camera. The system can be scaled by the distance of images from a camera. Figure 4
and Figure 5 shows you how much recognize planar and declined image respectively. As shown in
Figure 4 and Figure 5, the recognition system may detect the image rotated by any angle, and may
catch the tilted image within about 25 degree above and below.
Figure 5. Recognizing the tilted image
The recognition system has been implemented under OpenCV2.0, Microsoft Visual Studio 2008,
and Windows XP. This system has been tested using the Point Gray’s Flea2 IEEE 1394b compatible
camera.
Figure 6 shows the picture after recognizing the specific cards. As shown in these pictures, after
detecting the card the annotating information (name, value of rock-paper-scissors. figure of power, et
cetera) of the card is displayed on the top-left corner of a screen.
Figure 6. After recognizing the cards
4.3. System flow
We use the whole structures of origin TCG games as is. In other words this means that we may
apply The Card Recognition System to origin TCG game application independently as like plug-in.
The system communicates to TCG client program through RPC message passing mechanism. In

7. addition to origin system we just attached a Map Table to catch Card ID from detected the image. So,
as soon as the Recognition System detects an image the system send Image ID to TCG Client. TCG
Client gets Card ID using the Image ID and sends it to TCG Server. The overall system flow is
depicted in Figure 7.
Figure 7. New TCG system flow
5. Conclusion and future work
I am embarking on the exploration of developing interfaces that can work with missing intelligence
and aesthetics. To do this, interfaces must understand our metaphors, must solicit information on its
own, must acquire experiences, must talk to a wide variety of people, must improve over time, and
must be intelligent in context. As part of this exploration, I attempted an experiment on camera-based
interface for networked dice game application and experimented for seamless gaming environment
between TCG virtual game space and our real behavioral space. The former experiment uses dice
artifacts putting a marker which the camera can distinguish the spot on a die, and uses marker-less real
card image for the latter. These new game interfaces guarantee the consistency between the game-
player’s behaviors and the gaming environment. This consistency increases friendliness and immersion
to a game as well as the ability of bodily sensation for a game. Nevertheless my trials are not complete
deployment to a running system served currently but just experiment covering a toy system, these new
interfaces, utilizing human-being's behaviors as input event, will open the new field of digital
interfaces for forthcoming ubiquitous computing.

8. I am still working on following areas.
Motion or gesture interface for networked new game interface
Game building design mixing real space and virtual space
Interaction mechanism to make real and virtual world seamless
6. References
[1] B. Ullmer, H. Ishii, “The metaDESK: Models and Prototypes for Tangible User Interfaces”,
Proceedings of UIST'97, 1997, 223-232.
[2] G. Fitzmaurice, H. Ishii, and W. Buxtori, “Bricks: Laying the Foundations for Graspable User
Interfaces”, Proceedings of CHI'95, 1995, 452-458.
[3] M. Resnick, F. Martin, R. Berg, R. Borovoy, V. Colella, Kramer, and B. Silverman, “Digital
Manipulatives”, Proceedings of CHI'98, 1998, 281-287.
[4] FALLMAN, D., LUND, A., AND WIBERG, M., “Scrollpad:Tangible scrolling with mobile
devices”, In Proceedings of the 37th Annual Hawaii International Conference on System Sciences
(HICSS’04), 2004
[5] E. Woods, P. Mason, M. Billinghurst, “MagicMouse: an Inexpensive 6-Degree-of-Freedom
Mouse”, In Proceedings of Graphite 2003, Feb 11th-13th, 2003, Melbourne.
[6] Wagner, D., and Schmalstieg, D., “First steps towards handheld augmented reality”, In
Proceedings of Seventh IEEE International Symposium on Wearable Computers, 2003.
[7] ROHS, M., “Real-world interaction with camera-phones”, In 2nd International Symposium on
Ubiquitous Computing Systems (UCS 2004), 2004.
[8] M. Hachet, J. Pouderoux, and P. Guitton, “A camerabased interface for interaction with mobile
handheld computers”, In Proceedings of ACM symposium on 3D interactive graphics and games
(I3D’O5). 2005.
[9] R.W. Picard and J. Healey, “Affective Wearables”, Proceedings of the First International
Symposium on Wearable Computers, IEEE, October 13–14, 1997, 90–97.