Virtual Realty for Entertainment - æćă»èšèȘèæŻă æćă»èšèȘèæŻăè¶ ăăç°ćąă«ăăăćœéçăšăłăżăă€ăĄăłăă·ăčăă ăă¶ă€ăłAkihiko Shirai
Virtual Realty for Entertainment - æćă»èšèȘèæŻă æćă»èšèȘèæŻăè¶ ăăç°ćąă«ăăăćœéçăšăłăżăă€ăĄăłăă·ăčăă ăă¶ă€ăł
MANPU2016 Invited Talk: Manga Generator, a future of interactive manga media
1. Manga Generator,
a future of interactive manga media
MANPU2016: Invited Talk
The First International Workshop on
coMics ANalysis, Processing and Understanding
1
Akihiko SHIRAI, Ph.D
Kanagawa Institute of Technology, JAPAN
@o_ob
2. Akihiko SHIRAi in 3 min
YumeNaviLive , pitch talk for high school students.
3. Akihiko SHIRAI, Ph.D (Engineering)
Department of Information Media, Kanagawa
Institute of Technology (KAIT), Japan
Chair of Laval Virtual ReVolution (2006~)
Executive Commttiee of International collegiate
Virtual Reality Contest (2002~)
ACM SIGGRAPH ASIA 2015 Youth Program Liaison
AKIHIKO SHIRAI, Ph.D in Engineering, has obtained a
bachelor of Photo Engineering and master of Image
Processing from Tokyo Institute of Polytechnics. Afterwards
he worked for Canon and Criterion as a game development
consultant to distribute RenderWare, a multi-platform
graphics middleware for the game industry until 2001. He
went back to academia to study intelligent systems and
obtained a Ph.D. in the Tokyo Institute of Technology in
Japan in 2004 with research concerning the âTangible
Playroomâ, an entertainment system for young children
using haptics, a floor screen and a physics engine. He was a
R&D researcher at NHK-ES in Japan, focusing on the next
generationâs TV production environment before moving to
ENSAM Presence & Innovation Laboratory in France from
2004 to 2007 for R&D of a Virtual Reality theme park project.
He worked for National Museum of Emerging Science and
Innovation (Miraikan), Tokyo Japan, as a science
communicator and exhibition planner from 2008 to 2010.
Starting from 2010, he works in the Information Media
Department at Kanagawa Institute of Technology (KAIT) as
an associate professor.
4. Manga Generator,
a future of interactive manga media
* Motivation from Student competition project to International new media
* Technology of Manga Generator
* IVRC and Laval Virtual
* Entertainment systems and Sensory motor play
* Research, Results and Deployments
* Collaboration with professional manga
* Player analysis
* Contents toolkit
* Future
* MGV: Multiplex Digital Signage
* Next generation as âInteractive Manga Mediaâ
6. Slide - 1
Motivation of Manga Generator
âą Entertainment system using VR technology.
âą Enjoyable to people for all.
âą Exploring future of âmedia mixedâ creation.
âą Tell Japanese manga culture to the world.
8. Slide - 3
Japanese manga culture
âą âShonenâ category for boyâs
Ă âDive into the manga worldâ is a boyâs dream.
Ă Children play âMake-Believe playâ as a hero.
âą Animated manga (Anime)
Ă Itâs a quite big media. However,
it is not an interactive content.
Imagination is not needed for audience
9. Slide -4
Daiki UMEDA
âReal-time manga-like depiction based on interpretation
of bodily movements by using kinectâ, 2012
Related Research
âComic Chatâ, 1996âComi Po!â, 2010
âManga cameraâ, 2012Masaki Hayashi
âResearch on automatic manga generation based on TVML scriptâ, 2002
10. Prototype
alpha
Version
beta1
Version
beta2
Version
RC1
Laval
Version
Slide - 7
2012/8 2012/9 2012/10 2012/11
ă» Manga-like Visual
ă» Automatic Layout Adjuster
ă» Set a story
ă» Dynamic Mood Selector
ă»Set a multi-story
*IVRC2012
semifinal
*IVRC2012
final
*TV AppearanceïŒTBS
*TV AppearanceïŒTV TOKYO
Road map
Better expression using user posture
2013/3
17. Slide - 11
Awarded
âą IVRC2012(International virtual reality contest)
âą Asia Digital Art Award
âą Asia student creative competition
âą Laval Virtual 2013 Award
18. Slide - 14
Conclusion in 2013
âą Entertainment system using VR technology
âą Enjoyable to people for all ages
âą The future of the media mixed creation
âą New promotion system
âą Tell the world of Japanese manga culture
19. Manga Generator,
a future of interactive manga media
* Motivation from Student competition project to International new media
* Technology of Manga Generator
* IVRC and Laval Virtual
* Entertainment systems and Sensory motor play
* Research, Results and Deployments
* Collaboration with professional manga
* Player analysis
* Contents toolkit
* Future
* MGV: Multiplex Digital Signage
* Next generation as âInteractive Manga Mediaâ
20. What is IVRC?
* International collegiate Virtual Reality Contest establish
in 1993 by Prof. Susumu Tachi
* Contest of interaction
and virtual reality, which
fuses art and technology
* Honors the greatest Virtual Reality and Interactive
works of students
* New education system through contest
21. What is IVRC?
* Organized by Virtual Reality Society of Japan
* Supported by Ministry of Economy, Trade and Industry and
Ministry of Education, Culture, Science and Technology
* Ambassade de France au japon, Service pour la Science et la
Technologie
* Held at Miraikan, National science museum of Emerging Science
and Innovation
Virtual Reality Society of Japan
http://www.vrsj.org/
22. IVRC: contest of the students, by the students, for the students
l IVRC is the acronym of âInternational collegiate Virtual Reality Contestâ, which has been held since 1993
and is the contest of Interactive products using advanced technologies such as virtual reality and robots.
Our English name changed for internationalization in 2004, because we began to invite awarded teams
from the French VR contest âLaval Virtualâ.
l When you hear the words such as âVirtual Realityâ and ârobotâ, you should feel that special expensive
machines are necessary to realize them. However, there are many ideas which can be actualized only
with handmade machines. The main purpose of this contest is for participants to gain deeper
understanding of and familiarity with interactive technologies through activities such as planning
interactive systems, making devices themselves, and improving quality of their contents.
l IVRC has not only the function of contest but also the function of education. It aims to construct the
new education system of the 21st century to develop excellent human resources, who can think, learn,
and work by themselves.
planning technology design
Team
management
23. Call for Submission
Deadline of submission
2nd stage, Demonstration
Final stage, Demonstration
SIGGRAPH/ Laval Virtual
Online submission
Improvement phase
1st
stage, Paper review
Development phase
IVRC Flow Chart
4 +1 projects
Annual Conference of VRSJ
DCEXPO: Digital Content Expo
April
Late June or Early July
September
October
24. * Semi final at the annual conference of the Virtual Reality Society of
Japan (9-11th September, Tokyo)
Demonstration in academic conference
* Grand final in Digital Contents EXPO 2015
22-25th October, Miraikan, Tokyo.
Demonstration in Public
* U-18: Youth section
* International Video Selection for the final stage
25. The First Presence at ACM SIGGRAPH 2002
* IVRC Committee helps the champion team submit to SIGGRAPH as
a reward to the winner.
ĂâVirtual Chambaraâ, SIGGRAPH2002
âVirtual Chambaraâ,
SIGGRAPH 2002
26. History of IVRC Presence at SIGGRAPH
* 2003 THE DIMENSION BOOK
* 2004 Fragra / Dis-Tansu
* 2005 Straw-like user interface / Kobito: Virtual Brownies / Ton2
* 2006 bubble cosmos / INVISIBLE ~The Shadow Chaser~ /
Powder Screen: A Virtual Materializer âŠ..
27. History of IVRC Presence at SIGGRAPH
* 2007 BYU-BYU-View / CoGAME
* 2008 Ants in the Pants / Landscape Bartender
* 2009 Baby type robot âYOTAROâ / Back to the mouth /
Funbrella: making rain fun
* 2010 Haptic Canvas
* 2011 Mommy Tummy
* 2015 Childhood/ vibroSkate
28. International Cooperation with Laval Virtual
âThe winner of the IVRC Award in France will be invited to the Tokyo
stage of the IVRC in Japan as a challenger. Similarly, the winner of the
Laval Virtual Award in Japan will be invited to the final stage of Virtual
Fantasy, category âDemosâ, student competition of Laval Virtual, in
France as a challenger. Conditions of invitation will be defined every year
by each part of this agreement.â
29.
30. Agreement on International Cooperation
at Laval Virtual 2004
IVRC and Laval Virtual will maintain an award partnership for
three years (2004-2007): the Laval Virtual Award at the IVRC
in Japan, the IVRC Award at âLaval Virtualâs Le Village de la
Creationsâ in France.
J.F. Fontaine
(Laval Virtual)
Prof. S. Tachi
(IVRC)
31. International Cooperation with ETC, CMU
âThe best work certified by CMU among other works created as student
project will be invited for the final contest of IVRC. The invited work will be
assessed in the final contest in the same way as other selected works in the
preliminary contest of IVRC and will be entitled to receive each award of
IVRC.â
33. http://ivrc.net/
Carnegie Mellon University (CMU) Entertainment Technology
Center (ETC) and International collegiate Virtual Reality Contest
(IVRC), in recognition of the growing importance of the roles
Virtual Reality and Entertainment Technology play in the 21st
century, have come to the agreement:
Agreement on International Cooperation in Virtual
Reality and Entertainment Technology
Michelle Macau
(ETC) Ambassador
Agent for Donald Marinelli
Executive Producer of ETC
Prof. S. Tachi
(IVRC)
34.
35. 3D moves in 30-year cycles
VR moves 10 years afterïŒD crazes
1st
3D Craze
ïŒthe1950s)
Dawn of
VR
(the1960s)
1940 1960 1980 2000 20201920
Dawn of
3D
ïŒthe1920s)
2nd
3D Craze
ïŒthe1980s)
3rd
3D Craze
ïŒthe2010s)
2nd
VR Wave
(the2020s)
?
1st
VR Wave
(the1990s)
movie movie
Theme-park
computer
game
TV
movie movie
36. Virtual Reality in the 1990s
* 1989 VPL Research Data Glove and Eyephone. âVirtual reality" was first used.
* 1990 Santa Barbara Conference (The Engineering Foundation)
Virtually all the researchers were summoned.
* 1991 ICAT (International Conference on Artificial Reality and Telexistence) was
held in Japan.
* 1992 Committee on Virtual Reality was established in National Research Council.
Its report was published in 1995.
* 1993 IVRïŒIndustrial Virtual Realty) was held in Japan.
* 1993 VRAIS (Virtual Reality Annual International Symposium) and Research
Frontiers in Virtual Reality were held. They gathered together to form IEEE
Virtual Reality Conference.
* 1993 IVRCïŒInternational Virtual Realty Contest) was held.
* 1996 VRSJ (Virtual Reality Society of Japan) was established.
* 2001 IEEE Virtual Reality Conference was held in Japan.
* 2008 National Academy of Engineering (NSF) announced 14 grand engineering
challenges for the 21st century on February 15, 2008. Enhance Virtual Reality is chosen as
one of the 14 challenges.
37. 23 yearsâŠ
* More than 200 projects have been realized through 23 IVRCs
including 21 projects selected as SIGGRAPH E-Tech.
* Average 5 persons for each project mean over 1,000 persons
have deeply experienced and understood VR!
* IVRC students become staffs, and staffs now become world
famous professors, designers, creators, and engineers.
39. 21 CENTURY'S GRAND ENGINEERING
CHALLENGES
* National Academy of Engineering announced 14 grand
engineering challenges for the 21st century on February 15, 2008.
http://www.engineeringchallenges.org/
* The final choices fall into four themes that are essential for
humanity to flourish -- sustainability, health, reducing
vulnerability, and joy of living.
* Tremendous advances in quality of life have come from
improved technology in areas such as farming and
manufacturing. If we focus our effort on the important grand
challenges of our age, we can hugely improve the future.
* Enhance Virtual Reality is chosen one of the 14 challenges.
41. 3D moves in 30-year cycles
VR moves 10 years afterïŒD crazes
1st
3D Craze
ïŒthe1950s)
Dawn of
VR
(the1960s)
1940 1960 1980 2000 20201920
Dawn of
3D
ïŒthe1920s)
2nd
3D Craze
ïŒthe1980s)
3rd
3D Craze
ïŒthe2010s)
2nd
VR Wave
(the2020s)
?
1st
VR Wave
(the1990s)
movie movie
Theme-park
computer
game
TV
movie movie
45. * Keynotes
* Workshop + Aural Presentation
ACM VRIC: Virtual Reality International Conference
Dr. Pattie Maes, MIT Media LaboratoryPr Alain Berthoz, College de France
Virtual avatars and real brains: how do they interact? AR and VR for learning and collaboration
Dr. Oliver Bimber, Johannes Kepler Univ. Linz
Light-Field Technology: A Revolution to Imaging and Display
Dr. Carolina Cruz-Neira, Emerging Analytics Center, University of Arkansas at Little Rock
Beyond Fun and Games: VR as a Tool of the Trade
Christine Perey, AR for Enterprise Alliance
The Future of Work with Augmented Reality
Dr. Skip Rizzo
Director for Medical Virtual Reality Institute for Creative
Technologies - Research Professor USC Davis School of
Gerontology and USC Keck School of Medicine Department of
Psychiatry & Behavioral Sciences
Beyond Video Games:
A Virtual Reality Revolution in Behavioral Health
47. Special Session on Friday
14:00-14:20 Opening Address âReal-Virtualityâ, Prof. Akihiko SHIRAI (KAIT)
14:20-14:30 âIVRC to Laval Virtual / Jack and the Beanstalkâ, Mr. Shota SUGIMOTO, (Univ Keio)
14:30-14:50 âEmbodied Mediaâ, Prof. Kouta MINAMIZAWA (Univ keio)
14:50-15:00 âMetamorphosis Hand: Interactive Experience of Embodying Virtually Transformed Handsâ, Nami
Ogawa (Univ of Tokyo)
15:00-15:30 âVR to Superhuman Sportsâ Prof. Masahiko INAMI (Univ of Tokyo)
15:30-16:00 Round table and debate, exchanges with the audience
ACM VRIC: Virtual Reality International Conference
http://superhuman-sports.org/
48. Manga Generator,
a future of interactive manga media
* Motivation from Student competition project to International new media
* Technology of Manga Generator
* IVRC and Laval Virtual
* Entertainment systems and Sensory motor play
* Research, Results and Deployments
* Collaboration with professional manga
* Player analysis
* Contents toolkit
* Future
* MGV: Multiplex Digital Signage
* Next generation as âInteractive Manga Mediaâ
49. [definition] Entertainment system is System
which affects to human amusement.
Video game, museum exhibition, entertainment VR, media arts, e-sports,
broadcasting, social networkâŠ
1. Device and/or System
2. Service and/or Contents
3. Evaluation methods
Research in Entertainment system
50. * Jean Piaget (1896-1980) Swiss clinical psychologist known for his pioneering work in
child development.
* Sensorimotor play
* Symbolic play
* Rule Play
âSensory-motor Playâ
52. Manga Generator,
a future of interactive manga media
* Motivation from Student competition project to International new media
* Technology of Manga Generator
* IVRC and Laval Virtual
* Entertainment systems and Sensory motor play
* Research, Results and Deployments
* Collaboration with professional manga
* Player analysis
* Contents toolkit
* Future
* MGV: Multiplex Digital Signage
* Next generation as âInteractive Manga Mediaâ
53. Research, Results and Deployments
*Collaboration with professional manga
*Player analysis
*Contents toolkit
60. * ExPixel is a Pixel-Shader Technology that can show multiple images on one screen at
same time.
* ExPixel
* Extended Pixel
* X-Pol (FPR for 3D)
* Extra ImageâŠ
What is ExPixel
61. Related Works: Multiplex display
* Direction based separation
* âBois-Clairâ (Gaspar Antoine, 1692)
* âVari-Vueâ (Victor Anderson, 1962)
* âLumisight Tableâ(2005), âUlteriorScapeâ (2008) by Kakehi
* Time/Shutter based technologies
* âSnared illuminationâ (fakespace lab, 2004)
* âThirdEyeâ (Pranav Mistry, 2009)
* âSimulViewâ (SONY, 2010-)
* Polarization based technologies
* âFairyFinderâ (Hachiya, 2005)
* âMysterious Pondâ, âA visibility control system for collaborative digital
tableâ (Kitamura, Sakurai. 2009)
* Hidden but not shown for naked eyes
* Patent: JPA2005-092227 (Semiconductor Energy Lab, 2005)
62. *Adding a new value of 3D devices.
*Realizing multiplex-hidden imagery on a consumer 3D flat
panel without any hardware modification.
*Providing content creation environment for wider
creators.
Challenge
63. Scritter to ExPixel: Scritter
NAGANO, K., UTSUGI, T., HIRANO, M., HAMADA, T., SHIRAI,A., AND NAKAJIMA, M. 2010.
A new âmultiplex contentâ displaying system compatible with current 3d projection technology.
In ACM SIGGRAPH 2010 Posters, ACM, New York,NY, USA, SIGGRAPH â10, 79:1â79:1.
Multiplex by
two projectors.
64. Scritter to ExPixel: ScritterH
Wataru FUJIMURA, Yukua KOIDE, Robert SONGER, Takahiro HAYAKAWA, Akihiko SHIRAI, Kazuhisa YANAKA,
"2x3D: Real-Time Shader for Simultaneous 2D/3D Hybrid Theater",
ACM SIGGRAPH ASIA 2012 Emerging Technologies, Singapore, 2012
Hidden imagery
with two projectors.
65. ScritterH, Projector based hidden imagery
* Merge an arbitrary image A for
naked eyes and an image B that
you would like to hide to generate
a different image C and multiple-
project B and C.
* Image A visible to naked eyes while
image B with glasses.
* Figure from ScritterH project
66. * Naked eye: Left + Filter: Right.
* It is Implemented as a GPU based shader.
* Good for theater but not suitable for consumer usage.
2x3D: Hybrid 2D + 3D display
Wataru FUJIMURA, Yukua KOIDE, Robert SONGER, Takahiro HAYAKAWA, Akihiko SHIRAI, Kazuhisa YANAKA,
"2x3D: Real-Time Shader for Simultaneous 2D/3D Hybrid Theater",
ACM SIGGRAPH ASIA 2012 Emerging Technologies, Singapore, 2012
67. * Scritter/ScritterH doesnât need synchronized devices.
Only needs polarization filter to separate two channels.
* However, it still needs specific products such as DMD projector,
polarization supported screen.
* It is difficult to prepare two projectors as a stuck.
ScritterH, Projector based hidden imagery
68. Algorithm of ExPixel (1/3)
With a circular
polarization filter
Contrast compression
72. Algorithm of ExPixel (3/3)
With a circular
polarization filter
Even odd selection
73. Line-by-line: Even odd selection
R
L
Image A
Image B
R
L
R
L
R
L
Display
P polarization
S polarization
P polarization
S polarization
P polarization
S polarization
Normal Use: Parallax images
ExPixe Use: Arbitrary images
80. Using Scene Naked Eye Hidden Eye Experience
Presentation Listener Presenter Only presenter can check scenario.
Classroom No hint Hint available Show hints for student who needs hint.
Museum Children Adult Explications for different age.
Shop Non
member
Member Show special price for registered member.
Sightseeing Children Adult Explications for different interests.
Medical Patient Doctor Supplementary information which should not
be shown to patients.
Municipal
services
Local
residents
People from
abroad
Communication tool between domestic
people and foreigner.
Various applications
81. * Converter
* XNA
* Static image
* Video
* Unity
* Static image
* Video
* Game
* AR
* Presentation
* Player Shader model 3.0 Converter
* Processing
* PHP
* Static image
* FPGA Hardware
Content creation tools
82. *Realizing multiplex-hidden imagery on a consumer 3D flat
panel without any hardware modification.
*Providing content creation environment for wider
creators.
*Adding a new value of 3D devices.
Conclusion
83. *Why it cancel the hidden image while canceling image and
hidden image are not on the same line?
*Resolution of display is very high that people canât
recognize pixels.
*It is not recognized as a pixel,
(because itâs same size as pixels!!)
ExPixel = Contrast compression + sub pixel
85. * Contrast
* It needs to reduce each imageâs contrast according to current algorithm.
* Resolution
* Vertical resolution became half, but horizontal resolution can be kept actual amount.
* Image with black background for Naked Image
* After pass contrast compression step, black color level 0 is pushed up to gray color level 127.
Weak Points
88. Manga Generator,
a future of interactive manga media
* Motivation from Student competition project to International new media
* Technology of Manga Generator
* IVRC and Laval Virtual
* Entertainment systems and Sensory motor play
* Research, Results and Deployments
* Collaboration with professional manga
* Player analysis
* Contents toolkit
* Future
* MGV: Multiplex Digital Signage
* Next generation as âInteractive Manga Mediaâ
92. * We would mention thanks to all
contributors of Manga Generator
project, Yukua KOIDE, Wataru
FUJIMURA, Yuto NARA, Genki
KUNITOMI, Kosaku OGAWA,
Ayumi ANDO, Ryo OKAMOTO,
Hisataka SUZUKI and Rex HSIEH.
* The project also continue parallel
with âManga Generator Proâ by
Progmind Inc.
* www.progmind.jp/service/manga-
generator/
ACKNOWLEDGMENTS
93. Manga Generator,
a future of interactive manga media
* Motivation from Student competition project to International new media
* Technology of Manga Generator
* IVRC and Laval Virtual
* Entertainment systems and Sensory motor play
* Research, Results and Deployments
* Collaboration with professional manga
* Player analysis
* Contents toolkit
* Future
* MGV: Multiplex Digital Signage
* Next generation as âInteractive Manga Mediaâ
94. Q: Original Authors says yes for Manga
Generator?
A: We have afraid with our respect for the
authors but authors are already trusted
their publishers and Photoshop toolkit
helped the understanding.
Q: Dynamic scenario is possible?
A: Yes, âManga Generator KAIâ project
successes to implement dynamic scenario.
https://twitter.com/mangageneratorK
Discussion from audiences