1. WP 2 Presentation
Stefan Goetze

2. Structure for the Talk
• goals
• achievementsachievements
• problems
• outlook/ future workoutlook/ future work
• lessons learned
• list of publications• list of publications
12/06/2013 WP 2 Presentation 2

3. Goals in WP2 (Human-Machine-Interface)
• T2.1 Creation and implementation of a symbolic keyboard for elderly
T2 2 D fi iti f th d t d b l• T2.2 Definition of the day to day vocabulary usage
• T2 3 Creation and implementation of an interface to connect differentT2.3 Creation and implementation of an interface to connect different
modules
• T2.4 Creating a speech activated user interface
• T2 5 Software modules merger• T2.5 Software-modules merger
• T2.6 Final adaption of the software componentsp p

4. Achievements - Virtual Keyboard (T2.1)
• Third-party software
I d i h Ali• Integrated in the Alias
GUI
S pports a to completion• Supports auto-completion
• Used e.g. for web-
browser and Skype chatbrowser and Skype chat

5.  Users want to interact with ALIAS system by various phrases
Definition of the day to day vocabulary usage
 Users want to interact with ALIAS system by various phrases.
~~~ Alert Scenario ~~~
alias hilfe
robin alarm
alias alarm
robin hilfe
~~~ Game Scenario ~~~
zeig mir die spiele
zeig mir deine spiele
oeffne bitte die spieleliste
oeffne die spieleliste
~~~ Telefone Scenario ~~~
bitte ruf bob an
ruf bob an
bitte ruf felicitas an
ruf felicitas an
~~~ Navigation Scenario ~~~
robin komm her
alias komm her
komm bitte her
alias komm mal rueber
bi k l bi b
spieleliste oeffnen
bitte spieleliste anzeigen
spieleliste anzeigen
ich moechte spielen
ich will spielen
spiele anzeigen
bitt i i l li t
ich moechte bob anrufen
ich moechte felicitas anrufen
ich moechte telefonieren
bitte bob anrufen
bitte felicitas anrufen
bitte zeige kontaktliste an
bi i k k lirobin komm mal bitte rueber
robin komm mal rueber
alias komm mal bitte rueber
komm mal rueber
komm mal bitte rueber
alias komm zu mir
komm bitte zu mir
bitte zeige spieleliste
starte solitaer
starte schach
starte sudoku
starte tic_tac_toe
bitte schach starten
bitte sodoku starten
bitte zeige kontaktliste
zeige kontaktliste an
zeige kontaktliste
bitte zeige telefonliste an
bitte zeige telefonliste
zeige telefonliste an
zeige telefonlistekomm bitte zu mir
robin komm naeher
robin komm bitte naeher
alias komm naeher
alias komm bitte naeher
geh bitte zur seite
robin geh zur seite
bitte sodoku starten
bitte tic_tac_toe starten
bitte solitaer starten
schach starten
sodoku starten
tic_tac_toe starten
solitaer starten
zeige telefonliste
kontaktliste anzeigen
telefonliste anzeigen
kontaktliste oeffnen
telefonliste oeffnen
starte skype
skype startenrobin geh zur seite
robin geh bitte zur seite
alias geh bitte zur seite
mach platz
mach bitte platz
nicht weiter gehen
halt stopp
solitaer starten
spiel beenden
~~~ Internet Scenario ~~~
browser starten
internetbrowser oeffnen
starte den internetbrowser
skype starten
bitte anhalten
du stehst im weg
starte bitte das internet

6. Interface to Connect Different Modules (T2.3)
• Modules centered
around the Dialogue
ManagerManager
• Input interfaces
• Cameras
• Ultrasonic Sensors
• Laser-Scanner
• MicrophonesMicrophones
• BCI
• Touch-Screen
O f• Output interfaces
• Loudspeaker
• Screen

7. Achievements - Speech Activated GUI (T2.4)
• Menu structure
• New web-basedNew web based
eMail module
12/06/2013 WP 2 Presentation 8

8. Speech activated user interface (T2.4)

9. Achievements - New GUI Design (I)
12/06/2013 WP 2 Presentation 10

10. Achievements / New GUI Design (II)
12/06/2013 WP 2 Presentation 11

11. D l d I i fDevelopment and Integration of a game
collection (related to T3.4)
• Available games are chess, Solitair, Sudoku and Tic-Tac-Toe.
• The “WinTV 7” software has been integrated with the GUI that enables access
to the TV tuner hardware => watching TV and using the wiito the TV tuner hardware => watching TV and using the wii

12. Achievements / Tic-Tac-Toe
• New visual design
• AI opponentAI opponent
• Bug fixes
12/06/2013 WP 2 Presentation 13

13. Achievements / BCI Integration and Test
• We were able to run the BCI simulation on the robot
• Still some bugs and problems
 BCI software still under development
12/06/2013 WP 2 Presentation 14

14. Skype Interface
• New integrated functions:
 Video telephony
 Contact list
 Chat

15. Achievements / Video Overlay
• Added video overlay functionality to display tutorials
• Supports DviX/XviD videos with MP3 sound by defaultSupports DviX/XviD videos with MP3 sound by default
12/06/2013 WP 2 Presentation 16

16. T2.4 - Statusbar
• Speech recognition status display / toggle button
• BCI overlay status display / toggle buttonBCI overlay status display / toggle button
• A clock

17. T2.4 Implementation of an Alarm Count Down
• Emergency call possible everywhere• Emergency call possible everywhere

18. Achievements / Microphones
• Supplied partners Cognesys and TU Ilmenau with
microphones and pre-amps in order to operate the ASRp p p p
12/06/2013 WP 2 Presentation 19

19. Creating a speech activated user interface
ASR Device (T2.4)
 Achieved steps:
A l d f t lk ASR t h b d l d A close- and a far-talk ASR system has been developed.
 Close-talk recognizer can be used for exhibitions etc.
 Far-talk recognizer has been used for the mid-term review.g
 Furthermore, the dialog-manager gets parallel input from a strict and a
soft keyword spotter.
 Noteworthy progress towards large
vocabulary speech recognition, though
additional work requiredadditional work required

20. I t ti f W b B i t GUI ( l t dIntegration of Web Browser into GUI (related
to TT3.5 Web 2.0 wrapper for web services)
• Decision has been made to use the QtWebKit (less problems with integration
into GUI compared to FireFox etc.)

21. Problems / GUI
I t ti f thi d t ft / f k i• Integration of third party software / frameworks is
cumbersome
• The Qt WebKit framework causes several conflicts with• The Qt WebKit framework causes several conflicts with
the surrounding Qt framework
– Browser plug-ins don’t work properly, may crash the wholeBrowser plug ins don t work properly, may crash the whole
application, or have adverse effects on the entire GUI
– New HTML5 standard not fully supported
• TV Module (often) doesn’t work due to bad reception
• Audio playback distorted by interference from the touch-
itscreen unit
12/06/2013 WP 2 Presentation 23

22. Problems / ASR
• True Large Vocabulary Speech Recognition more
cumbersome than expectedp
• A lot of different options for improvements to evaluate
• Still some work requiredq
12/06/2013 WP 2 Presentation 25

23. Outlook / Future Work
• Further improvements on the ASR system
12/06/2013 WP 2 Presentation 26

24. Lessons Learned
• (Social) robot projects lead to “cool demonstrators”
• As usual: Many things take longer than expectedAs usual: Many things take longer than expected
12/06/2013 WP 2 Presentation 27

25. Alias2Market
Identification of the economic utility by:
• Technology Research
– Direct competitors regarding assistance in communications
(not limited to robots)
Identification of (true) USPs– Identification of (true) USPs
• Market Research
– Identification of addressable markets or market niches
– size of market, quantification of demand
– willingness to pay
– necessary / optional features
survey of USPs– survey of USPs
12/06/2013 WP 2 Presentation 28

26. Alias2Market
• markets in scope:
– professional health care
– nursing facilities / homes
– private homes
• acquirement by users• acquirement by users
• acquirement by relatives
• survey of alternative financing
models
leasing– leasing
– combination of leasing &
accompanying services
12/06/2013 WP 2 Presentation 29

27. List of Publications
• Moritz N, Goetze S, Appell JE. 2011. Ambiente Sprachsteuerung für einen Persönlichen Aktivitäts-und
Haushaltsassistenten. Ambient Assisted Living-AAL., AAL Kongress, Berlin
• Moritz N, Goetze S, Appell J-E. 2011. Ambient Voice Control for a Personal Activity and Household Assistant.
Springer Verlag.
M it N G t S A ll J E 2011 A bi t S h t fü i P ö li h Akti ität d• Moritz N, Goetze S, Appell J-E. 2011. Ambiente Sprachsteuerung für einen Persönlichen Aktivitäts- und
Haushaltsassistenten. VDE, 4. Deutscher AAL-Kongress, Berlin.
• Moritz N, Anemüller J, Kollmeier B. 2011. Amlitude Modulation Spectrogram Based Features for Robust
Speech Recognition in Noisy and Reverberant Environments. ICASSP 2011.
• Schröder J Wabnik S van Hengel PWJ Goetze S 2011 Detektion und Klassifikation von akustischen• Schröder J, Wabnik S, van Hengel PWJ, Goetze S. 2011. Detektion und Klassifikation von akustischen
Ereignissen zur häuslichen Pflege. 4. Deutscher AAL-Kongress.
• Gerlach S, Goetze S, Bitzer J, Doclo S. 2011. Evaluation of Joint Position-Pitch Estimation Algorithm for
Localising Multiple Speakers in Adverse Acoustical Environments. DAGA. :633-634.
• Kortlang S, Schröder J, Hollosi D, Anemüller J, Kollmeier B. 2011. A Hierarchical Approach to Content-Basedo t a g S, Sc öde J, o os , e ü e J, o e e 0 e a c ca pp oac to Co te t ased
Classification of Environmental Sounds Using a Predefined Taxonomy. DAGA 2011.
• Kodrasi I, Rohdenburg T, Doclo S. 2011. Microphone Position Optimization for Planar Superdirective
Beamforming.
• Moritz N, Anemüller J, Kollmeier B. 2011. Modulation Feature Extraction for Robust Automatic Speech
Recognition. DAGA 2011.
• Jens Schröder, Jan Rennies FXJASG. 2011. Real-time Room Reverberation Estimation for Online Speech
Intelligibility Monitoring. DAGA 2011.
• Rehr R, Goetze S, Hollosi D, Appell J-E, Bitzer J. 2011. Speech / Non-Speech Discrimination for Acoustic
Monitoring Considering Privacy Issues Proc 37th Annual Convention for Acoustics (DAGA)Monitoring Considering Privacy Issues. Proc. 37th Annual Convention for Acoustics (DAGA).
12/06/2013 WP 2 Presentation 30

28. List of Publications
Wilk S G S H ll i D A ll J E Bi J 2011 S h A i i D i f A i i M i i• Wilksen S, Goetze S, Hollosi D, Appell J-E, Bitzer J. 2011. Speech Activity Detection for Activity Monitoring
using an Embedded Platform. Proc. 37th Annual Convention for Acoustics (DAGA).
• Schröder J, Wabnik S, van Hengel PWJ, Goetze S. 2011. Detection and Classification of Acoustic Events for
In-Home Care. Ambient Assisted Living. :181-195.
• Rennies J Goetze S Appell J E 2011 Human Centered Design of E Health Technologies: Concepts Methods• Rennies J, Goetze S, Appell J-E. 2011. Human-Centered Design of E-Health Technologies: Concepts, Methods
and Applications. Human-Centered Design of E-Health Technologies: Concepts, Methods and Applications. pp.
180–207.
• B. Cauchi, S. Goetze, S. Doclo: „Reduction of Non-stationary Noise for a Robotic Living Assistant using Sparse
Non-negative Matrix Factorization“. In: Proc. Speech and Multimodal Interaction in Assistive Environments
(SMIAE2012), Jeju Island, Republic of Korea, Juli 2012.
• S. Goetze, S. Fischer, N. Moritz, J.-E. Appell, F. Wallhoff: „Multimodal Human-Machine Interaction for Service
Robots in Home-Care Environments“. In: Proc. Speech and Multimodal Interaction in Assistive Environments
(SMIAE 2012), Jeju Island, Republic of Korea, Juli 2012.
M R hl d d S G t C t ti l Effi i t N i R d ti f Di l S t i C E i t• M. Ruhland and S. Goetze: „Computational Efficient Noise Reduction for Dialogue Systems in Car Environments
based on Binary Time-Frequency Masking and Autoregressive Interpolation“, Workshop on „Dialog systems that
think along – Do they really understand me?“, 35th German conference on Artificial Intelligence (KI 2012),
Saarbrücken, Germany, Sept. 2012.
• M. Ruhland, S. Goetze, M. Brandt, J. Bitzer and S. Doclo: „A New Approach for Reduction of SupergaussianM. Ruhland, S. Goetze, M. Brandt, J. Bitzer and S. Doclo: „A New Approach for Reduction of Supergaussian
Noise using Autoregressive Interpolation and Time-Frequency Masking“, In Proc. International Workshop on
Acoustic Signal Enhancement (IWAENC 2012), Aachen, Germany, 4. – 6. September 2012.
• N. Moritz, J. Anemüller, B. Kollmeier: „Amplitude Modulation Filters as Feature Sets for Robust ASR: Constant
Absolute or Relative Bandwidth?“. In: Proc. InterSpeech 2012, 13th Annual Conference of the International
S h C i ti A i ti P tl d O USA S t b 9 13 2012Speech Communication Association, Portland, Oregon, USA, September 9-13, 2012.
12/06/2013 WP 2 Presentation 31

29. Student Work
Internship• Internship
Thomas Tomczyszyn: Integration of Skype Video Telephony and Web-
Browsing in a common Graphical User Interface
• Bachelor thesis
Olga Schwarz: „Untersuchung zur Usability von hierarchischen und planen
M ü t kt B i i l d R b t l ttf ALIAS“Menüstrukturen am Beispiel der Roboterplattform ALIAS“
• Bachelor thesis
Dörte Fischer: „Ansätze zur blinden adaptiven einkanaligen Entzerrung für
akustische Sprach- und Ereigniserkennersysteme“
12/06/2013 WP 2 Presentation 32

30. Public Events
 Open Company Day (8 10 6 2011) Open-Company-Day (8.-10.6.2011)
 eHealth – Chronisch Kranke zu Hause unterstützen
(15 6 2011)(15.6.2011)
 Fachtagung AAL in Niedersachsen (2011-11-11)
 Wirtschaft trifft Wissenschaft (2011-09-23)
 Tag der offenen Tür im Haus des Hörens (2011-09-03)
 Jubiläumsfeier – Zehn Jahre Kompetenzzentrum HörTech in Oldenburg
(2011-09-01)
 eHealth – Chronisch Kranke eHealth – Chronisch Kranke
zu Hause unterstützen
(2011 06 15)(2011-06-15)

31. Public Events
 Altenpflege Altenpflege
(27.-29.3.2012)
 AAL F 2012 i Ei dh AAL Forum 2012 in Eindhoven
(24.-27.09.2012)
 Press Breakfast Oldenburg
(2.10.2012)
 RehaCare Düsseldorf (10.-13.10.2012)
 WAGT-Veranstaltung Bremen
(23.10. 2012)
 Day of Open Door, BMFSFJ
(09 / 2013)