Citation: @article{Harper2012uq, Abstract = {Disabled people typically use methods of `sensory translation' to access a Web-page via assistive technology. These technologies conventionally render screen content under the direction of the user into a form that can be perceived by that user -- in effect the interface and content are adapted to suit their sensory requirements -- but simple sensory translation is not enough. Why is this -- and how can things be better? In this talk we touch on accessibility, sensory transcoding, multi-talker systems, auditory perception, and Neuroscience to help us in our search for equivalent interactive experiences tailored to the sensory modality of the user.}, Author = {Simon Harper}, Date-Added = {2013-02-15 10:31:27 +0000}, Date-Modified = {2013-02-15 10:39:41 +0000}, Howpublished = {Slideshare}, Journal = {Invited Talk - Technical Superior Insitute, LaSIGE, Lisbon, Portugal}, Month = {September}, Title = {Deep Accessibility: Adapting Interfaces to Suit Our Senses - http://goo.gl/VT5BE}, Url = {\url{http://www.slideshare.net/simon-harper/adapting-sensory-interfaces}}, Year = {2012}, doi={10.6084/m9.figshare.678330}, Bdsk-Url-2 = {http://dx.doi.org/10.6084/m9.figshare.678330}, Bdsk-Url-1 = {http://www.slideshare.net/simon-harper/adapting-sensory-interfaces}}

1. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Deep Accessibility
Adapting Interfaces to Suit Our
Senses
Simon Harper
University of Manchester
07 September, 2012
Deep Accessibility 1 / 31

2. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
The Story...
Sensory Translation
Transcoding
Content Driven Transcoding
Experience Driven Transcoding
Modality Driven Transcoding
Auditory Perception
Cocktail Party Problem
Properties of Sound
Multi-Talker Display
The Ear
Neurophysiology
Signal Transmission
The Auditory Cortex
http://goo.gl/VT5BE
Neuro Plasticity
Next?
Deep Accessibility 2 / 31

3. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Adapting Interfaces to Suit Our Senses
Disabled people typically use methods of ‘sensory translation’ to
access a Web-page via assistive technology. These technologies
conventionally render screen content under the direction of the
user into a form that can be perceived by that user – in effect the
interface and content are adapted to suit their sensory
requirements – but simple sensory translation is not enough.
Why is this – and how can things be better?
Deep Accessibility Sensory Translation 3 / 31

4. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Transcoding
“A category of technologies to transform inaccessible content
to accessible content on the fly”
To Accomplish Approaches
Text Magnification; Syntactic: such as removing images;
Colour Scheme Semantic: rearrangements and
Changes; fragmentation;
Serialisation; Annotation: created by a reader; and
Text Insertion; Generated: annotations by CMS.
Page Rearrangement;
and
Simplification.
Deep Accessibility Transcoding 4 / 31

5. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Content Driven Transcoding
Screen-Scraping - 1990’s
Originally content was ‘made accessible’ via a method called
screen-scraping which used the visual rendering to create
accessible content, by creating an off-screen model (a
representation of the GUI). Screen-scraping was problematic
because it was often wrong, did not take account of structure,
and could not form accurate semantics between elements of the
content.
DOM Analysis - 2000’s
Document Object Model analysis took over from screen-scraping
and enables accurate structural semantics to be created as all
elements and attribute values are available to the assitive
technology.
Deep Accessibility Transcoding 5 / 31

6. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
IBM Homepage Reader (HTML Annotation - 2000)
Before
After
Deep Accessibility Transcoding 6 / 31

7. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Structural-Semantics for Accessibility and Device
Independence (CSS Annotation - 2008)
SADIe Transcoding Method
Deep Accessibility Transcoding 7 / 31

8. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Structural-Semantics for Accessibility and Device
Independence (CSS Annotation)
. c n n C e i l n a v , d i v . cnnHeaderNav
{uom− s t r u c t u r a l − r o l e : L i n k L i s t ; }
input . cnnInput
{uom− s t r u c t u r a l − r o l e : S e a r c h E n g i n e ; }
d i v . CNNhomeBox , o l . c n n M o s t P o p u l a r , d i v#c n n T o p S t o r i e s . . .
{uom− s t r u c t u r a l − r o l e : Chunk ; }
d i v#c n n H e a d e r R i g h t C o l u l
{uom− s t r u c t u r a l − r o l e : PageSummary ; }
Deep Accessibility Transcoding 8 / 31

9. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
SaaS Transcoding (Code Analysis) - 2011
WIMWAT / SCWeb2 - Subtle Transcoding
Deep Accessibility Transcoding 9 / 31

10. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Readability - 2011
Before
After
Deep Accessibility Transcoding 10 / 31

11. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Evernote Clearly - 2012
Before
After
Deep Accessibility Transcoding 11 / 31

12. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Experience Driven Transcoding - 2010
‘Content Driven Transcoding’ is
focused on transforming the content
based on its representation in the
DOM; however ‘Experience Driven
Transcoding’ goes a step further
and attempts to transform the
content based on both its
representation, and the predicted
experience of the user into an
equivalent sensory experience .
BBC News with AoI’s
Deep Accessibility Transcoding 12 / 31

13. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Example – Order
Table: Areas of Interest (AoI) – BBC News (s = Seconds)
ID Area of Interest Vis. Order Mean Fix. Time Linear Order
1 Header 1 0.7 s 1
2 BBC Ticker Static 3 8.3 s (Lost) 11
3 BBC One Boss 2 1.9 s 4
4 LHS Menu Top 6 11.9 s 2
5 Other Top Stories 4 9.1 s 5
6 LHS Menu Bottom 8 14.7 s 3
7 Features Views Analysis1 5 11.3 s 6
8 Video and Audio 9 15.3 s 7
9 Have Your Say 7 13.1 s 9
10 Around the World 10 17.7 s 8
11 Features Views Analysis2 11 28.2 s 10
Deep Accessibility Transcoding 13 / 31

14. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Example – Graph
Ranked Order Comparison
Deep Accessibility Transcoding 14 / 31

15. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
SASWAT 2010
http://goo.gl/ty7OR
Deep Accessibility Transcoding 15 / 31

16. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
IBM aiBrowser - 2010
IBMs aiBrowser
Deep Accessibility Transcoding 16 / 31

17. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
VICRAM & EIVAA - 2011
Visual Complexity Rankings & Visual Aesthetics Rankings
VICRAM Heat Map of Visual Complexity
Deep Accessibility Transcoding 17 / 31

18. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Modality Driven Transcoding
‘What the HECK?’
‘Content Driven Transcoding’ is focused on transforming the
content based on its
representation in the DOM;
‘Experience Driven Transcoding’ attempts to transform the
content based on both its
representation, and the
predicted equivalent experience
of the user;
‘Modality Driven Transcoding’ adds another step, attempting
to transform the content based
on its representation, the
predicted equivalent UX, and
tailored to the sensory modality
of the user;
Deep Accessibility Transcoding 18 / 31

19. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Auditory Perception – ‘Cocktail Party Problem’
Been talking about translating parallel visual experience into a
serial auditory one. BUT auditory perception is parallel too.
‘The separation of two simultaneously spoken messages’, in
which Cherry first poses the question ‘how do we recognise
what one person is saying when others are speaking at the
same time (the ‘cocktail party problem’)?’ is key.
This can only be useful work in the domain of blindness, situation
impairment, and accessibility in that it may be possible to convey
limited Web page information much faster because of the ability
to comprehend highly parallel speech.
Cherry, E. (1953). Some Experiments on the Recognition of Speech, with One and with Two Ears The Journal of the Acoustical
Society of America, 25 (5) DOI: 10.1121/1.1907229
Deep Accessibility Auditory Perception 19 / 31

20. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Properties of Sound
Deep Accessibility Auditory Perception 20 / 31

21. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Multi-Talker Display – Brungart
We now have between seven and nine people intelligibly
multi-talking at the same time based on spatial location and
voicing. This can only be useful work in the domain of
blindness, situation impairment, and accessibility in that it
may be possible to convey limited Web page information
spatially, dynamically, and with a high degree of
comprehension at seven (or nine) times faster because of the
ability to comprehend highly parallel speech.
Brungart, D., & Simpson, B. (2005). Optimizing the spatial configuration of a seven-talker speech display ACM Transactions on
Applied Perception, 2 (4), 430-436 DOI: 10.1145/1101530.1101538
A. W. Bronkhorst. The cocktail party phenomenon: A review of research on speech intelligibility in multiple-talker conditions. Acta
Acustica united with Acustica, 86:117–128(12), January/February 2000.
Deep Accessibility Auditory Perception 21 / 31

22. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
M. F. Bear, B. W. Connors, and M. A. Paradiso.
The Ear Neuroscience: Exploring the Brain. Lippincott Williams & Wilkins, 2006.
Deep Accessibility Auditory Perception 22 / 31

23. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Frequency Detection
M. F. Bear, B. W. Connors, and M. A. Paradiso. Neuroscience: Exploring the Brain. Lippincott Williams & Wilkins, 2006.
Deep Accessibility Auditory Perception 23 / 31

24. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Neurophysiology - Signal Transmission
M. F. Bear, B. W. Connors, and M. A. Paradiso. Neuroscience: Exploring the Brain. Lippincott Williams & Wilkins, 2006.
Deep Accessibility Neurophysiology 24 / 31

25. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
The Auditory Cortex
M. F. Bear, B. W. Connors, and M. A. Paradiso. Neuroscience: Exploring the Brain. Lippincott Williams & Wilkins, 2006.
Deep Accessibility Neurophysiology 25 / 31

26. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Language - Positron Emission Tomography
M. F. Bear, B. W. Connors, and M. A. Paradiso. Neuroscience: Exploring the Brain. Lippincott Williams & Wilkins, 2006.
Deep Accessibility Neurophysiology 26 / 31

27. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
So What?
Why Does this Matter
Under used areas of the Brain develop differently;
Repurposed for different effect;
Occurs as Brain develops upto 18–21; called
‘Neuro Plasticity’.
Deep Accessibility Neurophysiology 27 / 31

28. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Neuro Plasticity
H. Burton. Visual cortex activity in early and late blind people. Journal of Neuroscience, 23(10):4005-4011, May 2003.
Deep Accessibility Neurophysiology 28 / 31

29. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Modality Driven Transcoding Revisited
More Work
1. Can frequency bounding to Auditory Cortex’s tonotopy
enhance parallelisation –and therefore distinguishability– of
both talkers and sonified interfaces?
Deep Accessibility Next? 29 / 31

30. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Modality Driven Transcoding Revisited
More Work
1. Can frequency bounding to Auditory Cortex’s tonotopy
enhance parallelisation –and therefore distinguishability– of
both talkers and sonified interfaces?
2. Can congenitally blind users, and blind users who have
became advantageously blind up to 18, perceive more talkers
–in a multi-talker system– than sighted users?
Deep Accessibility Next? 29 / 31

31. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Modality Driven Transcoding Revisited
More Work
1. Can frequency bounding to Auditory Cortex’s tonotopy
enhance parallelisation –and therefore distinguishability– of
both talkers and sonified interfaces?
2. Can congenitally blind users, and blind users who have
became advantageously blind up to 18, perceive more talkers
–in a multi-talker system– than sighted users?
3. Is any of this practically applicable in a Web engineering
setting; will the User Agent support these modal transcodes
and how?
Deep Accessibility Next? 29 / 31

32. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Deep Accessibility
Not Just
Not just about the technology;
Not just about Vague UX;
Not HCI as Psychology or Sociology
Deep Accessibility Next? 30 / 31

33. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Deep Accessibility
Is Also About
It’s about Accessibility as Science
About Accessibility as Physiology
Accessibility as Neuroscience
Deep Accessibility Next? 30 / 31

34. Sensory Translation Transcoding Auditory Perception Neurophysiology Next?
Any Questions?
Contact (vCard) Citations (BibTex)
http://goo.gl/yzJFx http://goo.gl/ny2pD
Deep Accessibility Next? 31 / 31