Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web
•Als PPT, PDF herunterladen•
0 gefällt mir•3,940 views
IUI talk on TrailBlazer, work by Jeffrey P. Bigham, Tessa Lau, and Jeffrey Nichols.
![Scale of Access
37 Million Blind People in the World
(125 million with low vision)
Increasing
Aging Population
Situational Impairments
Access technology benefits 25% of web users1
[1] “The Market for Accessible Technology. “ Forrester Research, 2003.](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web
Ähnlich wie Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web (20)
Mehr von Jeffrey Bigham
Mehr von Jeffrey Bigham (8)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web
- 1. Jeffrey P. Bigham University of Washington CSE jbigham@cs.washington.edu abling Blind Users to Blaze Trails Through the W Tessa Lau and Jeffrey Nichols IBM Almaden Research {tessalau, jwnichols}@us.ibm.com start
- 2. Scale of Access 37 Million Blind People in the World (125 million with low vision) Increasing Aging Population Situational Impairments Access technology benefits 25% of web users1 [1] “The Market for Accessible Technology. “ Forrester Research, 2003.
- 3. Web Potential for Blind People Web Advantages Convertible to Accessible Form Shop From Home Blind People Read 7x as Many Books The Law U.S. Government Requires Accessibility1,2 NFB vs. Target Corporation2 Making Access Possible is Not Enough Should be Intuitive and Fun [1] Section 508 of the U.S. Rehabilitation Act. [2] Americans with Disabilities Act.
- 7. X X X X <span class=“heading”> e b b START <span class=“heading”> <span class=“heading”> .heading { font-weight: bold; } Keyboard Shortcuts
- 8. Task: Placing Book on Hold 1. Enter “Book You Want” 2. Click “Find” Button 3. Find Correct Result 4. Click “Place Hold” Button 5. Click “Request” Button 6. Verify Queue Position
- 9. Task: Placing Book on Hold 1. Enter “Book You Want” 2. Click “Find” Button 3. Find Correct Result 4. Click “Place Hold” Button 5. Click “Request” Button 6. Verify Queue Position
- 10. Task: Placing Book on Hold 1. Enter “Book You Want” 2. Click “Find” Button 3. Find Correct Result 4. Click “Place Hold” Button 5. Click “Request” Button 6. Verify Queue Position
- 11. Task: Placing Book on Hold 1. Enter “Book You Want” 2. Click “Find” Button 3. Find Correct Result 4. Click “Place Hold” Button 5. Click “Request” Button 6. Verify Queue Position
- 12. How can blind people more effectively complete web tasks? Avoid Linear Searches Direct Users to Important Content Users Maintain Control Interface should be understandable Guide users not direct them Flexible and Adaptable Work in full web domain Compatibility with Tools Users Know Supplement to existing tools (screen readers)
- 13. CoScripter1 : Recording and Replaying Tasks Pseudo-Natural Language Commands [1] Little et al. “Koala: capture, share, automate, personalize business processes on the web.” CHI 2007. 1. go to “http://www.united.com” 2. enter your “flight number” (e.g., 0168) into the “Flight number” textbox 3. enter your “airport” (e.g., SJC) into the “From:” textbox 4. click the “Check” button CoScript Repository
- 14. Example TrailBlazer Placing a book on hold at the Seattle Public Library.
- 15. Set Focus & Provide Contextual Help Step description. Inline Playback Controls
- 17. Generalizing with Suggestions Choose from one of the following suggestions: Click the “Check” button. Enter into the “From:” textbox. Enter into the “To:” textbox. Select from the “Date:” listbox. Select from the “Time:” listbox.
- 18. Providing Suggestions User enters a short task description (query) “flight status on united” “place hold on book at Seattle library” (etc.) Machine learning model ranks all actions. Features Task Description Related Scripts from the Script Repository User’s History of Actions Apply existing scripts to new tasks or new sites No need to explicitly find script first Click the “profile” link. Click the “Home” link. Click the “Books” link. … Enter into the “Name” textbox. Enter into the “Zip” textbox. … Click the “Submit” button. Click the “Check Now” button. … Turn on the “Yes” Radio Button. …. Example Action List x1 x2 x6 … y
- 19. 0.08 0.27 0.10 0.76 0.20 Generalizing Scripts to New Sites “flight status on united” Task Description Repository 1. go to “www.aa.com” 2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox 3. click the fifth “GO” button Flight Status on American Airlines 1. go to “http://www.southwest.com/” 2. select “SJC” from the “Select the Departure City:” listbox 3. enter your “flight number” into the first “Enter the Flight Number” textbox 4. click the first “View Flight Status Information” button Flight Status on Southwest Airlines Click “my profile” link Click “Flight Information” link Enter into “Mileage Plus” textbox Enter into “flight number” textbox Click “Check” button Possible Actions similarity( , )
- 20. Click “my profile” link Enter into ‘password’ password box Click “Login” button Enter into “flight number” textbox Click “Check” button 0.08 0.88 0.10 0.36 0.20 Using History to Match Sub- Tasks Last Action Repository Update Facebook Status Reserve UW Meeting Room Possible Actions 1. go to “www.facebook.com” 2. enter your “facebook username” into the “username” textbox 3. enter your “facebook password” into the password box 4. Click “Login” button 5. … 7. … 8. enter your “CSE login” into the “username” textbox 9. enter your “CSE password” into the password box 10. … 0.00 0.87 0.46 0.14 0.04 0.00 0.75 0.42 0.13 “enter into ‘username’ textbox” similarity( , )
- 21. Evaluation Tested on 102 Actions 15 Most Popular CoScripts User-provided title used as task description 6.8 steps per script 42 possible actions Leave-One-Out Cross-Validation Removed from repository too Top 5 in 76% of cases CoScript Repository(5.1 words on avg.) Suggestions Provided
- 22. Ongoing and Future Work User Studies What happens when TrailBlazer is wrong? Incentives vs. Cost Better Model Continuous Learning Popularity, Past Behavior, etc. Improved Web Access Voice Over Mobile Phones Small Screens
- 23. Conclusion TrailBlazer makes web access more efficient Record, replay and share trails Targets non-visual use Uses existing repository of trails Suggestion Component Finds appropriate trails Generalizes existing scripts to new tasks and sites
- 24. The End Thanks to: Allen Cypher, Clemens Drews, Jimmy Lin, Yevgen Borodin, Charles Chen and T.V. Raman. Want to hire me? jbigham@cs.washington.edu www.jeffreybigham.com
- 25. Directing Users to Content Direct to Content Avoid inefficient linear searches 5 Participants/20 Regions Mechanical Turk 3 Commonalities Semantic Desciption (78%) Heading of Region (53%) Words Used in Region (37%) Clip the “search results” region
- 26. Directing Users to Content Direct to Content Avoid linear searches 5 Descriptions of 20 Regions Mechanical Turk 27 Users Clip the “search results” region Clip the region containing “batting average”
- 27. 0.08 0.27 0.10 0.76 0.20 Generalizing Scripts to New Sites “flight status on united” Task Description Repository 1. go to “www.aa.com” 2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox 3. click the fifth “GO” button Flight Status on American Airlines 1. go to “http://www.southwest.com/” 2. select “SJC” from the “Select the Departure City:” listbox 3. enter your “flight number” into the first “Enter the Flight Number” textbox 4. click the first “View Flight Status Information” button Flight Status on Southwest Airlines Click “my profile” link Click “Flight Information” link Enter into “Mileage Plus” textbox Enter into “flight number” textbox Click “Check” button Possible Actions Actions a Scripts s 1: enter 1: flight 1: number 1: textbox 2: button 1: city 2: click 1: departure 3: enter 4: flight 3: go 1: information 1: listbox 4: number 1: select 1: SJC 1: status 2: textbox 1: view 2: your Similarity = cos(a, s)
- 28. Click “my profile” link Enter into ‘password’ password box Click “Login” button Enter into “flight number” textbox Click “Check” button 0.08 0.88 0.10 0.76 0.20 Matching Partial Scripts Last Action Repository Update Facebook Status Reserve UW Meeting Room Possible Actions Action a Scripts s 1: enter 2: password 1: box box: 0.9 button: 0.1 click: 0.0 CSE: 1.2 enter: 2.5 facebook: 1.2 go: 0.0 login: 0.1 password: 1.8 textbox: 1.6 W(si) = d * W(si-1) + cos(t, si) 1. go to “www.facebook.com” 2. enter your “facebook username” into the “username” textbox 3. enter your “facebook password” into the password box 4. Click “Login” button 5. … 7. … 8. enter your “CSE login” into the “username” textbox 9. enter your “CSE password” into the password box 10. … 0.30 * 0.00 0.30 * 0.00 + 0.87 0.30 * 0.87 + 0.20 0.30 * 0.46 + 0.00 0.30 * 0.14 + 0.00 0.30 * 0.00 + 0.00 0.30 * 0.00 + 0.75 0.30 * 0.75 + 0.20 0.30 * 0.43 + … 0.00 0.87 0.46 0.14 0.04 0.00 0.75 0.42 0.13 d = 0.30 “enter into ‘username’ textbox”
Hinweis der Redaktion
- First, in terms of scale, there are a lot of blind people in the world. 37 million people who are blind, and 125 million with low vision. And the need for software supporting non-visual use is increasing, both as the population ages, and as more of us are wanting to access the web in contexts that introduce situational impairments – such as trying to access the web while driving or even just walking down the street. According to a study popularly cited by business folks and funded by Microsoft, nearly 25% of current computer users have some sort of visual difficulty that could be improved with the use of access technology.
- For this large (and growing) population, the web represents a great promise. Because content is stored electronically, it can be easily converted to an accessible form as opposed to printed material of the past. And for a variety of reasons blind people may prefer accessing online services from home instead of venturing outside of their homes. I recently gave a talk at the first annual Amazon accessibility day and they were particularly fond of citing a study showing that blind people read 7 times more books than sighted people. They wanted to support access by blind users so they’d buy more books. And so the potential of an accessible web is for businesses too. The law is also on the side of disabled users. The U.S. government requires technology it buys and information it produces to be accessible, and increasingly tech companies are seeing accessible products as a competitive advantage. The National Federation of the Blind recently won a multi-million dollar lawsuit against Target over the inaccessibility of the target.com web site, which has caused many companies to begin thinking more seriously about the accessibility of their web presence. Most work in this area has gone into making access possible at all, but using the existing tools remains difficult and frustrating. The majority of blind people will not learn to use the web with today’s tools. My goal is to make access usable, intuitive, and maybe even fun. TrailBlazer is an important first step in that direction.
- But, lets back up a little to see why the web can be difficult for blind people to use. To interact with their computers, blind people use software programs called screen readers. Here’s a screen reader reading the GMail inbox.
- Although it may have felt longer, that was only 20 seconds. In 20 seconds, most of us would have found the message we wanted, and many of us not only would have found that message, but also read it and, if you’re my advisor, replied with a carefully-considered, well-thought-out response. At this point, a blind user new to this site still has no idea that an inbox even exists.
- Screen readers do more than just read through pages from start to finish, they also provide keyboard shortcuts that make it easier to navigate through content. Consider this landscape view of the Gmail inbox. The commercial state-of-the-art for navigating through this landscape is to make various components of the page into landmarks accessible using keyboard shortcuts. With the appropriate keyboard shortcuts, users can navigate through such elements as heading tags, checkboxes, edit boxes, and buttons. All told there are on the order of 100 different shortcut keys to remember and learn, just for web browsing, and so in practice, real users utilize very few of them.
- Worse yet, these shortcuts are closely tied to how the web page was written, and so if we replace the heading tags on this page with the following code that visually appears identical, the heading shortcut will no longer work. It turns out that the fastest way to get to the Gmail inbox is to press the ‘x’ key to jump to the first checkbox, which preceeds each message. This is impossible for new users of the site to know and its arbitrary nature makes this mapping difficult for most users to understand. TrailBlazer helps add semantics to the actions available on web pages, and helps connect individual web page components into complete tasks.
- Lets take a closer look at the steps required to complete a task on the web. In this case, placing a book on hold at the Seattle public library. First, you load the library homepage. You enter the book you want. Press the find button.
- Find the book you want in the results that are presented. Then press the hold button.
- The site brings up a confirmation page, and you find and press the Request button.
- Finally, you verify that your request was successful and check your position in the hold queue. All told, you visited 4 different pages and interacted with 6 different elements. I’m sure this design could be improved for sighted users too, but imagine how long this could take a screen reader user, and how many arbitrary mappings they’d have to remember to quickly reach each interesting piece of content. For a user new to the site, each of those six interactions requires a linear scan through the page, a time-consuming, frustrating and potentially confusing process. Most alarming is that there’s nothing explicitly inaccessible about this site -- I believe we need to go beyond making it possible for blind people to use the web and start making it easy.
- This leads me to our overarching research question – how can blind people more effectively complete web tasks? In our discussions with blind web users, we came up with the following design goals –
- I looked around and found there’s already a system that on the surface seems to do a lot of what we want. CoScripter is a Firefox extension that records, replays and shares web tasks. Tasks are recorded using psuedo-natural language commands which are easy to understand and are stored in a common repository that contains a couple thousand scripts recorded by its users. TrailBlazer uses the same scripting language as CoScripter, but is a new system designed for non-visual use. By reusing the CoScripter language, blind users can immediately leverage the work that sighted users have already done.
- What we came up with was TrailBlazer. And so, without further ado, I present TrailBlazer. In this example, the user is placing a book on hold at the Seattle public library. Watch how the system provides the user the opportunity to understand what is happening at each point, and helps the user avoid linear searches entirely. At each step along the way, TrailBlazer directs the user to the component of the page necessary to complete the current step. At several points, the user will decide to explore the page, which is easy to do because their position in the linear view of the page accurately reflects where TrailBlazer is pointing them. ,,,,,,,,,,,,,,,,,,,,,,,,,,………………………… In this example, the user chose to read each step, but once they were used to the script and trusted it, TrailBlazer could have simply run the script without feedback, only pausing when it needed feedback.
- The controls for TrailBlazer are inserted into the pages that are viewed in a meaningful semantic ordering, and users are directed to the step description at each step. As each step is focused, the current cursor position is set at the step description. Proceeding forward from that location lets them immediately explore the target element, in this case a listbox, and also access the TrailBlazer controls.
- To address these concerns, we started thinking about how we could make the system useful even when no script is available, or when finding an appropriate script might take too long, as it might for someone completing a new task. The idea we came up with is to suggest a few of the many of actions available on a page, thereby dramatically shortening the linear space that needs to be searched. Here, the user is checking their flight status on United Airlines. They’ve input their flight number, and the system is offering several suggestions of what they might want to do next.
- To formulate its suggestions, TrailBlazer first asks users to input a short task description that gives it a starting point. These are unstructured, short, and can even be omitted. TrailBlazer then ranks all the possible actions on the page using a machine learning model – in our case we just used Naïve Bayes. The most interesting part of this are the features used by the model – These features, in practice, enable TrailBlazer to generalize knowledge contained within existing scripts to other tasks on other websites.
- To see how this works, lets look at a couple of these features and how they are calculated. The first is a feature that works to apply existing scripts to new sites. To walk through this scenario, first assume that the user has entered the task description “flight status on united.” TrailBlazer queries the repository of existing scripts with this string. For the purpose of this example, there is no script for finding flight status on United, but there is a script for finding the flight status on American Airlines and Southwest Airlines. These scripts don’t directly apply to the United Airlines site, but they are similar. TrailBlazer treats the text in the scripts returned as document. It also treats each action as a document. It then computes the similarity of these two documents, essentially using a word vector cosine as is common in information retrieval applications. Intuitively, you can see that values closer to one make the action seem more likely, and values close to zero make it less likely. This feature is calculated in a very simple way, which allows it to be robust to new commands and a variety of different types of web content.
- We can also match partial scripts, or sub-tasks contained within the tasks in the repository and turn those into features that can help TrailBlazer make better predictions. In this case, TrailBlazer takes the natural language representation of the last action that the user completed and uses it to query the repository. This returns a number of scripts. This time, each action in the scripts are assigned a weight according to how well it matches the user’s last action. This influence is then fed forward, weighting more highly actions appearing soon after the matched action. TrailBlazer forms a weighted document of the words in these scripts, and then computes the same cosine similarity metric as before to come up with the feature values. The effect is that the system is able to apply sub-tasks contained with the repository to new sites and situations.
- Scripts covered everything from searching on Google to completing arcane business processes at IBM.
- Obviously, the next thing we want to do is to try the suggestion version of TrailBlazer with users. Looking at not only what happens when it’s suggestions are right, but also what happens when it’s wrong, and how we make a more optimal cost-benefit trade-off appropriate to actually support users. The model we have has advantages in its simplicity – namely that it is fast to compute, robust to all sorts of different kinds of content and instructions, and different ways of expressing similar actions. But, we think we can do a lot more by more carefully targeting the model to this domain – for example, treating action types (clicking, entering information, etc.) as special. We’d also like to explore more continuous learning, so how much better can we do if we observe everything that users do instead of just what they choose to record. And finally, we’ve already started looking at other applications for the suggestion component of trailblazer. If we have an idea of what the user might want to do next, can we make a usable voice-based interface to the web over cheap mobile phones, or use the predictions to improve the interface to the web on small screens. Everyone seems to have this idea that the iPhone interface to the web is great, and it is better than most other interfaces, but it’s still annoying to zoom in and out content. If we know what areas of the page users are likely to want to see, we can provide visual suggestions too. This certainly isn’t a replacement for zooming, but could be much more usable than zooming when it works and possibly not cost too much if it’s wrong.
- Now I’m at the end of my talk, and I just want to throw out a quick plug – I’m on the market for jobs, so if you’re someone that has jobs, I’d especially like to speak with you. Thanks!
- The CoScripter language re-used by TrailBlazer previously couldn’t direct users to specific content in a web page. But this is often important. To construct a new “clip” command that would serve this purpose, we conducted a study with users.
- The CoScripter language re-used by TrailBlazer previously couldn’t direct users to specific content in a web page. But this is often important. To construct a new “clip” command that would serve this purpose, we used Mechanical Turk to gather descriptions of 20 regions from 5 different users. The most common
- To see how this works, lets look at a couple of these features and how they are calculated. The first is a feature that works to apply existing scripts to new sites. To walk through this scenario, assume that the user has entered the task description “flight status on united.” TrailBlazer queries the repository of existing scripts with this string. For the purpose of this example, there is no script for finding flight status on United, but there is a script for finding the flight status on American Airlines and Southwest Airlines. These scripts don’t directly apply to the United Airlines site, but they are similar. To convert this information into a feature, TrailBlazer first collects each possible action. It converts each to a word vector, and it also converts all the words in the retrieved scripts to a giant word vector, removing stopwords. The value of the feature is then computed just as the vector cosine between these two vectors. Intuitively, you can see that values closer to one make the action seem more likely, and values close to zero make it less likely.
- We can also match partial scripts, or sub-tasks contained within the tasks in the repository and turn those into features that can help TrailBlazer make better predictions. In this case, TrailBlazer takes the natural language representation of the last action that the user completed and uses it to query the repository. This returns a number of scripts. This time, each action in the scripts are assigned a weight according to the following formula: A fractional inclusion of the previous step’s weight plus vector cosine of the user’s last step and the step in the script. The effect is that influence is fed forward from the actions that matched the last action the user completed, weighting more highly actions that come later in these other scripts.