3. Agenda
• What is an accessibility issue?
• Why prioritize?
• Understanding risk
• Challenges
• Remediation Approaches
– Considerations
– Simple Prioritization
– Advanced Prioritization
4. Things to keep in mind
• I am mathematically challenged
• This topic is exploratory, not declarative
– Please participate, ask questions, offer new ideas
6. What is an Accessibility Issue?
• Bug: Term used to describe
an error, flaw, mistake,
failure, or fault in a computer
program or system that
produces an incorrect or
unexpected result, or causes
it to behave in unintended
ways.
8. Why Prioritize?
• Apply resources most effectively
• Minimize accessibility’s impact on business
• Motivate development staff
• Maximize positive impact for users
• Reduction of Risk
10. Understanding Risk
• Risk is the potential that a
chosen action or activity
(including the choice of
inaction) will lead to a loss
(an undesirable outcome).
11. Understanding Risk
• Ultimately, remediation of bugs is an effort at
risk mitigation
• Risks of
– Poor quality (Users having problem with system)
– Lost income
– Ancillary losses
– Administrative Complaint (public sector)
– Litigation
16. Challenges
• Not all accessibility problems are equal
– Time
– Impact
• Impact on Users
• Impact on Business
• WCAG Level & SC is inappropriate for
determining priority
17. Challenges
• Time
– Often at a premium
– Time spent on after-the-
fact bug repairs is time
that is taken away from
meeting other business
needs
– See, “Technical Debt”,
Martin Fowler
20. Simple Prioritization
• Simple Prioritization
– Focused solely on time
and (simple) impact
– How long will it take?
– How bad is the problem?
21. Simple Prioritization
• Pros
– Focused on the user
– Super simple
– Often, “hunch” from
expert is as good as
something more formal
• Cons
– Does not take into
consideration impact on
business or system
22. Advanced Prioritization
• User Impact
• Ease & Speed
• Impact on Interface
• Volume
• Location
• Secondary Benefit
• Each item ranked: None (0), Low (1), Medium (2),
High (3)
23. Advanced Prioritization
• Impact on Users with Disabilities
– Broken down by type of user &
impact on each
• IB - Blind
• IV – Visually Impaired (non-blind)
• IH – Deaf & HoH
• IM – Motor
• IC - Cognitive
• IS – Speech
• Impact* = (IB + IV + IH
+ IM + IC + IS)
26. Advanced Prioritization
• Volume of Repeat
Issues
– How many times does
the exact same issue
occur?
– How many times do
(very) similar issues
occur?
27.
28. Advanced Prioritization
• Secondary Benefit
– Older Users
– Low Literacy Users
– Low Bandwidth Users
– Reduced Dev/ Maintenance time
– Alternate Devices
– SEO
– Usability
• Tie to org goals
29. Advanced Prioritization
(Impact + Repair Speed + Location + UI Impact +
Secondary Benefits) * Volume = Priority
• Sort all issues according to priority
• Fix em!
30. Next Steps?
• Come up with final formula. Current version
has weaknesses.
– Provide weighting to factors
– 1st Round Delphi results:
• User Impact: 3.95
• Impact on Interface: 1.5
• Location of Error: 1.5
• Secondary Benefit: 1
• Ease & Speed of Repair: 2.25
33. Managing Remediation
• Managing Remediation is a process not unlike
dilution in chemistry
– Dilution: The process of reducing the
concentration of a solute in solution
• In our case: reducing concentration (defect
density) in a system
Concentration = numBugs/linesOfCode
38. Conclusion
• In the quest for perfection, prioritization helps
us get closer quicker
• We must maximize efficiency to have high
positive impact
• Multiple factors exist that can be used to
determine priority
• Iterate remediation efforts to progressively
dilute them
• We can measure success
We need to move beyond the impression (that we’ve created, IMO) that accessibility issues are something separate.Accessibility issues aren’t “nice to haves”, or enhancements. They are bugs.We have created a flaw in the system which harms its ability to be used by real people.Photo Credithttp://www.flickr.com/photos/threeheadedmonkey/with/3856171871/
During my career, I’ve performed scores of audits and expert reviews to look at the usability and accessibility of web-based systems (websites, web-based applications, intranets, etc.). Of those, I can count on one hand the number of systems tested that were in a pre-production phase prior to the accessibility work. In nearly all cases, the testing was performed on systems that were already released to users. In such a scenario, an organization’s level of risk is at its highest, as people are using the system. Therefore the organization and its developers require sufficiently informative data output from the accessibility testing which is detailed, clear, and actionable. It doesn’t do the client any good to throw them a report filled with a bunch of accessibility violations unless that report also includes information to help them fix their problems. They need to know what their problems are, where their problems are, and how to fix them. There’s also an additional item they need to know: When to fix them.Prioritization of remediation helps to apply resources most effectively. Post-deployment bug fixes are a needless expense. We should apply resources in a manner that reaches deepest toward the greater goodPrioritizing remediation helps minimize accessibility’s impact on business. Generally, accessibility is too often regarded as expensive and hard. Properly prioritized, this expense and difficulty can be significantly diminished. This diminished impact can motivate staff and help get their buy-in for both this effort and future effortsThe end goal, for us as accessibility advocates is the real reason, IMO, we should prioritize: It allows us to maximize the positive impact for users. Fixing the most important stuff first will mean that we can more quickly make things better for the end userFinally, prioritization also helps reduce our risk.
Photo Credit http://www.flickr.com/photos/kyz/
Although people generally think of legal risk as the only risk when it comes to accessibility, there are many types of risks that are created by bugs in a system.Poor quality: Operability or performance problems for the end userLost income: The volume or severity of bugs is such that users who want to buy our products or consume our content do not or cannot do so and we lose moneyAncillary losses:Losses caused by spending money you don’t need to: Primarily, a11y fixes after the fact. Oops, we decided a11y is important after the fact, now we must spend money we didn’t intend to spend and that negatively impacts the money we have available to do other thingsAdministrative Complaint: Compliance is a requirement in some industries and noncompliance raises the chance that we can have the heavy hand of regulators beating down on usLitigation: Certain industries see an increased amount of litigation surrounding accessibility, which is a significant risk.Many of these risks are related and have a multiplying effect upon our business case.Intelligently remediating our bugs will work to rapidly reduce risk
What is your probability of an actualized risk? In its simplest form: Probability = (number of negative events) / (population). Your probability of anything happening is based on the size of your population and the number of negative events within that population. In other words, if 10 people out of every 100 get a speeding ticket every 5 years, then your probability of a speeding ticket every 5 years is 10% (10/100 = .10).In terms of accessibility lawsuits, it is important to understand how to determine your population. Your population isn’t “all the other websites on the Web”, but rather it is your peer companies. How you define your peer companies involves a number of factors such as industry, site features, audience demographics, reach, and engagement in risky behavior. Note that I use the word “peer” rather than “competition”. Your competition could be a brick & mortar entity, or it could be a company that is much smaller. While they may compete in the marketplace, they are not a “peer”. Again, to use a simple analogy: Lung Cancer Probability = number of cancer cases/ number of people engaging in cancer-causing behaviors. Although everyone has some level of probability for lung cancer, people who are engaging in cancer-causing behaviors are in a very different peer group from those who don’t. Their engagement in that behavior increases their probability of the negative event.Photo credit http://www.flickr.com/photos/bourguiboeuf/6888114273/sizes/z/in/photostream/
From the probability, we can determine how much we are risking. Our risk amount is our probability multiplied by our expected loss: Risk Amount = (probability of a negative event) * (expected loss in case of negative event). For example, if we have a 10% probability, our risk amount is 10% of the total expected loss (.10*totalExpectedLoss). To use an extreme case, we’ll use the Target lawsuit. Target settled with the NFB for $6,000,000 plus $4,000,000 in attorneys fees, etc. Based on their population (very large e-commerce sites) they had an 8% probability: .08 * $10,000,000 = $800,000. If you’re a company that is a peer of Target, you are risking $800,000. Some people may argue that you’re actually risking $10,000,000 because that’s what Target settled for, but that doesn’t take into consideration your probability of the lawsuit.http://www.flickr.com/photos/rmgimages/4881843809/sizes/m/in/photostream/
Your Return on Investment (ROI) for accessibility work (assuming 8% probability) is $800,000 minus the money you spend on fixing or remediating accessibility related bugs.For those who are intending to try to sell accessibility internally, it may make sense to take this into consideration when trying to gather budget for the efforts.
The Web Content Accessibility Guidelines structures its Success Criteria in terms of “Level”. Success Criterion can be Level A, Level AA, or Level AAA. Unfortunately, their explanation of the levels isn’t terribly clear. It was much more clear in WCAG 1.0, however the WCAG 1.0 Priority levels did not take into consideration any factors other than user impact. It is clear that in the early days of work toward WCAG 2.0, they understood that conformance level needs to take more things into consideration, as evidenced by this early draft. Unfortunately, the current WCAG documentation on Understanding Conformance currently lacks a description of any kind whatsoever on the topic of what the conformance levels mean and what differentiates each from the rest.Consequently, people tend to believe WCAG 2.0’s Level is synonymous with WCAG 1.0’s PriorityWhat is clear, despite the lack of guidance in WCAG, is that they acknowledge through the use of different levels that not all accessibility issues are created equal. Some items are more impacting on users than others. Some items are technically difficult or impossible to achieve. Some items don’t have good support among user agents or assistive technologies. As a consequence of all these factors and more, it would be rather reckless to treat everything as if they were all equal. In some cases, doing so could risk budgets while not helping the end user. Instead, remediation should be targeted to provide the best benefit quickly and cheaply. Turning back to impact, for a moment, I decided to take my own list of Best Practices and sort them by impact, giving an impact of '1' for each user population affected (Low Vision, Blind, Hearing Impaired, Mobility Impaired, and Cognitive). In other words violating a Best Practice that impacts all of them got a '5' for impact. Violating a Best Practice that impacts only Blind people got a '1'. Here's what I found: 5 - 2A ; 1AAA (66% A; 33%AAA)4 - 30 A; 10 AA; 15 AAA (55% A; 18% AA; 27% AAA)3 - 35 A; 5 AA; 12 AAA (67% A; 9% AA; 23% AAA)2 - 31 A; 3 AA; 12 AAA (67% A; 7% AA; 26% AAA1 - 51 A; 11 AA; 13 AAA (68% A; 15% AA; 17% AAA) What I found was that, at least in my subjective interpretation, reliance on WCAG level alone meant that we could be including best practices that have rather low user impact. Also, we could be giving high priority to things that are difficult or time-consuming to remediate The problem is, obviously, not all best practices are created equally. "Provide a help link on each page" (mapped to 3.3.5) is definitely not the same as "Provide ability to pause, stop, or hide content which updates automatically" (2.2.2) and "Provide the ability for users to turn on sound only at their request" (1.4.2). The latter case illustrates the need to rank the impact of individual populations. The impact of auto-starting audio for Blind people is very high and the impact on Mobility Impaired and Low-Vision persons is much higher than hearing impaired or cognitive. I'd say 3,2,2,1,1 on that one.
Doing bug remediation takes time. Ideally this would be time we don’t need to spend because we’d get everything right in the first place. That is often not the case.Creating bugs causes us to incur Technical Debt - a metaphor referring to the eventual consequences of poor design. This debt must eventually be repaid in some form – the remediation of bugs, for instance. The cost of that debt in this instance is the time that the repair of the bug requires.“We’ll do it later” obligates you to pay the extra time cost of actually doing it later when you should have had a commitment to quality in the first place. (Hard to stop a moving train, repairing of bugs is chasing the train)Very rarely are development staff just sitting around, looking for something to do. So the repair of bugs takes time away from doing other things. Often times, those other things are things which have a direct line on ROI: Important new features that need to ship now.Repair of bugs is a time expense and time is money, so we need to ensure we prioritize on the important things so that effort is not used frivolously.Photo Credit: http://www.flickr.com/photos/mao_lini/3053649344/sizes/l/in/photostream/
So, given what’s been said already: Bug repair will also cause an impact on the budgets, resources and even to the systemDo these bugs cause us to require outside help from a consultant? Do we need to outsource this repair to a third party? Or, conversely, do we need to take that work away from third party and use a specific internal resource who is an expert at accessibility?Also, how will fixing this bug impact the actual system? Will it impact the presentation logic? Will it impact the business logic? Are there other features in the system or in upcoming builds that are going to be impacted by fixing this item?These are all important things that can cause challenges in determining priority
In the simple prioritization scheme, you focus on only two factors: Time and ImpactFor time: How long will it take to fix this issue?For impact: How bad is the problem for users?
With advanced prioritization, there are six items to take into consideration:As before: User ImpactEase & Speed of RepairPLUS:Impact on the system: Will this change the interface? Will it cause modifications to business logicVolume: How often does this issue occur?Location: Where is this problem? Secondary Benefit: Can we get any additional business boost from fixing this?In all 6 cases we’ll be ranking these factors with 4 scores:0 – None, in other words, either it is not a consideration or there is no weight given by this consideration1 – Low, in other words the weight for this factor is low2 – Medium, in other words there’s a moderate weight for this factor3 – High, in other words this has a lot of weight in this case
The more people are impacted – and the greater that impact – the more important it is that we give this priority.In this case, impact on user is ranked Low, Medium, High where “High Impact” is how bad it would be if the error was left unresolved.High Impact. Users will be unable to perform important system tasks or unable to understand important content if this item is left not repaired.Medium Impact. Users will be able to perform important system task with some level of difficulty or will be able to understand important content with difficulty if this item is left not repaired.Low Impact. Users will be inconvenienced by leaving this item not repaired, but will be able to accomplish all tasks.Something that offers no impact should be taken from consideration.Note the asterisk(*) on impact. As we’ll discuss later, there may be better alternativesSome prioritization schemes offered by others also take this into consideration, but they consider user impact for all users, without distinction. I would argue that we should segment the different types of disabilities. This is because issues which impact multiple disability types will need to be given higher priority. Also, some items will impact different user populations in a different way. For instance, focus control will impact nearly all users with disabilities, but will be an absolute barrier for some users but mostly a frustration for others.Photo credit http://www.flickr.com/photos/revnaomi/6765951145/sizes/z/in/photostream/
As before, if the goal of our remediation effort is to fix the system, it only makes sense to utilize our developers’ time wisely. Part of doing so is knocking out the easier things before the hard things. Easy things tend to take less time, which means we can get further down the road toward an accessible system by prioritizing the easy things over the hard. Doing so will give us the ability to get quick improvements with minimal effort.Ease & Speed of Repair relates to what it will take to fix the issue. Ease & Speed are related but may be exclusive of each other. It may be very easy to do but could take a long time to get it done. Consider both of these when calculating them.One thing to keep in mind: In this case, the Faster & Easier something is, the higher score (and therefore priority). “Super easy” = 3, “Super Difficult” = 1, etc.Photo Credit: http://www.flickr.com/photos/hopefulnebula/3631357698/sizes/z/in/photostream/
This relates to both the appearance and the functionality of the system. There may be times when an accessibility-related repair may be necessary which would have a negative impact on the user interface and/ or operation of the interface as it is currently designed. One area where I see this happen very often is in color contrast of primary site templates. In these cases poor color contrast is used which is related to the organization’s branding as a whole. Improving the color contrast would essentially require a redesign of the site. In this scenario, things which will require significant redesign will need to be given much lower precedence over things which will have little-to-no impact on the interface.Something that has zero impact on the interface = 3Something that has high impact on the interface = 1Photo Credit: http://www.flickr.com/photos/calrion/61037070/sizes/z/in/photostream/
The more often the exact violation occurs, the more impactful it is.What I often find when doing accessibility testing is that the same mistakes will be found over and over. Some of those mistakes will be repeated because they’re part of a template. Some of the mistakes will be repeated because common code is used throughout the site. For instance, forms may be created using a server-side library or class which is used to make all forms. In either case what happens is that the volume of similar issues will be quite high while the actual volume of offending code will be small. In this case, a large benefit can be realized quickly by fixing the templates or classes which cause the most common issues first.It is important to understand that in this case, we must refer to the exact same violation (in other words, the exact code) is found.Similar violations may exist (such as missing alt text), but that doesn’t mean the code which causes the issue is the same. Some prioritization schemes do not make a distinction regarding whether the violation is exactly the same vs. merely the same type of problem. There’s value in tracking similar violations as “patterns”, but volume of identical issues tends to be a multiplier in terms of priority.Photo Credit: http://www.flickr.com/photos/gratapictures/6023846323/
Accessibility issues that don’t impact anyone aren’t issues at all. One area where this is especially true is when it comes to the location of the issues. In any site, there will be those pages which get far more traffic than others. People commonly understand this concept as the 80/20 rule. In reality, it might be that 15% of your pages get 90% of your traffic. Whatever the case, you will want to remediate issues which occur on the pages with the highest traffic before those which don’t get much traffic at all.Also a factor is the criticality of the location of violation. If the violation exists in a primary template or in the content area of a critical workflow, its priority is higher.One example would be a multi-step interaction where screen 1 of the interaction is free of problems, screen 3 of the interaction is free of problems, but the middle screen has a high-impact flaw which causes a roadblock to success for the user. Because this causes the entire interaction to fail, the issues on screen 2 need higher priority
Certain accessibility best practices also have a tie-in with other benefits that could be realized if violations are repaired. For instance, depending upon the nature of the issue, you may find that you’ll be able to improve on-page factors for SEO or improve display for alternate devices like tablets.These should be tied to the organization’s goals. For instance, if you’re already using standards based production, then the reduced development and maintenance time isn’t going to mean much.Photo Credit: http://www.flickr.com/photos/stansich/183456721/sizes/z/in/photostream/
Very simply, each priority factor for the issue can be added together, multiplied by the volume of occurance for this issue and you have your priority.Then sort the issues by the priority and get to work fixing them!
Providing a simple formula like the one proposed, which is: (Impact + Repair Speed + Location + Impact on Interface + Secondary Benefits) * Volume = PriorityWhere each item is ranked 1-3 means that some things may be artificially boosted in terms of priority. For instance, all other items being equal, a violation that has a “Location of Error” score of “3” would weigh more than something that has a User Impact score of “1” and this would skew the results in a direction that’s probably unintended.Clearly, as has been said many times already, some things are more important than others. Therefore, in order to properly weigh priority, we must give each item a weight.This spring, I held the first round of a Delphi exercise in which I asked several peers to weigh in on which items were most important and to also rank each item. The results were as follows:User Impact: 3.95Impact on Interface: 1.5Location of Error: 1.5Secondary Benefit: 1Ease & Speed of Repair: 2.25Ostensibly, this means the aforementioned formula could be:((Impact * 3.95) + (Repair Speed * 2.25) + (Location * 1.5) + (Impact on Interface * 1.5) + (Secondary Benefits * 1)) * Volume = PriorityThis formula seems to make sense though it has yet to be fully tested in the field.
In chemistry, dilution is the process of reducing the concentration of a solute in solution, usually simply by mixing with more solvent. In our case, the solute are bugs. The solvent is the code itself.We want to add more “solvent”, which is clean code.
Continuing our dilution analogy, we see three beakers indicating different states of concentration.Given the three examples, we see our start point, mid-point, and end point represented by beakers of solution.In our starting point we see a beaker that has a high concentration of problems – especially high priority problemsIn our mid-point, we see that the vast majority of high priority problems have been eliminated, leaving us to address our medium priority problemsFinally, our end point, we see almost all of our problems are low priority. We can choose to fix them if we’d like – and I believe we should.
And here is our unicorn – a perfectly accessible system
We can measure the improvement of our system by tracking it over time.Very simply, we can measure our starting concentration minus our ending concentration (in other words, our improvement) divided by time.There is an inherent flaw with such a simple approach in that it is most accurate when the time interval is a short one. A two point representation for a first derivative such as above is deemed 1st order accurate since this approximation has an error that scales with the time interval to the 1st power.Another key point to keep in mind is that the # of lines of code WILL change when bugs are fixed, so that you cannot just continually count bugs each time, you must also count lines each time as well.
