automated testing once WCAG 2 lands
Home / Legal Issues & Web Standards / automated testing once WCAG 2 lands
(originally meant as a reply to this post in the WaiZilla forum)
something i've been mulling over just the other night (yes, preparing for the @media2006 panel):
| Alex McKee wrote: |
True enough I suppose, although it could be made the first tool with WCAG2 testing support  |
i think automated testing for WCAG 2 is actually going to be a tricky one. the core document (which is the only normative one of the bunch) is technology agnostic, and doesn't specify exactly how success criteria can be passed. the techniques document is only informative, and by the look of it WAI intend it to be a live document that is constantly changing and evolving in line with best practices. they offer a variety of viable solutions for most checkpoints already, but it's not a cut and dry list a la WCAG 1.0. plus the whole baseline issue comes into play.
in short, i seriously doubt that we'll get any automated checkers that can confidently test for WCAG 2, despite the fact that WAI tried their best to make every success criterion testable. the way i see it, we'll have an increased need for human testers, assisted by tools like TAW3 to help them carry out manual tests more easily and record their findings.
_________________
Patrick H. Lauke / webmaster / University of Salford
co-lead: WaSP Accesibility Task Force
take it to the streets ... WaSP Street Team
personal: splintered | photographia | redux
co-author: Web Accessibility - Web Standards and Regulatory Compliance
Share: 
It's going to be tricky, and the end result is going to be in that fuzzy grey area, but given that WCAG 2 checkpoints have definative metrics the roll for tools should become in time a little more precise.
Given that there are metrics it would be nice to see any automated reporting returning multiple levels of response, i.e.:
It's wrong
It smells wrong
I've got a funny feeling about this (got to get a star wars quote in here somewhere)
Seems okay
I guess we'll only see over time.
_________________
Red Ant
Share:
| redux wrote: |
| in short, i seriously doubt that we'll get any automated checkers that can confidently test for WCAG 2 |
So, no change from WCAG 1 then?
_________________
Dan Champion, Champion IS, Mooch Marketing, Revish
Share:
Redux wrote:
| Quote: |
| i seriously doubt that we'll get any automated checkers that can confidently test for WCAG 2 |
from the conformance section of WCAG 2.0
| Quote: |
| All WCAG 2.0 success criteria are testable. While some can be tested by computer programs, others must be tested by qualified human testers. Sometimes, a combination of computer programs and qualified human testers may be used. When people who understand WCAG 2.0 test the same content using the same success criteria, the same results should be obtained with high inter-rater reliability. |
alex wrote:
| Quote: |
| although it could be made the first tool with WCAG2 testing support |
The ATRC Web Accessibility Checker tests against WCAG 2.0
_________________
Steve Faulkner
Technical Director
TPG Europe
The Paciello Group | Web Accessibility Tools Consortium
Share:
Just gave that tool a quick test in a couple of Calthorpe pages. It brought up "potential errors" such as:
- Alt text is not empty and image may be decorative.
- Image may contain text that is not in Alt text.
- h1 may be used for formatting.
- Anchor text may not identify the link destination.
- h2 may be used for formatting.
(EDIT) Told it to check this topic on Accessify and it failed us on:
- Data table element missing a summary attribute.
- select element missing an associated label.
- select element missing an associated label.
- select element missing an associated label.
Problems: 4 known, 1 likely, 160 potential.
The potential ones were much like the first list. There isn't a great deal it can fully verify, it just generates a list of "potential" problems which can sometimes occur from using particular elements. These are very numerous and so you'll still spend a lot of time manually testing.
At least it isn't overstating what it can test. That's the most important thing.
_________________
My CV type thing and my Life of Ben (Blog). Nigel Peck's Accessify Forum Requirements.
Share:
| danchamp wrote: |
| So, no change from WCAG 1 then? |
yeah, no purely automated checks against WCAG 1 can be confident either, but it just seems more so for WCAG 2, with the interplay of baselines and the informative nature of the techniques.
_________________
Patrick H. Lauke / webmaster / University of Salford
co-lead: WaSP Accesibility Task Force
take it to the streets ... WaSP Street Team
personal: splintered | photographia | redux
co-author: Web Accessibility - Web Standards and Regulatory Compliance
Share:
| stevefaulkner wrote: |
| The ATRC Web Accessibility Checker tests against WCAG 2.0 |
not bad, if your baseline is just HTML (which will probably be the case for most government/education sites in the foreseeable future, i'd expect). but the key will be to ensure that, as the techniques document will be constantly changing (though that will happen at the speed of WAI, so i don't expect nightlies to start appearing), these tools change as well. and that basically every single point is pretty much checked by a human (yes, as with WCAG 1), as particularly once you start bringing in ECMAscript and similar into the baseline mix it will be practically impossible to test stuff programmaticaly...
_________________
Patrick H. Lauke / webmaster / University of Salford
co-lead: WaSP Accesibility Task Force
take it to the streets ... WaSP Street Team
personal: splintered | photographia | redux
co-author: Web Accessibility - Web Standards and Regulatory Compliance
Share:
Yes I agree patrick, I don't think it is possible, and nor should it be a necessity that all success criteria are machine testable, otherwise I would be out of a job 
_________________
Steve Faulkner
Technical Director
TPG Europe
The Paciello Group | Web Accessibility Tools Consortium
Share:
As most WCAG 1.0 checkers will do a fair job of finding accessibility violations, they'll probably do a faily good job of detecting WCAG 2.0 violations. It's just a bit of a cross-referencing nightmare.
Shaw Trust Web Accreditations are written by "principle"; images, sound, multimedia, colour contrast, link quality, and so on. Before working on the format for the Shaw Trust Web Accreditation I used to write out reports by checkpoint. This made for very dull reading and a lot of unnecessary writing. I doubt that the actual manual test won't differ much at all as a result of WCAG 2.0 but the report will need to use a different set of references for each section. I imagine as a result of studying these cross-references that the need to check some additional info may come to light, but the principles of accessibility testing essentially remain the same I think, and due to the way the report is written a WCAG 1.0 tool will probably work just fine.
_________________
Grant Broome
Blog
CDSM
Shaw Trust
Last edited by Grant Broome on 30 Jun 2006 05:13 pm; edited 1 time in total
Share:
Right, well I've not been on for a while as you can tell. Sorry for bumping this!
First off, I didn't know that the ATRC tool tested against WCAG 2.0 checkpoints, that's good to know. Thanks.
Secondly, more importantly, I recognise that there can be no automated testing tool that actually produces failsafe results. That is obvious. However, they are useful just for catching more basic accessibility pitfalls and for debugging. For those who actually don't give any consideration to accessibility when designing - and there are a lot of them - an automated tool can be quite valuable simply to highlight the basic problems. As they research those problems, they will encounter information pertaining to more complex accessibility problems.
I personally regard all automated accessibility tools as educational, nothing more.
Share:
sorry to bump this, but I was just wondering whether there were any other automated WCAG 2.0 checkers, or if it's just still the ATRC for now..._________________
Jack Pickard The Pickards Information Services| Blog | Twit
Share:
It's interesting to see the list of potential problems, at least as interpreted by this vendor.
I found a couple of questionable examples. For instance, Accessibility Test 47 requires that "All h6 elements are not used for formatting", which is fine, but the Pass Example given (47-2: Header is not used to format text) is invalid as no headers are used at all. There is a similar problem with Accessibility Tests 43-46.
The interpretation of this recommendation also needs some attention: Accessibility Test 41. The page title and header state that "The header following an h5 is h6 or any header less than h3", which is incorrect, while the Expected Results section of the Test Process states: "The header following an h5 is h6 or any header less than h6", which it is impossible to fail. (The Fail Example shows an h6 following an H4, which is irrelevant.)
_________________
Nick
Share:
| coplanar wrote: |
| I found a couple of questionable examples. For instance, Accessibility Test 47 requires that "All h6 elements are not used for formatting", which is fine, but the Pass Example given (47-2: Header is not used to format text) is invalid as no headers are used at all. There is a similar problem with Accessibility Tests 43-46. |
Phil.
Share:
| pkiff wrote: | ||
Phil. |
I understand what you're saying about not using headings for emphasis, but that's the decision fail criterion: "This h6 element is used to format text (not really a section header)." The pass criterion -- "This h6 element is really a section header" -- cannot be illustrated by a test file that does not contain an h6! Instead, they should show an example of correct usage of an h6 header.
_________________
Nick
Share:
| Cerbera wrote: |
| Just gave that tool [http://checker.atrc.utoronto.ca/index.html, GdV] a quick test in a couple of Calthorpe pages. [...]
There isn't a great deal it can fully verify, it just generates a list of "potential" problems which can sometimes occur from using particular elements. These are very numerous and so you'll still spend a lot of time manually testing. |
Sometimes it warns you for a potential problem that it might as well check itself:
| Test result wrote: |
|
Accessibility Problem Problem Name: title text may be too long. Title Text: Het Jules Verne Genootschap HTML Code: <TITLE>Het Jules Verne Genootschap</TITLE> Requirement: title is short. Test Process View the text that is contained by the title element. The title element is expected to contain less than 150 characters (English) or the user must confirm that it is the shortest possible title. Expected Result title is short. |
Other than that, there are indeed tons of "potential problems" that should really be checked by the user. And while it's a good thing to draw the user's attention to these, it may be a bit OTT to pop up "Anchor text may not identify the link destination" for each and every link on the page, or to say:
| Test result wrote: |
|
HTML Code: <BODY/> Requirement: Table markup is used for all tabular information. Test Process Check the text content of the document. Check for any information that is presented in a tabular manner. Expected Result Table markup is used for all tabular information. |
| Test result wrote: |
|
HTML Code: <BODY/> Requirement: All changes in text direction are marked using the dir attribute. Test Process Check all text in the document. Determine if there are blocks of text that are not in the primary document language. Determine if the blocks of text have a reading order that is different from the primary language. Check if the blocks of text are marked using the dir attribute. Expected Result All changes in text direction are marked using the dir attribute. |
Anyway, it's a good start.
Share:
Page 1 of 1
All times are GMT
You cannot post new topics in this forumYou cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum