SiteMorse Q & A
Home / Site Building & Testing / SiteMorse Q & A
| Grant Broome wrote: |
| You publish league tables based on how one site performs next to another. How many checkpoints you can comprehensively measure is a critical aspect of the accuracy of these tables. That is why my question is relevant. |
Well, there you have another mistaken assumption. Priority 1 checkpoints only contribute 12% of the overall score, so they could not possibly be considered "critical". That's not to say they're not important, but "critical" is clearly inaccurate. You're also making another mistake if you think that how many checkpoints are measured has any relation to the "accuracy" of the tables - how are you defining "accuracy"?
| Grant Broome wrote: |
| I will note that in your answer to Q.2, you have not provided an exact number |
Fair enough, so long as you also note that you did not provide an exact question.
| Grant Broome wrote: |
| The next question is similar to Question 2. It is relevant to web managers that I have met and spoken to who believe that some web sites with a higher SiteMorse accessibility ranking than their own may in fact be far less accessible |
If others have a higher accessibility rating, then their site has flaws they should be fixing. Why are they worrying about the motes in others' eyes instead of the beams in their own?
| Grant Broome wrote: |
| Q.3 Is it theoretically possible that a web page that has 0% WCAG priority 1 checkpoint compliance could still be awarded 100% SiteMorse accessibility compliance? |
Again, your question is vague. What does "0% compliance" mean? The answer could be "yes" or "no" depending on your definitions - but then again, if you're going down that route, you could probably hypothetically create a web page which broke all of the WAI guidelines and yet was accessible nevertheless. Does that mean we should scrap the WAI guidelines as being useless?
If you want a short answer to your question, I shall make reasonable assumptions about your definitions and the answer is "no".
Share: 
Jon, I hope you are not saying that my questions are vague simply because you don't want to give a straight answer.
I'll just quickly address some of your comments before we move onto the next question.
Accessibility contributes 12% of the overall score. You say that you do not feel that this is a critical figure. You do not feel that a difference of 12% is critical. Many do. Regardless. In my statement I said:
| Quote: |
|
How many checkpoints you can comprehensively measure is a critical aspect of the accuracy of these tables. |
I don't believe that I have made any assumptions. We are only discussing accessibility which is 100% of 12% of your report. If it is not critical or accurate then one can be forgiven for questioning its value.
| Jon R wrote: |
| how are you defining "accuracy"? |
Accuracy as in "An accurate representation of how accessible a website is when measured against individual WCAG checkpoints." I don't see how that could have been taken in any other context.
| Jon R wrote: | ||
Fair enough, so long as you also note that you did not provide an exact question. |
The question:
| Quote: |
| How many Priority 1 checkpoints can be comprehensively measured by SiteMorse alone and require no manual verification? |
Again, I don't see how this can be considered vague. The question is quite straightforward and I think you have reluctantly (and vaguely) conceeded that the answer is zero. I was trying to be gracious by not re-iterating it, but just for clarification:
| Jon R wrote: |
| Of course, I suspect the answer you are fishing for is "none", if by "comprehensive" you mean that all possible failures (whatever that means, given the ambiguous nature of many of the guidelines) will be positively identified and none will be missed. |
| Jon R wrote: |
|
That question was not genuine - Grant already knew the answer, and was trying to make a point. That's fair enough, which is why I addressed the point he was making. |
Let's have no more accusations of vagueness. You had a very clear idea of the meaning of my question and you gave your response.
| Jon R wrote: |
| Again, your question is vague. What does "0% compliance" mean? |
"0% WCAG priority 1 compliance" means that the page fails to fully satisfy any WCAG priority 1 checkpoints. There's no hidden meaning. I'm just attempting to establish the facts. This isn't new language to either of us:
| SiteMorse Disclaimer wrote: |
| The range of tests [Web Accessibility Initiative WAI] that can be completed automatically are limited, 100% compliance with the automated tests does not mean 100% compliance with the requirements. |
I believe that your answer to this question is "no". We may come back to this later.
Well before you agreed to this Q&A, Gez Lemon of Juicy Studios - Accessibility guru and respected member of this forum has created a page to test the accuracy of automated testing tools. The page acts like a Turing test for automated checkers.
Bruce Lawson - another well respected, experienced and knowledgeable Guru and frequenter of these forums has created another page that by coincidence would act as an ideal test for automated testing tools.
Both sites are littered with accessibilty errors. Gez states that his page fails 11 of the 17 priority 1 guidelines.
Q.4 As a measure of the accuracy of the SiteMorse accessibility report, what accessibility rating does SiteMorse give:
a. Juicy studio Inaccessible page"
b. Bruce Lawson "Geocities 1996"
_________________
Grant Broome
Blog
CDSM
Shaw Trust
Share:
| Grant Broome wrote: |
| Jon, I hope you are not saying that my questions are vague simply because you don't want to give a straight answer. |
No, I'm saying it because they are vague.
| Grant Broome wrote: | ||
Accuracy as in "An accurate representation of how accessible a website is when measured against individual WCAG checkpoints." I don't see how that could have been taken in any other context. |
OK, but is that how the league tables are defined? If not, your definition of "accuracy" is meaningless. It seems to me that you are trying very hard to show that they are not something that they never claimed to be in the first place.
| Grant Broome wrote: | ||
The question:
Again, I don't see how this can be considered vague. The question is quite straightforward |
You're playing with definitions again. "Is this a good website?" is another example of a straightforward question, but an extremely vague one, that could never have a precise answer.
| Grant Broome wrote: |
| Q.4 As a measure of the accuracy of the SiteMorse accessibility report, what accessibility rating does SiteMorse give: |
Since that's meaningless as a measure of the accuracy of anything, I'll pass, thanks. (I'm not even sure it's a meaningful concept to consider the "accessibility" of a page whose only function is to be inaccessible).
That's the third time you've asked essentially the same question, which had already been answered before you asked the first time: no, SiteMorse (or indeed any other automated tool, current or future) does not and could never check all the WAI guidelines.
I'd appreciate it if you would stop circling around the point you want to make, and actually make it.
Share:
Thanks for your honesty Jon, it is appreciated.
Of course, the point I am making and that you have verified is that in fact none of the Priority 1 checkpoints can be verified by automated testing alone.
This is in stark contrast to SiteMorse marketing material the report itself and calls into question the relevance of automated accessibility testing in any league table.
Let's continue...
Headings are an essential navigation aid for many disabled users. Without headings pages can become very painful to navigate.
I have spoken to public sector web masters who have acheived a higher SiteMorse ranking by removing all headings from their website. This is because SiteMorse has reported errors in heirarchies of headings without offering best practice advice and do not see a site without headings as having any errors at all. This is one of many examples of how a developer under pressure can cut corners to gain higher rankings.
Q.5 Does SiteMorse support (as the results suggest) the notion that a site without headings is more accessible than one with a single heading heirarchy error?
_________________
Grant Broome
Blog
CDSM
Shaw Trust
Share:
| Grant Broome wrote: |
| Of course, the point I am making and that you have verified is that in fact none of the Priority 1 checkpoints can be verified by automated testing alone. |
If you insert the word "fully" before "verified by automated testing" in that sentence, then I agree.
| Grant Broome wrote: |
| This is in stark contrast to SiteMorse marketing material the report itself and calls into question the relevance of automated accessibility testing in any league table. |
On all of those points, however, I of course disagree completely.
| Grant Broome wrote: |
| I have spoken to public sector web masters who have acheived a higher SiteMorse ranking by removing all headings from their website. |
That of course begs the question of why they did that, rather than fix the actual problem.
| Grant Broome wrote: |
| This is because SiteMorse has reported errors in heirarchies of headings without offering best practice advice |
Now that is flat-out wrong. The 'additional help' for the 'useheadings' diagnostic includes useful quotes from the WCAG such as "Sections should be introduced with the HTML heading elements (H1-H6)" and "it is important to use [headings] appropriately to convey document structure", as well as a link to the appropriate WCAG section.
| Grant Broome wrote: |
| Q.5 Does SiteMorse support (as the results suggest) the notion that a site without headings is more accessible than one with a single heading heirarchy error? |
Without a specific example to comment on I cannot be absolute, but of course in general the answer to your question is "no".
That brings us neatly to a related and possibly constructive point, actually. I would love to be able to also generate a diagnostic if headings are not used. However, simply flagging all pages without an <h1> (as anything other than a 'warning') would almost certainly be too crude. What about flagging all pages with more than a certain amount of textual content? What do you think? Are there real-world use-cases where large amounts of text without any headings at all is correct?
Share:
Thanks Jon,
There are many points for discussion following your answers, but in the interest of covering the most relevant topics, I'll just mention a couple of things...
| Jon R wrote: |
| Now that is flat-out wrong. The 'additional help' for the 'useheadings' diagnostic includes useful quotes from the WCAG such as "Sections should be introduced with the HTML heading elements (H1-H6)" and "it is important to use [headings] appropriately to convey document structure", as well as a link to the appropriate WCAG section. |
The fact remains that your tool does not find any issue with pages with no headings and highlights one (of many) examples of how the league tables cannot accurately measure how accessible a website is compared to the next. The results of automated testing alone are virtually meaningless.
| Jon R wrote: |
| Since that's meaningless as a measure of the accuracy of anything, I'll pass, thanks. (I'm not even sure it's a meaningful concept to consider the "accessibility" of a page whose only function is to be inaccessible). |
I also have a relevant update.
You declined to measure the accessibilty of 2 pages known to have several serious accessibility issues.
Someone has kindly sent the results to me.
Gez's inaccessible page which is designed to fail 11 out of 17 Priority 1 checkpoints scored 10/10 for accessibilty with SiteMorse.
Bruce's page equally demonstrates how automated tools cannot be used to measure accessibility by scoring 10/10 for accessibility with SiteMorse.
Both sites score top marks with SiteMorse which was unable to detect any of the errors on either page.
The results page also informs your client that "manual testing would have taken 29 minutes" In both these cases I think that would have been 29 minutes well spent as all of the issues missed or ignored by SiteMorse could have been detected manually.
| Jon R wrote: |
| However, simply flagging all pages without an <h1> (as anything other than a 'warning') would almost certainly be too crude. |
This is one issue of many Jon. I would consider that marking any Priority 1 checkpoint and probably half of the priority 2 checkpoints as anything other than a warning is highly misleading. That is my honest opinion. Feel free to use that in a constructive way.
I have tried to be as clear as possible with my next question. I appreciate that you regard many of my questions as vague. I will try to do better with my next but If you feel that I am being vague I will try again as this is important.
This question relates to the relevance of automated accessibility test results.
Here is the statement that you said that you agree to:
| Quote: |
| None of the Priority 1 checkpoints can be fully verified by automated testing alone. |
Q.6 Considering:
a. A web manager can and has acheived a higher SiteMorse score by meticulously deleting all his headings (please don't do this at home),
b. Demonstrably inaccessible pages with multiple and serious checkpoint violations are awarded 10/10 by the SiteMorse accessibility testing tool,
c. None of the priority 1 checkpoints can be fully verified by automated testing alone,
How are the results from the SiteMorse tool accurate enough to be used to compare one website with another?
_________________
Grant Broome
Blog
CDSM
Shaw Trust
Share:
Sorry guys I know you wanted this to be between the both of you, but at only 20% of the final rating and with the disclaimer of:
| Sitemorse UK Gov report April 06 wrote: |
| Automated testing does not indicate overall accessibility or eGMS compliance - Manual testing is still required |
The general line of questioning does seem somewhat harsh.
Sorry, just felt the need to stick my 2p in - I'll let the both of you get on with it.
_________________
Red Ant
Share:
| Grant Broome wrote: |
| You declined to measure the accessibilty of 2 pages known to have several serious accessibility issues.
Someone has kindly sent the results to me. |
As I said, testing those pages is devoid of meaning. What definition of "accessibility" are you using that applies to pages which contain no information and provide no service? SiteMorse is designed to test real sites - testing deliberately-constructed pathological cases proves absolutely nothing.
| Grant Broome wrote: | ||
This is one issue of many Jon. |
It's a shame that you declined to answer my question there. I would be interested in your answer, and I think it would likely be a constructive and instructive topic to explore.
| Grant Broome wrote: |
| I would consider that marking any Priority 1 checkpoint and probably half of the priority 2 checkpoints as anything other than a warning is highly misleading. That is my honest opinion. Feel free to use that in a constructive way. |
I'm sure it goes without saying that I fundamentally disagree with you on that point. I am not sure how you think that your statement could be "used constructively" given that you have presented merely an entirely negative opinion without any accompanying justification.
| Grant Broome wrote: |
| a. A web manager can and has acheived a higher SiteMorse score by meticulously deleting all his headings (please don't do this at home), |
I am not sure of the relevance of this.
| Grant Broome wrote: |
| b. Demonstrably inaccessible pages with multiple and serious checkpoint violations are awarded 10/10 by the SiteMorse accessibility testing tool, |
I strongly dispute that you have shown this to be true, as explained above.
| Grant Broome wrote: |
| How are the results from the SiteMorse tool accurate enough to be used to compare one website with another? |
SiteMorse is extremely accurate at measuring what it is designed to measure - what it is advertised as measuring - automated testing of (among many other things) supported WCAG guidelines.
Share:
| Jon R wrote: |
| testing deliberately-constructed pathological cases proves absolutely nothing. |
Gez's page was designed to test the ability of automated tools to detect checkpoint violations. It was an honest development by an honest and respected accessibility specialist that explores the limitations of automated tools. SiteMorse performed poorly as it was not able to detect serious checkpoint violations. This demonstrates that SiteMorse cannot be depended upon to make the distinction between a web page that is accessible and one that is not.
I have frequently seen instances of Gez's test cases in many websites that I have helped by testing their web pages manually. You seem to be either saying that instances of these test cases do not exist or you are saying that they are not important. Either way, SiteMorse does not test for them.
| Jon R wrote: |
| It's a shame that you declined to answer my question there. I would be interested in your answer, and I think it would likely be a constructive and instructive topic to explore. |
I did answer your question. It should be nothing more than a warning that is in need of manual verification along with all priority 1 checkpoints and nearly half of priority 2. That's why SiteMorse scores based on "passing" these checkpoints are regarded by many as highly mis-leading and unethical which brings us back to our discussion.
| Jon R wrote: | ||
I am not sure of the relevance of this. |
OK, let me get this clear. You are not sure how the removal of a major accessibility feature can result in a better SiteMorse score and higher League Table ranking has any relevance to a discussion of the value of SiteMorse League tables.
| Jon R wrote: | ||
I strongly dispute that you have shown this to be true, as explained above. |
These are indisputable facts verified by your own tool. Anyone using the SiteMorse tool can test this for themselves. It is not good enough to dismiss the test cases as they are examples of problems that exist in the real world.
| Jone R wrote: |
| SiteMorse is extremely accurate at measuring what it is designed to measure - what it is advertised as measuring - automated testing of (among many other things) supported WCAG guidelines. |
Supported WCAG guidelines? Supported by whom?
Anyway, that doesn't answer the question. I did not ask how accurate SiteMorse is at measuring what it was designed to measure. That's a completely different discussion. This question is concerned with the comparison of one site with another.
Put another way:
Q.7 Is it possible to tell whether one site is more accessible than another using automated testing alone when we can both agree that not even one of the Priority 1 checkpoints can be fully tested using SiteMorse?
_________________
Grant Broome
Blog
CDSM
Shaw Trust
Share:
Well, I think I know what the answer to seven is going to be. 
_________________
};-) http://www.xhtmlcoder.com/
WVYFC chose the Yorkshire Air Ambulance as the main charity to fund raise for in 2006
Share:
Without opening this up to broader discussion at this point,
Jon, I appreciate that my questions have been very direct. "Brutal" as one of my colleagues described it. But we are discussing one of the major concerns that people of this forum have about the SiteMorse tool - its marketing methods and one of the main reasons why SiteMorse comes under so much fire from the accessibility community. My hope is that this thread will dissolve much of that bad feeling by directly addressing these concerns.
This is an important exercise in establishing some key facts about the value of automated testing and the SiteMorse tool.
Even though our views conflict on many points we have come to an agreement that automated tools such as SiteMorse cannot fully test any priority 1 checkpoint. This is enough on it's own to illustrate the limitations of automated testing alone.
If we can also agree that League tables are at best only a broad reflection of general accessibility and export that view in order to better inform the general population on how they should be used then I believe that we can move forward with some constructive discussion.
Please provide your response to question 7.
_________________
Grant Broome
Blog
CDSM
Shaw Trust
Share:
I have a question for Grant:
Do you accept and understand what Jon has stated several times - that the SiteMorse league tables/reports are based on performance across several criteria, only one of which is accessibility?
It seems to me that you are intent on proving that SiteMorse reports are worthless because they do not adequately test the accessibility of a site, whereas Jon has repeatedly stated that they test a lot more than just that one aspect.
_________________
The Watchmaker Project - my personal blog
29digital Design Studio - freelance web design/development
Share:
sorry to butt in here but while I agree SM tests more than just accessibility, the assumption should not be made that just because the accessibility aspect is being focussed upon by Grant that this means there is an agreement that all the other tests are valid.
Equally no contrary assumption (that other tests are invalid) should be drawn: no assumptions should be made, by anyone, on the validity or not of any test.
_________________
Jack Pickard The Pickards Information Services| Blog | Twit
Share:
Buddy,
| Quote: |
| SiteMorse league tables/reports are based on performance across several criteria, only one of which is accessibility? |
I have never debated this statement although it does distract from the fact that it is nevertheless an essential component of the tables and the one that most concerns members of this forum.
It should be made clear that this is a discussion about accessibility and the ethics of the inclusion of limited automated web accessiblity testing in league tables as a marketing tool. I have no comment on the accuracy of other tests that make up the league tables.
Can we just keep the thread clear until Jon has had a chance to reply to this last question and then if Jon agrees we'll open up the discussion.
_________________
Grant Broome
Blog
CDSM
Shaw Trust
Share:
| Grant Broome wrote: |
| Gez's page was designed to test the ability of automated tools to detect checkpoint violations. It was an honest development by an honest and respected accessibility specialist that explores the limitations of automated tools. |
I am not sure why you are emphasising his honesty, since it was not, as far as I am aware, in question. However, you have still not explained how such a deliberately-constructed test page can even be considered to possess such a quality as "accessibility" or "inaccessibility". Actually I don't believe it even "fails" as many guidelines as he thinks it does, no matter how you look at it, but that's getting severely off track
| Grant Broome wrote: | ||
I did answer your question. It should be nothing more than a warning that is in need of manual verification along with all priority 1 checkpoints and nearly half of priority 2. |
I don't believe you did answer the question. You have now got closer to answering it, but the obvious next step is - why do you think that? Using the specific example I already gave, let us suppose that SiteMorse generated a priority 2 'fail' if a document contained over, say, 1kB of textual content but did not contain any headings (it does not currently do such a thing, this is just for the sake of argument). You would presumably believe that to be "bad". Why?
| Grant Broome wrote: |
| That's why SiteMorse scores based on "passing" these checkpoints are regarded by many as highly mis-leading and unethical |
Steady on. I have gone to considerable effort in this discussion to keep things civilised and objective. I would appreciate it if you would do me the same courtesy I am doing you. Also note that SiteMorse does not base anything on "passing" accessibility checkpoints.
| Grant Broome wrote: |
| OK, let me get this clear. You are not sure how the removal of a major accessibility feature can result in a better SiteMorse score and higher League Table ranking has any relevance to a discussion of the value of SiteMorse League tables. |
Indeed, given that people do not, in general, do such a thing.
| Grant Broome wrote: | ||||
These are indisputable facts verified by your own tool. |
No, they are not. The scores are indisputable facts (that's kind've the point of them, after all), but the accessibility or otherwise of your test pages is highly questionable.
| Grant Broome wrote: | ||
Supported WCAG guidelines? Supported by whom? |
Supported by the SiteMorse tool, of course.
| Grant Broome wrote: |
| Anyway, that doesn't answer the question. I did not ask how accurate SiteMorse is at measuring what it was designed to measure. That's a completely different discussion. This question is concerned with the comparison of one site with another. |
I would appreciate it if you stop mis-using the word "accuracy", then, if that is not what you mean.
| Grant Broome wrote: |
| Q.7 Is it possible to tell whether one site is more accessible than another using automated testing alone when we can both agree that not even one of the Priority 1 checkpoints can be fully tested using SiteMorse? |
Often it is, yes.
The gist of your argument seems to be that because it is not possible to measure everything, we should measure nothing. I don't see how that follows. Bear in mind that what we do with automated testing is simply not possible with manual testing - it would be too time-consuming and expensive. Even if an organisation buys in external consultants to manually verify the accessibility of a web site, they will usually only examine a few pages.
I have seen someone quote the saying "Be sure to measure what you value, because you will surely come to value what you measure" as if that is somehow an argument against SiteMorse's league tables. Ironically, it is surely in reality an argument in favour - if accessibility is not measured at all, the saying implies, it will not be valued at all either.
I believe that the SiteMorse league tables have, in the sectors we cover, increased the visibility of accessibility as an issue and therefore, in general, been a positive benefit to those within the organisations trying to promote accessibility, and to those users who have accessibility needs.
Share:
All times are GMT
You cannot post new topics in this forumYou cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum