←back to thread

125 points robin_reala | 1 comments | | HN request time: 0.193s | source
Show context
simonw ◴[] No.46203241[source]
Something I'm desperately keen to see is AI-assisted accessibility testing.

I'm not convinced at all by most of the heuristic-driven ARIA scanning tools. I don't want to know if my app appears to have the right ARIA attributes set - I want to know if my features work for screenreader users.

What I really want is for a Claude Code style agent to be able to drive my application in an automated fashion via a screenreader and record audio for me of successful or failed attempts to achieve goals.

Think Playwright browser tests but for popular screenreaders instead.

Every now and then I check to see if this is a solved problem yet.

I think we are close. https://www.guidepup.dev/ looks extremely promising - though I think it only supports VoiceOver on macOS or NVDA on Windows, which is a shame since asynchronous coding agent tools like Codex CLI and Claude Code for web only run Linux.

What I haven't seen yet is someone closing the loop on ensuring agentic tools like Claude Code can successfully drive these mechanisms.

replies(12): >>46203277 #>>46203374 #>>46203420 #>>46203447 #>>46203583 #>>46203605 #>>46203642 #>>46204338 #>>46204455 #>>46206651 #>>46206832 #>>46208023 #
1. westont5 ◴[] No.46208023[source]
The agent-driving-a-screenreader footgun is that it's quite easy to build UI that creates a reasonable screereader UX, but unintentionally creates accessibility barriers for other assistive technologies, like voice control.

Ex: a search control is built as <div aria-label="Search"><input type="search"/></div>. An agent driving a screenreader is trying to accomplish a goal that requires search. Perhaps it tries using Tab to explore, in which case it will hear "Search, edit blank" when the input is focused. Great! It moves on.

But voice control users can't say "click Search". That only works if "Search" is in the control's accessible name, which is still blank, the outer div's aria-label has no effect on components it wraps. Would an agent catch that nuance? Would you?

You could realign to "I want to know if my features work for screenreader, voice control, switch, keyboard, mobile keyboard [...] users", but you can imagine the inverse, where an improvement to the voice control UX unintentionally degrades the screenreader UX. Accessibility is full of these tensions, I worry an multi-agent approach would result in either the agents or you getting bogged down by them.

I think a solution needs to incorporate some heuristics, if you consider WCAG a heuristic. For all its faults, a lot of thought went into rules that balance the tensions reasonably well. I used to be more of the "forget WCAG, just show me the UX" attitude, but over the years I've come to appreciate the baseline it sets. To the example above, 2.5.3 Label in Name clearly guides you towards setting an accessible name (not description) on the search itself, patching up support for screenreaders and voice control.

Not that WCAG concerns itself with the details of ARIA (it packs all that complexity into the vague "accessibility supported"[1]). We do need more seamless ways of quickly evaluating whether ARIA or whatever markup pattern has the intended rendering in screen readers, voice control, etc, but at a level that's already constrained. In the example, WCAG should apply its constraints first. Only then should we start running real screen readers and analyzing their audio, and to avoid the footguns that analysis should be at a low level ("does the audio rendering of this control reasonably convey the expected name, state, value, etc?"), not a high level ("does the audio rendering contain the info necessary to move to the next step?").

Unfortunately both agents and heuristic-driven accessibility scanning tools struggle to apply WCAG today. Agents can go deeper than scanners, but they're inconsistent and in my experience really have trouble keeping 55+ high level rules in mind all the time. In the example, an agent would need to use the accessibility tree to accomplish its goal and need to reject a node with label "Search" containing a role=textbox as an option for searching (or at least flag it), which is made trickier by the fact that sometimes it _is_ ok to use container labels to understand context.

I think the answer might be to bundle a lot of those concerns into an E2E test framework, have the agent write a test it thinks can accomplish the goal, and enter a debug loop to shake out issues progressively. Ex: if the agent needs to select the search control for the task, and the only allowed selector syntax requires specifying the control's accessible name (i.e. Playwright's getByRole()), will the agent correctly get blocked? And say the control does have an accessible name, but has some ARIA misconfigurations — can the test framework automatically run against some real screenreaders and report an issue that things aren't working as expected? We've done some explorations on the framework side you might be interested in [2].

[1] https://www.w3.org/TR/WCAG22/#dfn-accessibility-supported [2] https://assistivlabs.com/articles/automating-screen-readers-...