No ARIA is better than bad ARIA

There are thousands of blind people on the net. Can't you hire one of them to test for you? Please?

Realistically I'm unlikely to do that for my dozens of non-income-generating personal projects.

I still want them to be accessible!

(The amount of accessibility testing I want to do would bankrupt me very quickly.)

If you don't want this to break eventually, you need it tested every time your CI/CD test suite runs. Manual testing just doesn't cut it

AI in your CI pipeline won't help either then, if it randomly gives different answers

An AI-generated automated testing script in your pipeline will do great though.

Why not both?

So does hiring a person or tests which rely on entropy because exhaustive testing is infeasible. If you can wrangle the randomness (each has different ways of going about that) then you end up with very useful tests in all 3 scenarios, but only automated tests scale to running every commit. You probably still want the non-automated tests per release or something as well if you can, depending what you're doing, but you don't necessarily want only invariant tests in either case.

And then we're back at your own:

> I'm not convinced at all by most of the heuristic-driven ARIA scanning tools.

That's entirely different.

ARIA scanning tools are things that throw an error if they see an element that's missing an attribute, without even attempting to invoke a real screenreader.

I'm arguing for automated testing scripts that use tools like Guidepup to launch a real screenreader and assert things like the new content that was added by fetch() being read out to the user after the form submission has completed.

I want LLMs and coding agents to help me write those scripts, so I can run them in CI along with the rest of my automated tests.

This. I've been doing a lot of accessibility work and it seems like the one thing nobody ever does is talk to a screenreader user.

We have the exact same problem with visual interfaces, and the combination of manual testing for major changes + deterministic UI testing works pretty well.

Actually it could be even easier to write tests for the screen reader workflow, since the interactions are all text I/O and pushing keys.

I think this is the way, automated testing for all patches, small changes, and manual testing for big releases.

Most of us can't, no. Sorry. There's just not enough money in the things we make.

> There are thousands of blind people on the net. Can't you hire one of them to test for you?

Testing is a professional skill -- not all blind people are good at accessibility testing, just as not all sighted people are good at GUI testing.

My team has carved out an accessibility budget so that every couple years we can hire an accessibility consultancy (which employs a couple blind testers) for a few tens of hours of work to review one of our application workflows. Based on the issues they identify we attempt to write tests to prevent those classes of issues across the whole application suite, but our budget means that less than one percent of our UI has ever been functionally tested for accessibility.

It comes down to cost/benefit. Good testers are expensive, good accessibility testers doubly-so. And while I personally think there's a moral imperative and maybe a marketing angle, improving accessibility truthfully doesn't seem to meaningfully improve sales. But if the testing costs came down by a couple orders of magnitude it would be a complete game-changer.

That's very different from what I thought you were arguing for in your top comment, though: a computer-use agent proving the app is usable through a screen reader alone (and hopefully caching a replayable trace to not prompt it on every run).

Guidepup already exists, if people cared they'd use it for tests with or without LLMs. Thanks for showing me this tool BTW! I agree testing against real readers is better than using a third-party's heuristics.

I think one path that could be used and wholy underrated is simply sitting with a user while they use/navigate your application. The user themselves doesn't necessarily need to be skilled tester, and you will need to have more session time than a skilled tester, but it does work and can help a lot.

Also, try using your app/site without a mouse. I've found it funny how many actual, experienced, sighted testers don't actually know the keyboard navigation for things like triggering menus, select boxes, etc. Personally, I don't think I could get used to the voice navigation myself, it's not that it doesn't work, it's just kind of noisy. Although, most sites are excessively noisy visually imo.

Completely agree on both counts. We do usability testing, including with keyboard-focused advanced users.

But usability testing with blind users presents some unique challenges. A past org I worked at ran some usability studies with blind users [1] and while I was only tangentially involved in that project it seemed that subject recruitment and observations were much more complex than typical usability studies. I haven't managed to run a usability study with blind participants at my current org though we have discussed ways we could recruit blind users for studies -- our software is complex enough that we'd need someone who is both blind and a prospective user of our software.

[1] https://www.bloomberg.com/ux/2018/08/28/visually-impaired-wo...