Episode 19 – UX Prototypes, Overconfident Accessibility, and the Automation Trap

Episode Summary

This episode examines three timely questions at the intersection of UX, accessibility, and AI: why AI prototyping ranks highly among UX methods, how the Dunning-Kruger effect can create misplaced confidence in accessibility work, and why fully automating accessibility testing remains unlikely. Mark Miller and Justin Stockton emphasize early testing with people with disabilities, healthy skepticism about automated results, and the continuing importance of human experience as technology evolves.

In this episode:

Mark and Justin discuss three accessibility pieces and what they signal for the industry right now.

The 100 Most Common UX Design Methods, Ranked by Value

Jakob Nielsen

• Jakob Nielsen ranked 100 UX design methods using breadth, decision-making altitude, persuasive power, and cost to estimate their overall value.
• AI prototyping ranked first because it can quickly produce an interactive concept that helps stakeholders understand and support an idea; sketching and problem statements also ranked highly.
• The hosts saw potential for accessible prototypes to support “shift-left” testing by validating workflows early with people with disabilities.
• They cautioned that a persuasive prototype can be mistaken for production-ready software, even though it may need to be discarded and rebuilt securely, accessibly, and to specification.
• Accessibility evaluations ranked 74th as a standalone method, but the hosts stressed that the ranking reflects individual methods and that accessibility testing remains essential when methods are combined.

2. The Dunning-Kruger effect in accessibility

Craig Abbott

• Craig Abbott applies the Dunning-Kruger effect to accessibility: limited knowledge can produce high confidence, while deeper expertise often reveals complexity and uncertainty.
• For developers, narrow feedback loops can make code appear complete while accessibility, security, and other concerns fall outside the immediate acceptance criteria.
• Automated checkers can reinforce false confidence because a high score does not cover every barrier and may include false positives or false negatives.
• Meaningful feedback from people with disabilities must be sought deliberately; users often abandon an inaccessible experience rather than reporting the problem.
• The same pattern can affect whole organizations when teams and leaders rely on incomplete automated results and collectively assume their process is sufficient.

3. Automating all accessibility testing is a pipe dream

Nicolas Steenhout

• Nic Steenhout challenges the idea that AI will soon automate all accessibility testing and eliminate the need for manual evaluation.
• The hosts agreed that AI may automate more WCAG checks, but vendor claims need independent verification and a consistent way to calculate coverage.
• AI cannot replace human experience when assessing usability, comprehension, timing, and whether an interaction truly works for people with disabilities.
• Rapid advances in AI may improve testing while also accelerating inaccessible code and changing digital interactions faster than standards and practices can adapt.
• They viewed an all-in automation strategy as risky today, while remaining open to future progress and emphasizing continued human oversight.

Go to all episodes

Transcript

Mark Miller (00:06)
Hey, welcome to the Accessibility Breakdown. I am Mark Miller, and this is.

Justin Stockton (00:10)
Justin Stockton.

Mark Miller (00:11)
Hey, every week we pay tribute to those who stand out to us as leading by accessibility by picking three topics that strike us in some way. This week we’re going to talk about the The 100 Most Common UX Design Methods, Ranked by Value. This is by Jacob Nielsen. And from there we’re going to go into the

Dunning Kruger effect in accessibility, really interesting.

use of the Dunning Kruger effect here to explain some common issues, I think, or things that we think about a lot in accessibility by Craig Abbott. And then Automating all accessibility testing is a pipe dream. And this is by our friend Nic Steenhout. Justin, you want to start with the The 100 Most Common UX Design Methods, Ranked by Value This article was we’ll call it complete.

Justin Stockton (01:11)
What do you

Mark Miller (01:12)
I was

I went to read this article and you know how it gives you like the time to read?

Justin Stockton (01:17)
Yeah.

Mark Miller (01:17)
Forty six minutes. I was like, might okay. Yeah.

Justin Stockton (01:20)
I didn’t even notice that at the top. That explains

why I kinda got sucked into it the other day.

Mark Miller (01:25)
Yeah, it’s very

interesting though. in it and it’s very well sort of studied, researched, thought out. so why don’t you why don’t you lay the groundwork for this one for us?

Justin Stockton (01:41)
Yeah, so this is an article written by Jakob Nielsen of the Nielsen Norman Group, or formerly of the Nielsen Norman Group, I guess. I had to look up because I noticed he was posting this on another site and

I was like,

Mark Miller (01:54)
Mm-hmm.

Justin Stockton (01:54)
wait, what happened there? Yeah, he retired from the from that UX group in twenty twenty three, I think I saw. anyway, tangent. anyway, so what this was was he went through and looked at

UX design methods, things like sketching, problem statements, design briefs, all these different types of things that all go that are all methods that are used within the particularly within the design groups within organizations. And then he ranked them based off of four criteria in order to determine to prescribe

some value to each one of these 100 and then he ranked them. so there are things so he looked at the breadth, like how many kinds of design problems, what are the problem stages that the method serves? so like is this something that can use like is this method something that can be used like one time in this

Mark Miller (02:59)
Mm-hmm.

Justin Stockton (03:00)
one kind of niche thing? Or does it apply to a lot of different phases throughout the design process?

there’s also altitude. So what is the level of decision that the method’s output primarily feeds into? So is this is it like surface decisions, is it structural decisions, all the way up to like some more strategic types of decisions that the method helps with? things it also looks at persuasion. so how effective is the method at getting

other people to care outside of just designers. like, you know, is this something that really just designers do because it means something to them? or is this something that is gonna be, it’s gonna actually start to persuade executives in the C suite

Mark Miller (03:52)
Mm-hmm.

Justin Stockton (03:53)
to get bring in budgets and that sort of stuff.

And then finally cost.

Mark Miller (03:56)
Sort of

yeah, to wrap it all up, I mean the way I looked at this when we got through all that was that it was all these things were for the purpose of measuring sort of best bang for your buck. If you spend time and when and when we’re talking about a design method, it’s and correct if I’m me if I’m wrong here, but it’s sort of how do we what method do we use to create the initial

look of this thing, right? So, so application like here’s the application we want to build. And

Justin Stockton (04:27)
Mm-hmm.

Mark Miller (04:28)
when we do that, which one of these methods works the best? And you just described all the different sort of parameters, but it’s like if we spend our time doing this one, what’s going to cover the most? What’s going to make the most people happy? What’s going to cover the most techniques? What’s to cover everything the most? So he lays it out like you did in these very specific measurable

ways to come up with that metric. So from there, this is where it’s this is where it gets really interesting to me, Justin, is in I I mean this he came up with a hundred results here. So we probably only talked about the first couple, right? But the number one result, I I was like, it’s the opposite of what people not necessarily in design, but what people keep saying, right?

Justin Stockton (05:18)
Yeah. Yeah, the number one the number one result was AI prototyping, which

Mark Miller (05:23)
Mm-hmm Which means

Justin Stockton (05:26)
well what what is AI prototyping?

Mark Miller (05:27)
Vibecoding.

It’s just AI prototyping.

Justin Stockton (05:30)
Not necess not necessarily. Vibcoding is actually a different a slightly different thing.

Mark Miller (05:36)
When you get down in the article, he talks about it as vibe

Justin Stockton (05:39)
Wow.

Mark Miller (05:39)
coding. Yeah.

Justin Stockton (05:42)
But

when I was looking at it, I was like, I was like, I mean it does make sense because like it does reach everywhere. It gets everyone

interested.

Mark Miller (05:52)
Mm-hmm.

Justin Stockton (05:53)
it’s very easy for you to expose for designers to expose what it is that they’re trying to convey outside of just the design group. And comparatively, it costs very little.

Mark Miller (06:08)
Yeah. So it didn’t cost a lot. He says in there, like, you can have it done before lunch and people can play with it. I think that that

Justin Stockton (06:14)
Mm-hmm. Yeah.

Mark Miller (06:16)
was the big thing. It’s like you’ve got something that people can play with. So that getting executives, getting that buy-in, getting the project to move forward was one of the big things that it ranked. And and by the way, it was four points above number two, which is the highest spread between one and one hundred that

Justin Stockton (06:36)
Yeah, the

value scoring reads a little bit like a PGA tournament where there’s like a number one, a number

Mark Miller (06:41)
Mm-hmm.

Justin Stockton (06:42)
two, and then like fourteen people in third place with

Mark Miller (06:45)
Right.

Justin Stockton (06:45)
the same score. So

Mark Miller (06:47)
And I think number two was drawing, right? Just drawing it out. Yeah, sketching.

Justin Stockton (06:50)
Yeah, sketching. But

what was inter what I did find was interesting was that problem statements was number three. That I

Mark Miller (06:59)
Mm.

Justin Stockton (06:59)
that clearly I did clearly articulating the problem that you’re trying to solve for is gives you lots of breath, lots of altitude. it’s can be good at persuading others outside of design.

But it’s also

Mark Miller (07:15)
Mm-hmm.

Justin Stockton (07:15)
low low cost. And problem statements, ironically, are one of the big things that if you put together a good problem statement, feeds into things like AI prototyping and gives you a much better response from AI.

Mark Miller (07:28)
That

was one of the that was one of the questions I kinda had, like

It seems like a cute few of these methodologies, particularly if your your base methodology was the AI prototyping, would feed into to that, right? That by by the time you get down to the point of

creating something good with AI, right, or or vibe coding. Like we’ve all tried to vibe

code before.

Justin Stockton (07:51)
Mm-hmm.

Mark Miller (07:52)
you you actually have done a bit of work to kind of get to the point of that that final thing. But I I don’t think it I don’t think that changes changes the methodology at all. And then the other great point I thought he made is he’s like, and this it and this starts to ta show us like the value of vibe coding because obviously in an accessibility in across the board

There’s a lot of question about that method of of of coding and how good the results are and particularly how accessible the results are. But the point here was like, yeah, make this thing, let people play with it. And then when they go, okay, yeah, let’s build it, scrap that code and start over. He w there was no suggestion that you keep building from that point, but that that’s the point in which you can it and this is the way I looked at it as like,

A a lot of that design methodology is let me try and get what’s up here in my head out in front of other people so they can understand w what I really want to build. And being able to build something that’s mostly working and that you can play with is just a fantastic way to do that, particularly when you can do it so quickly. My my question for you, Justin, on this one is what like what is what do you think the implic implications are

are for accessibility. If less just say we’re gonna dive in and we’re gonna use this as our method to create the prototype and and and sell it across the organization.

Does it have an impact on accessibility?

Justin Stockton (09:26)
Well, I mean, yes. of course it does because if you what what is it that you’re trying if you’re building a prototype, what is it that you’re trying to evaluate? Are you trying to evaluate the task itself? Are you trying to

Mark Miller (09:40)
Mm-hmm.

Justin Stockton (09:40)
make sure that the task itself is easy? If you’re trying to make sure that the task itself is easy for everyone, then you need an accessible prototype so that you can test with

everyone, right? I mean

Mark Miller (09:50)
Mm-hmm. Well well, part of what I was thinking

here too is that, you know, a lot of this discussion that we have around accessibility, you know, accessibility professionals like you and I and the folks we’ve had on this podcast, you know, we’re we’re always on the shift left soapbox. So it also seems to me that if this is the way that we’re gonna move forward with a design prototype, that we could actually start to more quickly prototype a workflow or a methodology.

And have in in shift left. So we could build this little, it doesn’t even have to be the whole application, but we can say we’re we’re gonna do X, Y, and Z and we want to make sure it’s accessible. And this is our this is how we think that this should code out and how it should behave. Let’s vibe it out real quick, hand it to a screen reader user, let somebody who has a hearing impairment mobility, you know, use it and and let them tell us what they think about how you know.

how it all works. And if that if the methodology we use pans out, or once we refine it, now we include it. Like to me, it’s sort of interesting in that early on testing to really, really proof things out before you start to build something completely.

Justin Stockton (11:08)
Yeah, I think I mean it’s it’s good for understanding the task and making sure that the workflow that you envision is actually going to work.

Mark Miller (11:17)
Mm-hmm.

Justin Stockton (11:18)
and help you to work through some of the some of the business logic and things like that behind the scenes and help everyone get everyone on the same page about what it is. My worry is that I don’t I’d be really curious to know from others, but anytime that I have built a prototype with

one exception, that prototype then becomes part of part of what’s deployed to production. Like it becomes the root and everything now grows from that tree

Mark Miller (11:46)
Well I think that’s the warning and the caveat. Yeah. Yeah.

Justin Stockton (11:51)
and you you’re not starting from the right spot.

Mark Miller (11:55)
But do you think that’s

our inexperience? Like we’re all very new at coding and using AI for all these things. Is that could we chalk that up to inexperience and say we need to we need to understand the value of these things that we’re building and we need to understand when we need to reel back and and scrap that and start over, you know? Not that AI is not part of the start over, but

Justin Stockton (12:19)
Well, I think it’s n no, I think what happened in my case, the the every time that I’ve ever built a prototype, it always comes down to, Hey, that looks great. When can we see it in production? When can we see it live? It’s like, Well, we’ve

Mark Miller (12:32)
Mm.

Justin Stockton (12:32)
got to go build it now. It’s like but but I just I just did it.

Mark Miller (12:34)
so you’re saying that that that

that the d the vibe sort of vibe coding it gives it to you live and working. Yeah, yeah, yeah. I gotcha. I gotcha. Yeah.

Justin Stockton (12:39)
Yes. Yes gives you it gives you the semblance

of wow, this thing actually works, even though there’s probably no database, no back end, there’s you know, it doesn’t have all of the things that it needs, but you’ve stood something up and now you’ve now you’ve sold it to other parts of the organization. You know, he talks about this

Mark Miller (12:59)
Mm-hmm.

Justin Stockton (12:59)
with persuasion. AI prototyping has a persuasion of nine out of ten. so you’ve convinced other parts of the organization this is a great thing.

Mark Miller (13:08)
They can play with it.

Justin Stockton (13:09)
And now they want you to go with it.

Mark Miller (13:11)
Yeah.

Justin Stockton (13:11)
Like, and so how do you say, no, no, this thing we all we did was we use this thing to prove a theory. Now that we ha now that we’ve worked through that cheaply, we’re gonna throw it away and we’re gonna we’re gonna have to build it for real.

Mark Miller (13:27)
Yeah. It’s almost as if like, you know, you you look at designing a building and you’ve got a model, right? And the model can be compelling. But if you could

Do a couple things and allow people to walk through it. And it actually had some pictures on the wall, and they could open the doors and they could look at how big the closets were or whatever the case is. Just

Justin Stockton (13:49)
Mm-hmm.

Mark Miller (13:50)
imagining how much more compelling that would be than that little model. I think that’s that’s why we see this big sort of sh difference. And you know, you same thing, you have to scrap the building and build it to code to spec safely, you know, all that kind of stuff.

so very interesting. I’m I’m interested. I I would love to think about how you could take that ability and start to apply it to early on shifting left accessibility testing before you started even you know, as you were building code, deciding how you were gonna code. I don’t know why you can’t break that thing into little pieces and test little pieces of it as you go on and then and then have that be part of your

How you inform how you assemble the rest of it. Yeah.

Justin Stockton (14:40)
you absolutely can, but it’s

it’s a it’s a matter of just having the right, you know, skills and things in place in your, you know, whatever

Mark Miller (14:48)
Mm-hmm.

Justin Stockton (14:49)
you’re using to do your prototyping, and it’ll help, you know, it’ll build things to be more accessible to you know, out of the gate than probably you would get from just, you know, no prompts

Mark Miller (15:00)
Yeah.

Justin Stockton (15:01)
no skills.

Mark Miller (15:02)
Yeah. I like the idea of building it and handing it to somebody. Saying, try this out. How’s it work for ya? You know, little pieces

Justin Stockton (15:07)
Yeah. What well what was it and

Mark Miller (15:08)
of it.

Justin Stockton (15:09)
yeah. And what was interesting about this article too is when I saw this in Ricky Onsman’s weekly digest, he pointed out

Mark Miller (15:17)
Mm-hmm. Fantastic digest.

Justin Stockton (15:21)
he pointed out that accessibility evaluations in this list were came in at seventy four, seventy four.

Mark Miller (15:30)
Mm-hmm.

Justin Stockton (15:31)
which is pretty far down the pretty far down the list.

And when I look at that, I think like specifically some of the the numbers that it that accessibility evaluations got. says that the breadth is four, so getting outside of the you know, it doesn’t really go much further outside of the design. the persuasion persuasion is pretty high at six.

Mark Miller (16:00)
Mm-hmm.

Justin Stockton (16:02)
cost is middling, so it’s it’s at sitting at five, so it’s

Not too expensive, not ch not too cheap. but that really that starts to drive those things start to drive the value of accessibility evaluations as a singular method in this toolbox that designers have. That’s the that’s the kind of thing that I came back for. Like when I first saw the the article in Ricky’s the the this article listed in Ricky’s

Mark Miller (16:28)
Mm-hmm.

Justin Stockton (16:28)
thing, I was like, the pitchforks are gonna come out for this.

But then once I read the article, like I understood like where he Jacob Nielsen was coming from.

Mark Miller (16:39)
It’s it’s also

looking at these things as a s is singular entities, right?

Justin Stockton (16:43)
Yes.

Mark Miller (16:44)
Like if if you’re going to only do one and I don’t that’s not how it works, right? That it works and that you you combine a few things to get exactly what you want. But this is great information to maybe inform how you would can combine things, right? ‘Cause you’ve got you’ve got to have accessibility testing.

Justin Stockton (17:00)
Yeah. And I also didn’t see anywhere where he this also s and I may have missed this in the article ’cause like you said, it is a the it’s a forty six minute read

Mark Miller (17:09)
Mm-hmm.

Justin Stockton (17:10)
for an article. So it’s a pretty fair fairly long article.

Mark Miller (17:13)
Which is about

an hour and two hours for me just

Justin Stockton (17:17)
But

I didn’t see any like this really kind of see it this seemed to be drawing on his experience as a leader

Mark Miller (17:24)
Mm-hmm.

Justin Stockton (17:25)
in UX design. and not it wasn’t, you know, community driven or he didn’t do a big research

Mark Miller (17:30)
Right, right.

Justin Stockton (17:31)
or anything. He just kind of went through and said, you know, AI prototyping blah, blah, blah, blah, blah. and so, you know, a more rigorous study, you know, the numbers may jump around, do different things based on different people’s experiences. So

Mark Miller (17:46)
Well, great, really interesting article by Jacob Nielsen. let’s jump to this next one, which was the Dunning Kruger effect in accessibility. And this is by Craig Abbott. The so first of all, I I’m sure everybody out there is going, Dunning Kruger effect. I

I I

Justin Stockton (18:06)
Mm-hmm.

Mark Miller (18:06)
I I’m supposed to know what that is. it’s is it Murphy’s Law? Is it is it Schrdinger’s cat? Is it

You know, like there’s all these little these things that I think sometimes we think about and we forget what they are. so the Dunning Kruger effect, I’m gonna bring it right back. Just snap it right back for people. And that is that if you think you’re do it it basically through studies, through these studies, it kind of the the final conclusion here is if you really are confident and think you know what you’re talking about, you probably don’t and aren’t. And if you are unsure and a little nervous.

that you don’t know what you’re talking about, you’re probably doing much better than you think. It’s kind of this reversal of whatever you think. It’s it’s the opposite. And Craig takes this and applies this to accessibility, which is a fantastic application of it because

We often you know, I think that

When you first start to learn accessibility, you know, you’re pretty clear that there’s a lot. And I in thinking about this from the standpoint of a designer primarily, right? but there’s a lot that you don’t know. And you I don’t know a lot. And you start to learn, and then very quickly, that surface level of accessibility knowledge gives you a whole bunch of confidence. And you start shipping stuff and don’t quite realize that.

It goes deep. There, the accessibility is a deep rabbit hole. And so once you start to get closer and closer to the bottom of that rabbit hole, you start learning all of the things you didn’t know you didn’t know. And that starts to shake your confidence. Cause if you’re like, if I was so confident and then I kept going, and there’s all these things I didn’t know, all these subtleties I didn’t think of, that’s where you start to take that step

back and go to our

really, really have this stuff. So it kind of makes, you know, the the d it in the the article challenges the Dunning Kruger effect. Like there’s been multiple people who’ve studied it and come up with different conclusions around it, blah, blah, blah. But it doesn’t matter, right? The the the overall concept is is interesting to think about accessibility through. Yeah,

Justin Stockton (20:28)
It’s a good thought exercise, yeah.

Mark Miller (20:30)
it’s a good thought exercise. So none of that matters. but I think that that’s a lot of people’s experience.

It seems easy once you get relatively into it, you know? But when you really get deep, those folks that have really been around are the ones that probably are the most cautious about whether or not they really have it figured out.

Justin Stockton (20:57)
Yeah. Yeah. This was an interesting article for me because at f like I saw the title. This is again another one pulled from Ricky’s list.

Mark Miller (21:06)
Mm-hmm.

Justin Stockton (21:07)
and I whenever I get an article, I’ll kind of like kind of quickly skim down it just to kind of see like I’m looking at the headings. I’m going through and I’m seeing how the how what’s the heading? What how are how is this article broken up? And I already knew what the Dunning Kruger effect was and so I was kinda going down and I saw the

How the the section on how the Dunning Kruger effect often plays out for developers. And the very first

Mark Miller (21:32)
Mm-hmm.

Justin Stockton (21:33)
section was on broken feedback loops. And I was like, okay,

Mark Miller (21:35)
Mm-hmm.

Justin Stockton (21:37)
this is interesting. And because as a developer, a former developer, I don’t know. I still write code. it’s like you are often kind of in this pocket, and you are working, you’re focused on a problem, and

You know what you’re expected to do, and you are just constantly trying to essentially make the test pass. Like whether the ex accept acceptance criteria, whatever, you’re trying you’re just going back and forth in a loop, trying to to build something out to where it works. And then it’s rare that you get to go outside of that loop to have someone else validate it. And

Mark Miller (22:21)
Right.

Justin Stockton (22:22)
by then, by then it’s usually through a pull request.

some sort of team review, that sort of thing. But when you look at this one specifically with the developers, I was just like, yes, because you can get everything working, everything can look good. You could have missed so many different things. Like you may have because you were focused on a very narrow set of problems. And when

Mark Miller (22:47)
Mm-hmm.

Justin Stockton (22:49)
accessibility, that’s when accessibility drops out. That’s when security drops out. Then what’s when all these extra

things that are outside of what it is that you’re actually building fall away and they don’t come back in until you reconnect that feedback loop and you get something that says, what about this? What about this? What about this?

Mark Miller (23:10)
There’s a couple pieces, I think, to this that i one of them is the is the automated piece, right? Like you see so many folks who are charged

Justin Stockton (23:18)
Mm-hmm.

Mark Miller (23:19)
with accessibility and kind of crash course themselves around accessibility going to an accessibility checker and and and trying to make it say a hundred percent or as close as they can or have the light turn

Justin Stockton (23:29)
Mm-hmm.

Mark Miller (23:29)
green or make the fireworks go off or whatever it is that that an accessibility checker does to say, like, hey, you made it.

Justin Stockton (23:35)
That’s when the checkers and more fireworks.

Mark Miller (23:37)
That little they do, wouldn’t it be? That’s you know, that’s what we need to make. But you know, that that circle that completes and the hundred percent in the middle of it.

Justin Stockton (23:44)
Yeah.

Mark Miller (23:45)
And you know, and that’s problematic because because accessibility checkers only d don’t catch everything, right? Or or or don’t catch you know, leave a lot. There’s a big gap. but you become very confident because you have this sort of it’s sort of gamified, right? You know, like I sh I I did it, right? I made the bell ring.

I got the brass ring. I can keep going. but the it and not only is it the fact that it doesn’t catch everything, but it also is that it can just be wrong. You know, it can just be a false, false positive or a false negative. And then you add that on top of the fact that this feedback loop that you’re talking about, a true feedback loop or true feedback from people with disabilities is is

Is difficult to obtain. You know, it’s not the guy next to you is very unlikely to happen to be a screen reader user. But they happen to be a person who shops online and all these other things, right? so you have to actively go out and seek that kind of feedback. And that this is me, man, right? Like

When I go on something and it doesn’t work, I move on. Full stop.

Justin Stockton (25:10)
Mm-hmm.

Mark Miller (25:12)
I can’t hit X quick enough. I can’t move on quick enough. I was trying to do some stuff on the RS what IRS website. Guess what? You’re not allowed to move on from that. You have to finish it. I spent two hours battling with bad usability. just trying to get logged in to to this thing. And

Justin Stockton (25:33)
Least.

Mark Miller (25:35)
If it had been anything else, if you were trying to get me to buy something, I really there’s nothing in this world I want bad enough to spend more than 30 seconds. So that’s the other problem is that people assume people with disabilities are gonna, you know, write them, email them, give them feedback, aren’t using their stuff, like a thousand assumptions. The reality is they’re probably just bouncing off of it pretty quick, never going back, and it’s crickets. You never know that you have a problem.

because that part of the feedback loop doesn’t happen. And quite frankly, they’re under no obligation.

Build something that works if you want people to to use it. So I think that that’s a big part of it too is just to understand that you have to go out and actively seek and really make an effort to make these things work. You can’t assume because you got the right number or you executed the right best practice or whatever the case is that you actually have something that works.

So I really like that and I really liked I love it when people frame things in different ways ’cause it just makes you think differently and I loved the framing around the Dunning Kruger effect. So high five, Craig.

Justin Stockton (26:46)
Yeah. No, this was yeah,

this was really good. And he also goes in and also do want to point out he does also go into like how the Dunning Kruger effect kind of plays out for individuals, but also for organizations.

Mark Miller (26:58)
Mm-hmm.

Justin Stockton (26:59)
like this there it doesn’t necessarily have to play out to individuals. It can play out, you know, across multiple people who all think that

Mark Miller (27:07)
Mm-hmm.

Justin Stockton (27:08)
they know the right thing and they feel confident in that, even though what they think they know isn’t maybe the right thing.

Mark Miller (27:15)
Yeah.

Justin Stockton (27:16)
So

Mark Miller (27:16)
Well, and that goes back to the automation. We see that in organizations when we do like organizational assessments. We’ll

Justin Stockton (27:22)
No.

Mark Miller (27:22)
often uncover the fact that s people are sort of passing things along because an automated tool gives them a certain result and the leadership is like, Yep, yeah, make sure you record that and move on and you’re good. And it’s not, you know, you and I know, probably most of people listening know that that’s not good enough.

So speaking of that, speaking of automated testing and

it not being good

Justin Stockton (27:49)
Yeah.

Mark Miller (27:49)
enough, our next article by Nic Steenhout is titled Automated Automating All Accessibility Testing is a Pipe Dream. And basically, what we’ve got here is that ever since AI has really made its debut on the scene here, there’s been

attempts an assumption, a

Justin Stockton (28:14)
Mm-hmm.

Mark Miller (28:15)
lot of talk that it’s going to be able to if we just go back to the f to to what we were talking about before with automation only catching 30 to 40 percent, that it’s going to be able to catch 100% an air quote of the issues, or that it’s going to fundamentally replace the need for manual testing or whatever fill in the blank that you

wanna fill in there because we’re talking about things that are difficult to do and you’re talking about AI coming in and making it easy to do. And Nic argues in this article that that dream that AI is going to come in and do that for us is not

Sucking out.

That’s it. Just yeah, you have nothing to add.

Justin Stockton (29:06)
I’m really torn on this.

Mark Miller (29:08)
Now that we’ve broken accessibility down for you, we’re done, got people. Tell us how you’re torn on this one.

Justin Stockton (29:15)
I’m

really torn on this. If it and there’s there’s two things that I’m torn on. One is the use is accessibility testing. And I’ve sometimes I get I can I can talk myself in a complete circle, like backwards

Mark Miller (29:32)
Mm-hmm.

Justin Stockton (29:33)
and forwards and upside down on this. But I d like

There’s a part, there’s a part of accessibility testing that is automated and can be automated. And I do think that there are a lot more of the WCAG success criteria, as we know and love them today, that can be automated that we were not able to previously automate before. How many? Don’t know. But I do think that there is going to be more and that the we will be able to do so utilizing.

AI and A and and similar types.

Mark Miller (30:11)
How

how many more since AI has shown up, how many more do we have automated than we did before?

Justin Stockton (30:20)
well that’s an interest well that’s the that’s the interesting thing because anyone that’s using AI, which is mm, mostly DQ and level, I think at this point, that are adding to their rules engine, they’re doing so behind closed doors. So I have no idea how far they’ve taken it. I know and DQ has said fifth fifty seven point three eight percent of their of the that that they think, but that number is

interestingly skewed. It’s based on the kind of the total number of things that can be found through an audit. And then of those things that are typically found through an audit, this is

Mark Miller (30:58)
Mm-hmm. Yeah.

Justin Stockton (31:01)
the percentage of those things that we can find through automation.

Mark Miller (31:03)
I

so that’s well and good and I’m not gonna question whether or not DQs

Or anybody for that matter is creating more tests. But we do need independent verification of that. And we need to come

Justin Stockton (31:16)
Yeah.

Mark Miller (31:16)
up with how we do the math, right? Because part of the problem is is that there’s been a lot of and I think Nic even talks about this in the article. And we know that there’s articles that have come out for previous automated checking prior to the whole AI thing that

played around with the numbers and made it seem like more was going on than what was really going on. And and that’s it’s not it’s not even that that’s wrong. It’s just a different way to look at it. That’s just a little bit more that that framing of it is little bit more appealing than other framing of it. And so I think we need to really understand it, we need a consistent measurement, consistent math on the measurement. But I’m I I will say this. This is so

Nick’s article rang true a little bit for me and and the reason is is because three or f three or four years ago at this point, I’m not gonna name names here, but major global organization coming to me saying, Manual testing is too slow for us. We are gonna go to a hundred percent automation because AI is gonna allow that to happen. We already know we know we there’s already a tool and we already know.

That we can do it.

And I said, no, you nope. I said, no, that’s not. This is three or four years ago. It’s not happened yet. They

Justin Stockton (32:40)
Wow.

Mark Miller (32:41)
that not only were they wrong about that moment, but four years later it’s not happening. And we I I think it’s it looks like it should work. But if you look at how quickly AI injected itself into so many other things, right? You want to make a PowerPoint, man? Prompt AI.

Boom, PowerPoint. So fast. You want to code an application, prompt AI, boom, application. You want there’s so many you want to do research that would have taken you a week.

Prompt AI and have it in minutes, right? It works. That stuff works. I mean, you have to hum it in the loop and be careful with it and all that kind of thing. But there’s no question that it’s but where is it with additional? It’s just, I haven’t seen it, Justin. And maybe it’s me, but I just haven’t seen it.

Justin Stockton (33:37)
It is

ever I there’s there’s aspects of i there’s so many ways there’s so many like threads they’re all coming together here. Yes, it’s a absolute I

Mark Miller (33:44)
I know we’re to speak speaking of rabbit holes, like we just whee, let’s jump into this one.

Justin Stockton (33:49)
was I was thinking if you watch the the Marvel series Loki, it’s like that where he reaches up and that at the very end I don’t w if you haven’t seen it, it’s been like three years. But there’s a part at the end where he reaches up at with the tree of life and he pulls in all the different pieces of the multiverse.

And it’s like all these threads coming together. Like that’s kind of what I was thinking about right in there.

Mark Miller (34:10)
That’s this, that’s this.

Justin Stockton (34:11)
But he the the the thing about AI that’s interesting is one, every every six to eight weeks, it gets incrementally better, like

Mark Miller (34:23)
Hundred percent.

Justin Stockton (34:24)
like way better than it was six to eight weeks before. Now that comes at huge costs, power environment, all these different things. But we haven’t seen a technology before.

that can is getting that much better that quick ever. Like we can’t even really keep up with it. Like as soon as you kind of learn how to like work with one AI model you know one week, you know, six weeks later, here’s a new version of it and it’s changed underneath the thing and it’s giving you better, different responses. So will we will we ever get there? Potentially. Will we kill the environment first?

Also potentially. I mean it’s but at the same time the thing that I want to get back to is that

Mark Miller (35:07)
Yeah, I I yeah. I I hear what you’re say yeah.

Justin Stockton (35:12)
there’s a part of there is a part of this article that I I do think we I think AI can certainly t address a number of the WCEG success criteria. What it can’t do, and I don’t know if it will be able to, well it will not be able to perform a usability test as if it was.

Mark Miller (35:30)
Replace the human experience.

Yeah. It’s it’s too

Justin Stockton (35:32)
I it won’t be able to replace the human experience. And I think those are two different things.

Mark Miller (35:37)
Yeah, and it’s kind of too smart for that, right? Like one of the points that Nic made was did the did the modal disappear too quickly and you couldn’t read it all? Or when you read it, did you understand it? You know, and how is how is a you know, that’s a human experience where you go, I didn’t that went by too

quick.

Justin Stockton (35:56)
But I but I read that. I’ve written enough skills for AI now that I read that and I was like, I could turn that into a prompt. Like Yeah.

Mark Miller (36:05)
Yeah. I mean, under certain parameters, you could say it’s a color contrast

ratio guidelines. You could say, look, you know, if this thing’s gotta stick around for this long

Justin Stockton (36:12)
yeah.

Mark Miller (36:12)
or whatever. I agree. But getting there in the first place. And I also think that you’re you know, your point about AI is a really good one. But what happens is that because about the r how rapid it changes, but because it changes so rapidly, that means that the way that we’re interfacing with that digital environment is also changing that rapidly. So

Justin Stockton (36:31)
Mm-hmm. Mm-hmm.

Mark Miller (36:33)
I wonder

‘Cause this what happens with the law, right? Like the law is always one step behind the technology, because the technology changes fast and law can’t keep up with it. if we’re gonna end up in that same sort of scenario where technology in general is changing so fast that the way that we interact with that environment, to your point, changes so fast that what needs to be done for accessibility changes so fast and it requires that human experience to understand the defects in the first place underneath

you know, in light of those changes. A human has to try then try to go use it and have the failure and you know, for whatever reason to change it. So I I wonder where it’s a little bit of a you know, kind of cat chasing its tail game there that that w where I have no idea how that’s gonna play out, you know?

Justin Stockton (37:25)
Yeah. And

I think it’s I I think it is too soon to tell. I think

Mark Miller (37:30)
Mm-hmm. I agree with that.

Justin Stockton (37:32)
are we going to end up, you know, is AI gonna lead us to Terminator or is it going to lead us to Star Trek? Like AI in both cases

Mark Miller (37:41)
Star Trek, please, Star Trek, please.

Justin Stockton (37:43)
I know, right? And hopefully it’s all narrated by Gene Roddenberry’s wife. Wouldn’t that be great? Marion Ros Roddenberry? Is that her name?

Mark Miller (37:52)
be great. I I don’t know. I don’t

know. I do want

Justin Stockton (37:56)
someone’s gonna cut.

Mark Miller (37:56)
Jean Luc Picard to be my captain.

Justin Stockton (38:00)
but just that idea of like maybe along this path, maybe the way in which we as human beings interact with computers evolves beyond a keyboard.

Mark Miller (38:13)
Yeah, that’s what I’m saying.

Yeah. it just w let’s go back

Justin Stockton (38:18)
Maybe the need for modal dialogues goes away.

Mark Miller (38:20)
to the Star Trek thing a little bit. There is the Borg, so let’s be a little careful about what we wish for there.

Justin Stockton (38:24)
There is the boring. That’s correct.

Mark Miller (38:26)
Lacutis, right? That was that was Picard’s

Justin Stockton (38:29)
Mm-hmm.

Mark Miller (38:31)
Borg. but so yeah, it’s all very interesting and I think it’s very difficult to unpack. great article by Nic though. I think it is

It the the point is is it makes you think, right? Like I don’t think you know, an article that comes to a conclusion you go,

Justin Stockton (38:45)
That’s the best part.

Mark Miller (38:47)
okay, for fine. You know, Nic really makes you think in this this article and that’s what I liked about it. And I’m sure he’s fine if you disagree with him a little bit.

Justin Stockton (38:58)
And I do think w we did maybe bury the lead on this article just a little bit in that

Mark Miller (39:03)
Okay.

Justin Stockton (39:04)
what he started off with was an announcement from the US web design system.

Mark Miller (39:10)
that’s right. Yeah.

Justin Stockton (39:10)
From well, excuse me, from the National Design Studio, which has taken over from ATF and has taken over the US web design system from ATF and similar since they shut down GSA. that the National Design Studio

Is going all in on AI and

Mark Miller (39:30)
Mm.

Justin Stockton (39:30)
with regards to accessibility. And so going back to your point, yes, three, four years ago, big multinational conglomerate said, We’re gonna go all in on this. Chances are if they’re gonna go all on in it now, also not gonna be successful. Will it will it be

Mark Miller (39:47)
Yeah.

Justin Stockton (39:48)
successful eighteen months from now? Maybe, thirty six, like five years, we don’t know. But yeah.

Mark Miller (39:54)
Yeah, all in seems dangerous. The and I think this

is the this is what it comes down to.

We in order to in order to work out this problem, you have to freeze time. And what I mean that by that is you have to say you can’t say, Will look let’s just let’s just boil it down, right? Will AI solve accessibility? Right? Plug in AI, accessible everything’s accessible, no problem. Will that happen? You can’t just say that, right? Because of what you said, because of how fast technology is evolving. You have to say, given

A website today, the way we code it and interact with it today, in the future will AI be able to make that accessible full stop? And the answer is like probably, right? Like go two hundred years into the future. And if somebody dug up an old, you know, Chrome browser and li websites and let’s apply our tech now and see if we can make this thing accessible. The answer’s probably yes. But

Justin Stockton (40:55)
Right. I also have a floppy disk.

Mark Miller (40:58)
floppy disk. Yeah, that’s good.

Yeah, I’m gonna keep my Chrome browser on a floppy disk. Mosaic. We go back to mosaic. but the but I I think that and and this isn’t really an ex article thing, but I th to me it’s a it’s the the pro the problem is understanding the problem. That that everything’s gonna move so fast that what we view as the problem today is just not going to be the same as the the the problem or the challenge tomorrow. I think that that’s

That’s the where we get in trouble with trying to think our way through this, particularly when we know ex AI is

Is creating a lot of really bad code. So the the the problem that we have on something as big and open is the internet is that it’s becoming less accessible because of how rapidly things are being developed in the internet now due to AI. and the so what you know, how is that gonna play into to solving these things in the future? Yeah. So, anyways.

Justin Stockton (42:06)
That’s it.

It is getting

Mark Miller (42:09)
But check it out, it’s a good article to make you think about these things and these are important things to think about.

How’d we do? Good? You happy?

Justin Stockton (42:20)
I think so. I don’t know. I feel like I’m I feel like there’s gonna be a lot of comments on this one.

Mark Miller (42:24)
Well, I think

yeah, and I know you’ve got I mean, the AI one is hard and I know you’ve got a lot of thoughts and a lot of

Justin Stockton (42:28)
Yeah.

Mark Miller (42:29)
but

Justin Stockton (42:30)
It’s interesting to see where people people are very passionate about AI, people and people are also very passionate about accessibility. And and so

Mark Miller (42:40)
Yeah, and please do comment. Justin and I are not pretending at all that we’ve got this stuff figured out. We’re just sharing our brains with you. So if you guys have

Justin Stockton (42:47)
I’m just okay.

Mark Miller (42:47)
thoughts on this, throw a comment. YouTube’s a great place to comment. if you if you go like, subscribe our channel, and comment on some of the videos. There’s been a few little discussions pop up there. So go for it. All right. I think we did it.

Justin Stockton (43:05)
Yeah, actually I would say that if using the Dunning Kruger thing from Craig’s thing, I’m with

Mark Miller (43:13)
Mm-hmm.

Justin Stockton (43:14)
a with regards to AI, I’m probably at number three, enough knowledge to realize the gaps, but not enough to reach

Mark Miller (43:21)
Look at you both bringing it around full circle. I’m so proud. I’m so proud right now.

Justin Stockton (43:23)
I feel like that’s about where

I am. Like, but I’m trying I’m I’m trying to stay open. I wanna know enough so that I can know so I can move on to stage four, which is enough knowledge to doubt everything.

Mark Miller (43:38)
Wh wh what what number am I where I just I don’t know anything, so I just assume I’m wrong. I guess that’s

Justin Stockton (43:45)
let’s see.

Mark Miller (43:46)
not Dunning Krueger. All right. Thank you. And thank you all for listening. And now that we have broken down accessibility for you, we hope you go forth and keep it accessible.