Meeting minutes
Review agenda and next meeting dates
Matt_King: Requests for changes to agenda?
lola: I'd like to add an item regarding my work adding tests for HTML "button"
Matt_King: Sure
Matt_King: Any other requests?
Matt_King: Hearing none
Matt_King: Next CG meeting: Wednesday August 26
Current interop reporting status
Matt_King: In the last two weeks, there have been a couple changes
Matt_King: We advanced the minimal data grid
Matt_King: It's now in the "candidate review" phase, bringing us from 22 to 23 test plans in that phase
Matt_King: I have completed the draft of the "tri-state checkbox" plan, and it's now ready for testing
Tri-State Checkbox Test Plan
Matt_King: In preparing this test plan, I discovered some more bugs that I will raise issues for
Matt_King: ...but this plan is ready for testing
Matt_King: Last night, the VoiceOver bot wasn't running.
Matt_King: I don't know why the bot isn't running at all
Matt_King: Maybe we should start with JAWS and NVDA
chrisc: The virtual machine might just need to be kicked. I can take a look later today
Matt_King: Thank you!
Matt_King: I put in JAWS 2025 and had the bot run that (because we don't have a JAWS 2026 bot, yet, and I didn't want to make people run the tests totally manually)
Matt_King: It's possible that if someone has already upgraded to 2026 that it doesn't take JAWS 2025 off your machine. You can have one version of JAWS from each release year on your machine simultaneously
Matt_King: Is anyone interested in taking up this testing?
Joe_Humbert: Sure
amruth: I can test, as well
Matt_King: Great. I can start another bot run and then assign it to you when it is complete
Matt_King: I also need to set you up in the system and send you some training information for running tests
elizabeth: I can test, as well
App issue 1666: Can't advance tabs test plan
Matt_King: I think folks need to build the app locally and then get the app into a state that they can reproduce the issue
chrisc: That sounds right
Matt_King: elizabeth, if you need help with that first step, we can arrange a meeting
elizabeth: That sounds good. I'll send you a message on Slack, chrisc
chrisc: Great, I should have availability next week
elizabeth: I'll make an attempt on my own in the mean time
Matt_King: Thank you both
PR 1399
<Matt_King> github: w3c-cg/
<lola> w3c-cg/
<lola> Github: w3c-cg/
lola: This pull request is an experiment of sorts
lola: It's my first time writing tests
lola: I'm trying to write tests for the "button" element as specified in HTML-AAM
lola: There is one set of tests for active buttons and one set for buttons that are disabled
lola: There aren't any existing tests for a disabled button, so I'm not sure whether what I've got in the data files (the ones for the screen reader commands) is correct
lola: So I'd like to draw my reviewers' attention to that aspect specifically
lola: Also, I'd appreciate it if anyone can point me to places where I can reuse tests if possible
Matt_King: Thank you, lola, this is awesome!
Matt_King: The only way we can move forward is with these experiments
Matt_King: One practical thing: it would be really helpful if this was in a branch of the repository instead of a fork (if you don't mind)
lola: Sure
Matt_King: The only thing I'm wondering about as your describing this is: testing two different things in the same test plan would be tricky. I think an active button and a disabled button each need their own dedicated test plan
Matt_King: Then we can continue the model of inserting links right before the element of concern and right after the element of concern
Matt_King: And if these were separate test plans, you would have two different reference files
lola: That makes sense
Matt_King: I think copying the test plan for the "command button" should work
lola: I don't think I made that many changes; it should be exactly the same
lola: I'm struggling to remember because the disabled button is amongst that, but once I split them up, then I think "command button" and "button" should be pretty much the same
Matt_King: It could be exactly the same test plan but with some navigation commands removed
Matt_King: The only navigation commands that might work would be the arrow-key commands
Matt_King: "j" in VoiceOver ignores a disabled button, I'm pretty sure
Matt_King: I'm positive that "f" and "v" do in JAWS and NVDA
Matt_King: you might want to start with the "active button" test plan, exactly copy it for the "disabled button" test plan, and then just remove commands for "disabled button"
Matt_King: I'm curious if you could share a little bit about your learning experience with the group
Matt_King: This is what I had in mind for the fifth agenda item in today's meeting
Matt_King: I want to remove barriers for people doing what you have boldly gone out and done
<Joe_Humbert> Can you write a guide?
lola: Sure. I had some questions
lola: There is some generation that happens which isn't immediately clear in the wiki
lola: In the "tests" folder and choose accordion
lola: In the reference for each test plan, there is one HTML file that you self-author. That's just to put the element in the pattern where it should be
Matt_King: Right, you write one test case
lola: It wasn't clear to me that the other HTML files are generated
lola: I inferred that eventually, but I had questions about how they are generated. It was a bit confusing
lola: Another challenge was knowing what commands to write
lola: Because I lifted from an existing test plan, it wasn't so big of a problem. But if I had to write the tests from scratch, I would not have known where to start
lola: Especially if you're not someone who users a screen reader in your day-to-day
lola: Otherwise, for me, it was a lot of reading documentation and trial-and-error
lola: The wiki page called "v2 format" was helpful
Matt_King: That's up-to-date. It's a kind of spec for tests. I wrote that. If you ever find an error there (or something that is unclear), I would certainly like to help improve that
Matt_King: I'm curious, for someone who is new and might be interested in writing a test plan, how can we take "baby steps" in terms of helping people be able to do that?
Matt_King: The most complicated thing in all of this, I think, is being familiar with GitHub and how GitHub works
Matt_King: I feel like once you're past that, then if you're familiar with running tests, then most of the rest is about learning to write the CSV files
lola: I think it depends on what you mean by using GitHub. Even if you're familiar with using GitHub for a continuous-integration workflow, there is a bigger hurdle about modifying the files themselves. Mainly because, as I said, you really have to read the CSV files (the ones that already exist) and get your head around what they are doing
lola: I think we should point to JAWS, NVDA, and VoiceOver documentation (where available) from the wiki
lola: Like about understanding what the available modes are, what they mean, and what you can do in each one
lola: When I'm talking with other people, they have no understanding of the existing of different modes, etc. I think having an understanding of how the screen reader works and also about how that relates to the contents of the CSV files as a bigger hurdle
Matt_King: I think this group is generally people who have familiarity with screen readers and how they work
Matt_King: But I think you're right: you can learn it over time, but it becomes mimicry if you're not a daily screen reader users or tester
ChrisCuellar: I was thinking about this, too. It's been a while for me. One immediate thought I had, while thinking about aam mapping: should we have "explainer" style documentation with tables that document similar commands between screen readers
ChrisCuellar: On another level, I was present for Matt_King's talk at TPAC, and I was wondering whether it might be helpful to have a human-readable document to describe a test plan. You have to do a lot of mental "joins" to understand the relationship between the various CSV files that comprise a test plan
ChrisCuellar: Even just to produce something as an artifact for test-plan writers to understand what a test plan really is at a high level
Matt_King: I guess starting with some kind of document... I've been hesitating to invest a lot of time in this, partly because it feels like I don't have a really solid understanding about what is hardest for a newcomer to piece together.
Matt_King: Like, when you look at a test report, that test report reflects the test plan
Matt_King: When you look at the test report, how understandable are the details? What parts are difficult to piece together?
Matt_King: Maybe we start there because those are a reflection of the plan
Matt_King: How apparent is the structure from the test report? How easy is it to decipher the anatomy of the plan?
lola: ARIA-AT is a completely different way of writing tests than any other work in my career. I don't want to make it seem that it's a completely alien thing; it could just be due to my personal experience
lola: I'm used to WPT or JavaScript or Ruby testing. I may just not have enough experience in this space
lola: ...but I would say, in general, I think there is a high barrier of entry to understanding the reports. And also, once you understand the reports, there's a barrier in being able to communicate their meaning to others.
lola: I think we could iterate to make those things easier. I don't think it's a case to refactor everything in one go
lola: And as Joe_Humbert mentioned, if the tests are easier to write, then it will be easier to onboard more people. The same goes for the reports
Matt_King: My impression is that the primary difference between this testing and other testing (in, for example WPT), is the complexity of the test cases
Matt_King: I wrote one test plan just for "aria-required on a text input". I think that is more like a WPT test in that it is such a narrow test case
Matt_King: Even still, I think that when we're talking about screen-reader interoperability, I think a big lesson we've learned in the past eight years as we've refined the testing approach, screen reader interoperability means that you can do certain things with each element. Broadly: you can navigate to it, you can get information about it, and you can operate it
Matt_King: There are different levels of complexity depending on the widget
Matt_King: So, saying that "you can interact with a menu", a menu isn't atomic--it's a collection. You get into tricky things from there, positions and set set, etc.
Matt_King: You could have different individual test cases for each detail, which would be more like WPT but harder for people to understand. Or you could write one encompassing test plan
Matt_King: That's how we ended up where we are
lola: That makes sense. A browser is not a screen reader, so even though WPT is doing similar things in terms of interacting with browsers, I think it's simpler than interacting with screen readers
lola: But I still do think that, while we may not need to communicate all of that to new testers, we should continue to brainstorm to better understand the barrier of entry
lola: I think ChrisCuellar's suggestion is a good first step. A human-readable overview would have been super-helpful to me
Matt_King: Well right now, other than me, you are the only test-writer. Scaling this project is a challenge! Especially scaling without the funding for an appropriately-size design and engineering team
lola: Are we planning anything at TPAC?
Matt_King: I'm still working out the details for my participation, but I'm reasonably confident that I will be there
lola: If you are, then I think doing some kind of "write ARIA-AT tests" hackathon for an hour or so could be beneficial
Matt_King: I would love to do that kind of thing! If you want to help, that would be awesome
lola: I will be at TPAC, and I think Cynthia will be, as well
ChrisCuellar: I just checked in on the VoiceOver bot. The machine was down, and I couldn't get it back online. That has never happened before, so I've filed a ticket with MacStadium. Hopefully, we will get this cleared up soon!