In February 2026 WebAIM ran automated tests against the top one million home pages and found detectable WCAG 2 failures on 95.9% of them, at an average of 56.1 errors per page — both worse than the year before. Those are only the failures a scanner can see. If your European Accessibility Act deadline is behind you and a testing budget request is in front of you, the useful question is not whether to test. It is which method finds which defect, what each one structurally cannot find, and what you are still blind to after paying. This guide compares four methods on coverage, cost and evidentiary value, routes defect classes to methods, and sets the gates we use between layers.
The four methods, and the fifth thing that is not one
Accessibility testing is not one line item. It is four activities with four different failure modes:
- Automated scanning: a rules engine (axe-core, IBM Equal Access, Lighthouse, or our own free WCAG audit tool) walks the rendered DOM and flags violations it can decide programmatically.
- Manual expert audit: a practitioner works through WCAG 2.2 Level AA, or the clauses of EN 301 549, against a sampled set of pages and states.
- Assistive-technology testing: the interface is driven with a screen reader, keyboard only, magnification and reflow, to see what is actually announced and reachable.
- Participant usability testing: disabled users attempt real tasks and you measure completion, not conformance.
An accessibility overlay or widget is not on that list: it is a runtime script that modifies the page for the visitor, not a way of finding defects. In April 2025 the US Federal Trade Commission approved a final order requiring one overlay vendor to pay $1 million over claims the FTC alleged were unsubstantiated. Its product, the complaint said, "did not make all user websites compliant with WCAG".
What each method actually detects
| Method | Reliably finds | Structurally cannot find | Cost shape | Evidence value |
|---|---|---|---|---|
| Automated scanning | Contrast ratios, missing alt attributes, unlabelled inputs, empty links and buttons, ARIA rule violations | Whether alt text is correct, focus order, whether an error message is understandable, whether a custom widget behaves as its role promises | Near-zero per run; CI wiring | Low alone. W3C WAI: tools "can not determine accessibility, they can only assist in doing so" |
| Manual expert audit | Judgement criteria: alt adequacy, heading and landmark structure, focus order, error identification, reflow at 320 CSS px, accessible authentication | Behaviour specific to one screen reader and browser pairing | Day-rate; scales with templates | High: what an accessibility statement or a regulator's file rests on |
| Assistive-technology testing | Name, role and value on custom components, status message announcement, keyboard traps, reading order after DOM insertion | Anything outside the AT combinations you chose; cognitive load | Day-rate plus screen reader licences | High for defects a developer can act on |
| Participant usability testing | Task completion, abandonment, the gap between conformant and usable | Criterion coverage: a sample, not an audit | Highest per finding | Moderate for conformance, highest for prioritisation |
Route the defect to the cheapest method that finds it
Read it in both directions: if a defect class has no method beside it in your plan, you are not covering it.
| Defect class | Cheapest method that finds it | Why |
|---|---|---|
| Low contrast text (83.9% of home pages in 2026); missing alt (53.1%), form labels (51%), empty links (46.3%) | Automated | Computable from colour values or attribute presence |
| Alt text that exists but describes the wrong thing; decorative images given descriptions | Manual audit | Needs to know what the image is for in context |
| Focus order and visible focus across a multi-step checkout; reflow at 320 CSS px and 400% zoom | Manual audit, keyboard only | Depends on sequence and layout, not markup alone |
| Name, role and value on a custom combobox or date picker; status messages for cart updates and async filters (WCAG 4.1.3) | AT testing | Rules engines check attribute validity, not announced behaviour |
| Error identification on a declined payment; accessible authentication (WCAG 3.3.8) such as puzzle CAPTCHA | Manual audit in the failure state | The state must be forced; the failure is cognitive, not markup |
| Whether a blind customer can complete checkout unaided | Participant study | No conformance check answers this |
Why the "57%" figure does not mean what procurement thinks
Vendors often cite a 2021 Deque study reporting that automation covered 57% of accessibility issues, against the long-standing 20–30% estimate. Both can be true, because they measure different things. The 20–30% figure is the share of WCAG success criteria testable programmatically. Deque's 57% is the share of total issue volume across about 13,000 pages. Because a few error types dominate by frequency, counting issues inflates apparent coverage. It is also a vendor study of the vendor's own engine.
The practical reading: automation clears most of your ticket count and almost none of your criterion risk. A clean axe report is a precondition for an audit, never a substitute for one.
Build the mix: four layers with explicit gates
- Baseline scan across templates, not pages. Enumerate distinct templates (home, category, PDP, cart, each checkout step, account, search, error pages) and scan one instance of each. Gate: zero WCAG 2.2 AA violations on every template before any manual time is bought. Paying a practitioner to rediscover missing form labels is how accessibility budgets get wasted.
- Manual audit on a sampled state set. Sample 12–20 pages: every template, plus each state that changes the DOM: invalid coupon, declined payment, out-of-stock variant, logged out and logged in. Gate: every finding carries a criterion number, a reproduction path, a severity and a suggested fix.
- AT pass on the top three revenue flows. Fix the matrix in writing: NVDA with Firefox, JAWS with Chrome, VoiceOver on iOS Safari, keyboard-only on desktop. Gate: each flow completed end to end on each combination, with the point of failure recorded.
- Participant study once a year. Five to eight participants per disability group you serve, on the same flows. Gate: at least one finding none of the layers above produced. If there is none, the tasks were too easy.
Cadence: layer 1 every build, layer 2 per release train and after any checkout or form change, layer 3 quarterly, layer 4 annually. [INTERNAL DATA NEEDED: share of findings in Switas manual audits that the automated layer had already flagged, by vertical — the number that makes the gate in step 1 persuasive.]
What this does not cover, and where it breaks down
- Web only. EN 301 549 also covers non-web documents (clause 10) and native apps (clause 11). This mix says nothing about your invoice PDFs or your mobile app.
- Conformance is not usability. A site can pass every criterion and still be slow to operate with a screen reader. In WebAIM's July–August 2026 survey of 1,780 screen reader users, 22.9% said web content had become less accessible over the previous year.
- The AT matrix is a choice, not coverage. Four combinations means accepting blindness to the rest; write down what you excluded.
- Standards move under you. WCAG 2.2 has been the W3C Recommendation since October 2023. EN 301 549 v3.2.1 (WCAG 2.1 AA) is the version cited in the Official Journal; ETSI published v4.1.1 (WCAG 2.2 AA) in September 2026, and it confers no presumption of conformity until cited there. Audit against v4.1.1 anyway.
- Not legal advice. EAA enforcement sits with national market surveillance authorities and varies by Member State.
Reading this from Turkey and the region
Directive (EU) 2019/882 applies to listed products placed on the EU market and services "provided to consumers after 28 June 2025", and its service list explicitly includes e-commerce services. The obligation follows the market, not the head office: a Turkish retailer, airline or travel platform selling to EU consumers is in scope, and so is the agency building the interface. Turkey's 2025 accessibility circular pushed public-sector sites toward the same technical baseline, so one audit against EN 301 549 and WCAG 2.2 AA can usually serve both. The regional difference is the AT matrix: screen reader and browser shares differ from Western European baselines, so derive yours from your own data. [INTERNAL DATA NEEDED: assistive technology and browser distribution across Switas audits of Turkish e-commerce and public-sector properties, to replace the imported default matrix.]
FAQ
Can an automated scanner tell me whether I am WCAG compliant?
No. W3C WAI states that evaluation tools cannot determine accessibility and can only assist in doing so. A scanner can prove some failures exist; it cannot prove none do.
If automation finds 57% of issues, why pay for an audit?
That figure counts issue volume, not success criteria, and a few high-frequency error types dominate the count. The criteria carrying legal and commercial risk (focus order, error handling, custom component semantics, authentication) sit mostly in the remainder.
Does installing an accessibility widget reduce our legal exposure?
The evidence says no. UsableNet counted more than 5,000 US digital accessibility lawsuits in 2025 and concluded that widgets "do not materially reduce legal risk". The FTC's 2025 order bars one vendor from claiming its automated product can make a site WCAG compliant without evidence.
Which screen reader and browser pairs should we test?
Start with NVDA plus Firefox, JAWS plus Chrome, VoiceOver on iOS Safari, and keyboard-only on desktop, then adjust from your own analytics. Write the matrix down so the exclusions are explicit.
Should we audit against EN 301 549 v3.2.1 or v4.1.1?
v3.2.1 is the version currently cited in the Official Journal, so a presumption of conformity rests on it today. Audit against v4.1.1 and WCAG 2.2 AA regardless, because those requirements are work you will owe once it is cited.
Does the European Accessibility Act reach a company outside the EU?
It applies to products placed on the EU market and services provided to consumers in the EU after 28 June 2025, including e-commerce services. Where your customers are decides it, not where you are incorporated.
Run this on your own checkout
Run layer one yourself with our free WCAG accessibility audit tool on your checkout templates, or have us run all four layers, with criterion-level findings and the AT matrix written down so they survive your next release. Talk to Switas about an accessibility audit.
Sources
- WebAIM Million 2026: accessibility of the top 1,000,000 home pages (data: February 2026)
- W3C WAI: Selecting Web Accessibility Evaluation Tools
- W3C WAI: WCAG 2 Overview (WCAG 2.2, 5 October 2023)
- Directive (EU) 2019/882 — European Accessibility Act
- Commission Implementing Decision (EU) 2021/1339 (citing EN 301 549 v3.2.1)
- Level Access: EN 301 549 v4.1.1 — what changed
- FTC: Final order requiring accessiBe to pay $1 million (22 April 2025)
- FTC: Order over deceptive accessibility claims (January 2025)
- Deque: Automated testing identifies 57% of accessibility issues (2021)
- UsableNet: ADA web lawsuit trends — 2025 filings
- WebAIM: Screen Reader User Survey #11 (July–August 2026)
- Pekin & Pekin: Turkey introduces accessibility obligations for websites and mobile applications (June 2025)
