finance.gov.au

Contact and help

The Australian Government's study into the Accessibility of the Portable Document Format for people with a disability

Phase two: technical evaluation – the technical perspective

Focus group participants and contributors to the public consultation, particularly those who are blind or have low vision, noted improvements in recent versions of their assistive technology for the Portable Document Format. To confirm these assertions, research was conducted to:

  1. Identify the most common assistive technologies used by people who are blind of have low vision in Australia;
  2. Ascertain the level of technical capability provided by the most common assistive technologies when interacting with the Portable Document Format, as claimed by vendors; and
  3. Test the technical conformance of PDF against WCAG 2.0 Success Criteria in commonly used assistive technologies to determine the level of technical capability for the Portable Document Format.

Common assistive technologies used in Australia

A series of telephone and email interviews with disability organisations assisted in establishing the most common assistive technologies used by people in Australia who are blind or have low vision. The results are represented in Table 1: Common assistive technologies in Australia. The table indicates the assistive technology version people are using together with the estimated total number of users, the current purchase cost (without software maintenance agreements) and upgrade costs where a user is upgrading from a previous version. The screen reader SATOGO is free for users but does require a constant internet connection when used. VoiceOver is also free as it comes preinstalled on Apple Macintosh computers.

Table 1: Common assistive technologies in Australia

Assistive Strategies AT Device Most common versions Total number of users Purchase cost Upgrade cost
Braille Notetaker BrailleNote NA 200 $8890.00 $350.00
PAC Mate NA Less than 200 $8779.00 $350.00
Screen Reader JAWS 8-11 3,000 $1502.00 $334.00
Window-Eyes 7 300 $1100.00 $315.00
SATOGO NA 100 Free NA
VoiceOver NA 100 Free NA
NVDA NA 50 Free NA
Screen Magnifier ZoomText 8 & 9 4,000 $591.00 $160.00
MAGic 9.5 - 11 750 $469.00 $262.00

The research found JAWS is the most commonly used screen reader, with an estimated 3,000 users, and ZoomText is the most commonly used screen magnifier with an estimated 4,000 users.

The number of users for each assistive technology is based on data provided by assistive technology vendors or service provider organisations. Of particular note, there is no data about the number of people who might be using older versions of assistive technologies. Feedback from Vision Australia clients indicates that some people continue to use earlier versions due to the prohibitive cost of upgrading their assistive technology. With each new release, assistive technologies generally provide more support for different applications and formats, particularly new formats or features that did not exist for previous versions. If people are unable to upgrade to the latest versions of their assistive technology, then successful interaction with technologies such as PDF files will be impeded.

The reported number of assistive technology users in Australia is relatively low compared to the number of people who are blind or have low vision (480,000). The figure fails to reflect the number of people in Australia who would benefit from using an assistive technology, but currently do not have access due to a lack of awareness, support or finance. It should be noted that the existence of relatively low numbers of assistive technology users does not provide any justification for inaction when providing accessible information.

The usage of portable Braille Notetakers (BrailleNote and PAC Mate) is relatively low. The high cost of these devices places them out of the financial reach of many users. These devices are more widely adopted by people in employment with access to equipment funding schemes.

In addition to the Australian findings, a correlation of the January 2009 WebAIM Screen Reader Survey12indicates that JAWS was also the most widely-used screen reader internationally. A significant proportion of the survey base, 74% of 1121 respondents, indicated they use JAWS.

The emergence of several free or low-cost screen reader options (NVDA, VoiceOver and SATOGO) is starting to penetrate the Australian market. These technologies currently have low uptake (in Australia), but their numbers are growing and this is expected to continue.

Vendor support for assistive technologies

To ascertain the level of technical capability between common assistive technologies and PDF files, assistive technology vendors (or their Australian resellers) were asked to provide advice on support for PDFs through their assistive technology device, including known issues and plans for future development.

The responses from the vendors and resellers were substantiated by a review of the documentation (i.e. vendor release notes) that accompanied the assistive technologies, where available.

Table 2: Responses from assistive technology vendors and resellers provides a summary of vendor statements. Noting that Freedom Scientific directed all correspondence regarding their products (i.e. PAC Mate, JAWS, Magic) in relation to this Study to the Australian reseller – Quantum Technology.

Table 2: Responses from assistive technology vendors and resellers

Adaptive Strategies AT Device Vendor/ Reseller Support introduced Current level of support Known issues Future developments
Braille Notetaker BrailleNote HumanWare No support provided None NA No plans to introduce support for PDF
PAC Mate Quantum Technology v 6.2 Requires separate PDF
reader, but "should experience good results"
Document is read one page at a time Resolve continuous reading issue. Future development will be upgraded in line with improvements in JAWS support
Screen Reader JAWS Quantum Technology v 8 Equivalent to HTML PDF security restricts access to some documents None
NVDA NV-Access v 2009.1 In line with HTML Around access to encrypted documents Ongoing, but nothing specific
SATOGO Serotek v 3.0 In line with HTML and Word Cannot navigate by headings No plans to address heading issue or provide further support
VoiceOver Apple Macintosh OS 10.5 Preview is a general file reader and does not provide specific support for PDF Structured tags not recognised None
Window-Eyes GW Micro v 4.5 Provides most of the functionality available for HTML Cannot read or navigate via headings Resolve heading issue
Screen Magnifier ZoomText Quantum Technology v 9.5 Equivalent to HTML None None
MAGic Ai Squared v 8 Functionality same as for HTML and Word Problems loading some types of PDF files with ZoomText running Working with Adobe to rectify loading issues

Based on statements provided by the vendors, the following assistive technologies provide the theoretical technical capability to support PDF files:

Overall, vendors indicate that support has generally been introduced in more recent versions. Therefore, versions that predate these are unlikely to provide sufficient technical capability for PDF files. Only JAWS, NVDA, SATOGO and MAGic claimed to provide an experience equivalent to HTML. The vendors of Window-Eyes and ZoomText claim to provide functionality largely in line with other formats, but did not state whether the level of support was equivalent to these other formats.

The interviewees reported that assistive technology development is guided by emerging technologies and industry trends, and the core focus for research and development is now moving towards Web 2.0, ARIA and HTML 5 (responses provided by vendors of SATOGO and Window-Eyes). Most vendors of assistive technologies common in Australia (BrailleNote, PAC Mate, JAWS, Window-Eyes and MAGic) felt that the responsibility for PDF accessibility now lies with document authors. Vendors felt their current level of support is sufficient and authors need to create more accessible PDF files before the vendors see further benefit in providing increased PDF support to end-users.

Adobe Reader is the most commonly-used PDF reader, and developments on the application programming interface (API) have enabled better support for assistive technologies. As such, all but one of the vendors stated their support for PDF was geared towards the latest version of Adobe Reader. The one exception was Apple Macintosh (VoiceOver) – Apple directs its PDF support into its own PDF reader (Preview). Vendors proposed the view that in order to offer equal support and functionality for other PDF readers, the developers of other PDF readers would need to provide the same level of support (as Adobe Reader) to the assistive technology first.

Technical testing

The aim of the technical evaluations (phase two of the Study) was to establish the level of technical capability of the Portable Document Format to conform to WCAG 2.0, and its interoperability with the common assistive technologies identified earlier. A secondary aim of the Study was to assess claims for 'Accessibility Support' in relation to the PDF technology.

Adobe Test Suite

In 2008, prior to WCAG 2.0 being released as a Web Standard, the W3C published Reports on Accessibility Support for Ways of Using Various Web Technologies. It included a report from Adobe to demonstrate how PDF files might address WCAG 2.0 Success Criteria. The report does not prove WCAG 2.0 conformance, nor suggest that the Portable Document Format is endorsed by the W3C as an 'Accessibility Supported' technology. The report developed by Adobe, Accessibility Support Documentation for PDF, formed part of this Study and helped progress WCAG 2.0 to the next stage in the W3C web standard development process.

The Accessibility Support Documentation for PDF report and associated testing from Adobe did not include a complete collection of commonly used assistive technologies used by people who are blind or have low vision, as identified in phase two of this Study.

The Adobe Test Suite originally evaluated three assistive technologies and user agent (web browser) configurations against limited, self-assessed WCAG 2.0 Success Criteria. For this Study, Vision Australia invited Adobe to extend its testing, applying the same methodology (as applied in the Adobe Test Suite) to include the most common assistive technologies (or versions) that Vision Australia identified as potentially providing sufficient support, based on the vendor claims. Table 3: Assistive technology testing requirement outlines the assistive technologies tested.

Exclusions

The BrailleNote and PAC Mate devices were not tested in this Study. BrailleNote does not support PDF. The PAC Mate vendor (Quantum Technology) claimed that PAC Mate uses JAWS speech and provides equivalent support apart from a known issue with continuous page reading.

Screen magnification software, such as ZoomText and Magic, does not interact with the structure of PDF files. For this reason, these products were also excluded from the testing. However, since the magnification component in ZoomText version 9 was included in the original Adobe report and ZoomText is the most commonly used assistive technology product in Australia, Adobe was invited to test the magnification component in ZoomText version 8 to confirm the original findings and vendor's claim of support since the earlier version. Further details about the methodology used in the Adobe Test Suite is contained in the Study approach and methodology section.

Table 3: Assistive technology testing requirement

Adaptive Strategies AT Device Version Not required Previously tested Testing required Comments
Braille Notetaker BrailleNote N/A x     Vendor claims no support for PDF
PAC Mate 6.5 x     Vendor claims PAC Mate uses JAWS speech and JAWS testing results will apply. Not independently tested
Screen Reader JAWS 8 - 11   x x JAWS 9 tested in 2008. Vendor stated no change in functionality since v9 so results applied to JAWS 10 &11. Vendor claim support including JAWS v.8, testing of earlier version is required
NVDA 2009.1     x Not previously tested, AT interacts with structure
SATOGO 3.0     x Not previously tested, AT interacts with structure
VoiceOver 10.5 & 10.6   x x OS 10.5 tested in 2009. Vendor claims improvements to VoiceOver in subsequent version. Testing of OSX 10.6 is required
Window-Eyes 7   x   Tested in 2008
Screen Magnifier ZoomText 9.5 - 11 x     Does not interact with structure testing not required
MAGic 8&9   x x Does not interact with structure. ZoomText 9 magnification tested in 2008. Vendor claims support since ZoomText v.8 testing of earlier version is required

Phase two – technical evaluation results

To summarise the data, three levels describe the technical capability of the assistive technologies with PDF files. These are:

  1. Sufficient: Provides technical capability that enables the assistive technology to interact with PDF files.
  2. Partially Sufficient: There are some technical capabilities using the assistive technology, but also potential issues that may impact upon the interaction with a PDF file.
  3. Not Sufficient: Provides inadequate technical capability for the assistive technology to interact with PDF files.

Table 4: Assistive technology support for PDF finding

Adaptive Strategies AT Device Version AdobeTest Suite Results Sufficient Partially sufficient Not sufficient Total number of Users
Braille Notetaker BrailleNote N/A N/A     x 200
PAC Mate 6.5 N/A   x   Under 200
Screen Reader JAWS 8 - 11 43/43 x     3,000
NVDA 2009.1 41/43   x   50
SATOGO 3.0 36/43   x   100
VoiceOver 10.5 & 10.6 9/43     x 100
Window-Eyes 7 36/43   x   300
Screen Magnifier ZoomText 9.5 - 11 NA x     750
MAGic 8&9 21/21* x     4,000

Notes: The total number of users refers to all versions of the assistive technology, not just the most commonly-used versions. Numbers are reflective of comments made by vendors and may not be truly indicative of the market.)

For ZoomText, 22 of the Adobe Test Cases were not applicable to the screen magnification component of the software as it does not interact with the structure of the PDF file.

The specific number of test results for each of the 43 tests applied is included in the Supplementary Report. Only the JAWS screen reader and the ZoomText screen magnifier (magnification component) met all of the applicable tests of the Adobe Test Suite (with support by vendor claims). Therefore the technical testing finds that the Portable Document Format provides 'sufficient' technical capability for JAWS and ZoomText only. When correlated with vendor claims and an examination of the functionality of MAGic, a 'sufficient' technical capability claim can also be applied to MAGic.

However, it is inappropriate to state that PDF files can be regarded as an 'Accessibility Supported' technology under the W3C definition, without qualification.

Only 33% of the most common assistive technologies used in Australia provide 'sufficient' technical capability (demonstrated through the Adobe Test Suite and vendor claim); it is estimated that these assistive technologies are used by 89% of the assistive technology user population. The remaining 66% provide 'partially sufficient' or 'not sufficient' technical capability; and it is estimated that these assistive technologies are used by 11% of the user population.

The Adobe Test Suite should not be considered proof of WCAG 2.0 conformance, although it does go some way in identifying attributes and features that improve accessibility between the technology and interoperability with assistive technologies.

An analysis of the specific test cases highlights a number of limitations, especially important to users of screen readers that were not covered in the Adobe Test Suite. For example, the presentations of text (bold, italicised), which is a component of WCAG 2.0 Success Criteria 1.3.1, is not included in the Adobe Test Suite and was therefore not tested. Adobe is currently increasing the scope of the test suite against WCAG 2.0 Success Criteria.

WCAG 2.0 conformance can be supported using Sufficient Techniques, however there are no available Sufficient Techniques for PDF at the time of writing (August 2010).

Summary qualification of the issues encountered with each assistive technology described below:

BrailleNote

BrailleNote does not provide support for PDF files, therefore BrailleNote was not tested.

PAC Mate

The PAC Mate reseller (Quantum Technology) claimed that PAC Mate uses JAWS speech and provides equivalent support apart from a known issue with continuous page reading. Therefore PAC Mate was not tested. NOTE: Findings from Phase 3 - user evaluations demonstrated significant technical problems using PDF files and therefore the overall technical capability of PAC Mate is deemed to be not sufficient for this Study.

JAWS 8-11

JAWS 8 and 9 satisfied all of the technical tests. The vendor stated that functionality for PDF files had not changed since version 9 and therefore versions 10 and 11 were not tested.

NVDA 2009.1

NVDA passed 41 out of the 43 test cases. NVDA failed to provide support for identifying the language of a document or changes to language within a document. NVDA currently does not provide this support for PDF files.

SATOGO 3.0

SATOGO provides insufficient support for the identification of a document's language, and a lack of identification when the language changes within a document. In relation to structural mark-up, the correct reading of tagged paragraphs is only partially supported by SATOGO, and headings, though identified, cannot be navigated via defined levels. SATOGO also failed to interact with review comments added to a PDF document.

VoiceOver 10.5 & 10.6

Structural mark-up and tags are not identified when using VoiceOver with Preview, leading to significant accessibility issues, including the inability of VoiceOver to read alternative descriptions for images or identify or allow navigation of headings and tables. VoiceOver was also unable to access and interact with form elements (e.g. checkboxes, edit fields) correctly, identify error messages or identify the language of the whole document or sections within it.

Window-Eyes 7

Window-Eyes failed to identify headings in a PDF document or provide support for heading navigation within a document. It failed to read the role or function of a nested list in a document, and identify the language of the whole document or sections within it.

ZoomText 8 & 9

ZoomText (magnification component) does not interact with structure. However, since ZoomText is the most commonly used screen magnifier in Australia, versions 8 and 9 were included in technical testing. ZoomText 8 and 9 (magnification component) satisfied all applicable technical tests.

MAGic

MAGic (magnification component) does not interact with structure. Therefore MAGic did not undergo technical testing. Vendor claims support since version 9.5.

Footnotes:

  1. WebAIM, 2009, Survey of Preferences of Screen Readers Users, Utah State University, viewed 10 March 2010, http://www.webaim.org/projects/screenreadersurvey/ External Site

 

Contact for information on this page: wcag2@finance.gov.au


Back to top

Last Modified: 22 June, 2011