finance.gov.au

Contact and help

The Australian Government's study into the Accessibility of the Portable Document Format for people with a disability - Supplementary Report

User evaluation test results by assistive technology

This section covers specific, detailed test results for each of the ATs used by participants in the user evaluations.

The discussion of issues that follow focuses on Collection A – the set of documents that were optimised for accessibility.

For Collection B – the set of representative documents – 74% of the issues experienced were the result of poor document design. In many cases, this prevented the participant from successfully completing the task. Therefore these individual issues are not discussed in detail as the document design issues overwhelmed the actual user experience. Full results of all ‘unique’ issues encountered for ATs against each task are reported in their individual section of this report.

For context, the issues encountered by the participants have been categorised into four groups:

  1. Document Design: the design of the document (e.g. missing tags or elements, problems with reading order etc) created barriers for the user, requiring use of an adaptive strategy or prevented them from interacting with the document at all.
  2. AT Support: the user’s AT did not provide sufficient functionality to enable the user to interact with the PDF file. In some cases this matched the findings from the technical evaluation; in others it highlighted new areas of technical incompatibility.
  3. User Skill: a lack of knowledge by the user about using their AT, Adobe Reader, or PDF files led to confusion and difficulty completing the task.
  4. Adobe Reader: features provided by the Adobe Reader did not support the user to interact with the document using their chosen adaptive strategy.

Further, the relative impact of each issue was rated based on the effect it had upon the participant and whether they were able to successfully complete the task:

Detailed test results by assistive technology

PAC Mate

The one PAC Mate participant was unable to undertake any of the twelve tasks due to technical problems using the PAC Mate and the Orneta PDF Reader Mobile – the PDF reader recommended by PAC Mate’s Australian distributor, Quantum Technology.

The Orneta reader failed to open any of the PDF documents used for the evaluations. It did open a pre–installed PAC Mate user guide from Freedom Scientific available on the PAC Mate. However the efficiency of this interaction was very poor and the PDF reader crashed repeatedly. The best experience was when the participant managed to move through the first page using the keyboards up and down arrows. All other reading and navigation commands failed to work. Based on this experience, no evaluations were conducted and the level of technical capability provided by PAC Mate was deemed to be not sufficient for this Study.

JAWS 9 & 10

Seven participants attempted the tasks using this assistive technology.

Success rates

Table 6: JAWS versions 9 & 10 test results

Collection Attempts Successes Failures
A 42 35 7
B 29 8 21

The JAWS (versions 9 and 10) users successfully completed 83% of the tasks they attempted with the documents in Collection A. Except in one case, all task failures were attributed to the task where users were asked to orientate by page number.

User Evaluation tests were conducted with users describing their experience with using their ATs as low, medium or high level skill. Participants of varying skill level were included – see Table 3: Participant profiles by disability group. There were no significant differences in the success rates (for tasks) between the different experience/skill levels of the participants using JAWS, nor in their acceptance of the time taken to complete the tasks.

However, there was an apparent variation across skill levels when users were asked to assign a level of ease to the task they had just completed. For Collection A, the participants with a high level of experience/skill found the tasks very easy overall. In contrast, the participants with moderate and low experience/skill levels acknowledged that there had been a learning curve as they worked their way through the tasks.

Specific issues encountered

Table 7: Issues encountered with Collection A using JAWS versions 9 & 10

Issue Cause Impact
Page number in document is not read aloud by JAWS Document design High
Page number of document does not correspond with
Adobe Page Navigation toolbar
Document design Moderate
Users expected table rows for 'item details' to be numbered Document design Low
'Page Down' does not act as user expects AT support & Document design Moderate
Cannot manually enter information into date field with
'auto forms mode' in JAWS 10 turned on
AT support Moderate
Cannot jump between paragraphs using JAWS AT support Low
User does not know JAWS table navigation commands User skill Moderate
User does not know how to navigate via headings User skill Moderate
User does not know how to search for a page using
Adobe Page Navigation toolbar
User skill Moderate
User tries to use Word commands User skill Moderate
User found it hard to use the drop down calendar picker User skill Low

JAWS users encountered three issues caused by a lack of AT support that were not identified in the technical testing:

  1. When JAWS 10 engaged with a PDF form with a drop down calendar picker in ‘auto forms mode’, the user was unable to input text into the field. To resolve the issue, the user had to exit out of ‘auto forms mode’, which they did successfully.
  2. All versions of JAWS tested do not support the ability to move through PDF documents via paragraph navigation. One JAWS user stated in the pre–evaluation interview, and during the evaluation, that they use paragraph navigation, but the lack of support did not significantly hinder their interaction or ability to complete the task.
  3. Users did not know what page they were on or where they had jumped to in a document after using the ‘Page Down’ feature (pressing the Page Down key on the keyboard). Participants expected this to move them to the next page of the document, however it only moves forward one ‘screen’ of information, which is often less than a full page. The document design and AT failed to work together in this instance to provide effective and consistent navigation.

Table 8: Full test result for user evaluation by task – JAWS 9 & 10

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
JAWS 9&10 A: Read a short document (general navigation) Cannot jump between paragraphs using JAWS Have to read all, sentence by sentence or navigate via headings JAWS does not support paragraph navigation AT support Low
JAWS 9&10 A: Navigate information in a table structure User does not know JAWS table navigation commands User has to read through each cell and guess/ discover correct cell Low experience of user meant they did not know JAWS table navigation commands User skill Moderate
JAWS 9&10 A: Access and understand information portrayed through an image/alt No issue
JAWS 9&10 A: Identify and move through a document using structure such as headings User does not know how to navigate via headings User believes tab key jumps between sections rather than links and controls Low experience of user meant they did not know basic JAWS navigation commands User skill Moderate
JAWS 9&10 A: Navigate through a large document using page numbers User does not know how to search for a page using Adobe Page Navigation toolbar User relied on page number in document being announced until observer informed him about the Navigation toolbar At least one participant from the Moderate and Low experience users did not know how to use the toolbar User skill Moderate
JAWS 9&10 A: Navigate through a large document using page numbers User tries to use Word commands User tried the Ctrl + G combination expecting it to activate the 'Go To' window but in PDF this is Ctrl + Shift + N Moderate and Low experience users are not aware of specific command when using PDF files User skill Moderate
JAWS 9&10 A: Navigate through a large document using page numbers Page number of document does not correspond with Adobe Page Navigation toolbar (Deafblind user) User confused as to what page they are on or if page exists Complexity of multi page (spread) print presentation Document design Moderate
JAWS 9&10 A: Navigate through a large document using page numbers Page number in document is not read aloud by JAWS User cannot establish what page they are on Author of document has not tagged and included the page number in the reading order. Author has used a spread design meant for printing not online viewing Document design High
JAWS 9&10 A: Navigate through a large document using page numbers 'Page Down' does not act as user expects Users do not know what page they are on, or where they have jumped to in document after using 'Page Down'. Shortcut does not take them to next screen or document page consistently Document design and AT do not work together for effective and consistent navigation AT support & Document design Moderate
JAWS 9&10 A: Interact and complete a form Cannot manually enter information into date field with 'auto forms mode' in JAWS 10 turned on Users have to turn off 'auto forms mode' in JAWS 10 to enter data into field. They did not know this initially This is a JAWS bug. When 'auto forms mode' is turned on the drop down calendar can be used but user has to arrow down to activate. Further, the field states' type in text', but provides no information regarding calendar widget AT support Moderate
JAWS 9&10 A: Interact and complete a form User expected table rows for 'item details' to be numbered Users are not sure how many rows are in the table Author of document has not provided a column for the row numbers or table summary Document design Low
JAWS 9&10 A: Interact and complete a form User found it hard to use the drop down calendar picker User was confused about how to use and exit the calendar picker at first Low experienced user did know how to use the calendar picker User skill Low
JAWS 9&10 B: Read a short document (general navigation) None of the contents of the first page are contained in the
structure tree
The text on the first page is not read by JAWS so users could not access information Author of document has not tagged and included the first page in the structure Document design High
JAWS 9&10 B: Read a short document (general navigation) Delay in loading of document A delay was experienced in the loading time of the document. User was confused and frustrated as to what was happening Author of document has not correctly tagged all of the document leading to conflicts with AT Document design Moderate
JAWS 9&10 B: Read a short document (general navigation) User does not know if document has loaded User not sure when or if document is loaded Moderate and low experience users did not know that JAWS automatically reads document when loaded or that when you press any reading key JAWS will announce "processing document" User skill Low
JAWS 9&10 B: Read a short document (general navigation) File freezes when moving between pages User has to wait for JAWS and Adobe Reader to commence interacting with each other. Author of document has not correctly tagged all of the document leading to conflicts with AT Document design Low
JAWS 9&10 B: Navigate information in a table structure Table has not been marked-up Header row of the table is read as one cell and users cannot navigate using JAWS table commands Author of document has not marked-up the table to identify header and data cells (<TH> & <TD>) Document design High
JAWS 9&10 B: Navigate information in a table structure Tables have not been identified so quick navigation is hindered User selects 'T' to jump to tables but JAWS announces "no tables found" so user believes there are none Author of document has not identified and marked up tables so JAWS short cut key cannot locate them Document design High
JAWS 9&10 B: Navigate information in a table structure User does not know how to locate tables User believes tab key will take them to the table on the page, but jumps to next link in document User has a low experience level with JAWS User skill Moderate
JAWS 9&10 B: Navigate information in a table structure Numerical data separated into 2 cells User was confused as numerical data is read as 2(space) 380 instead of '2380' Author of document has separated one number into 2 cells Document design Low
JAWS 9&10 B: Access and understand information portrayed through an image/alt The image has no alternative text The users cannot ascertain that an image is present Author of document has not applied an alternative text description to the image Document design High
JAWS 9&10 B: Identify and move through a document using structure such as headings Headings are not marked up correctly Cannot jump to headings so users have to read through all of the document Author of document not marked up headings consistently. <H2 Subhead> has been used instead of <H2>. JAWS does not recognise the <H2 Subhead> tag Document design Moderate
JAWS 9&10 B: Identify and move through a document using structure such as headings Headings are not marked up consistently Only one heading is marked-up correctly towards the end of the document, user jumps to that and reads to the end - confused as to where the information they are looking for is located Incorrect heading mark-up has significant impacts on the readability of content Document design Moderate
JAWS 9&10 B: Identify and move through a document using structure such as headings List items not grouped in document structure List is marked-up but read as two separate lists instead of one Author of document has separated the list in the reading order so the items are not associated Document design Low
JAWS 9&10 B: Navigate through a large document using page numbers Page number is not
read out or presented
on braille display
Users cannot verify what page they are on - have to base judgement on the Adobe Page Navigation toolbar Author of document has not tagged and included the page number in the reading order Document design High
JAWS 9&10 B: Navigate through a large document using page numbers User not know if document has loaded User not sure when or if document is loaded Moderate and low experience users did not know that JAWS automatically reads document when loaded or when you press a reading key JAWS will announce "processing document" User skill Low
JAWS 9&10 B: Interact and complete a form Focus automatically jumps to next form field without user control Users confused as upon entering information the
focus automatically moves
to the next field without
them tabbing
Author of document changes the focus to the next control without user operation Document design & User skill Low
JAWS 9&10 B: Interact and complete a form Form fields and labels
not associated
Users do not know what information to enter into
a field
Author of document not associated labels and form fields correctly Document design High

NVDA 2009.1

One participant attempted the tasks using this assistive technology.

Success rates

Table 9: NVDA 2009.1 test results

Collection Attempts Successes Failures
A 6 4 2
B 6 2 4

The NVDA 2009.1 participant attempted all of the tasks in both document sets (twelve tasks), and succeeded in six (50% success rate). The time taken to complete the tasks was deemed acceptable on all tasks and the level of ease by which this was achieved was also very high.

The specific tasks from Collection A that the participant failed were orientating by page number and navigating information in a table structure. As with the moderate and low experienced JAWS users, the NVDA 2009.1 participant experienced a learning curve when interacting with the PDF documents that helped him become more efficient as the tasks progressed.

Specific issues encountered

Table 10: Issues encountered with Collection A using NVDA 2009.1

Issue Cause Impact
Page number in document is not read aloud by NVDA 2009.1 Document design High
'Page Down' does not act as user expects AT support & Document design Moderate
NVDA 2009.1 does not provide a direct association between the header and data cells AT support High
Field name (postal code) is not read not out AT support Moderate
User does not know how to search for a page User skill Moderate

The NVDA 2009.1 participant experienced several issues caused by a lack of AT support that were not clearly identified in the technical testing:

  1. NVDA 2009.1 does not announce table header cell information as the user navigates between row and columns. This is common across all document formats, not just PDF. As a result, the participant had to establish the column headers and then try and recall the order of this information while moving through the rows of data.
  2. The user experienced an issue when trying to read out the ‘postal code’ field in the accessible form. In this case the file name of the document was read out instead of the label. The user had to move focus out of the field and then tab back into the field to hear the correct label. The cause of this issue is related to the combo box on the preceding field – an issue NVDA 2009.1 is aware of and working to correct.
  3. As with JAWS, the ‘Page Down’ functionality did not match the user’s expectations. The user did not know what page he was on, or where he had jumped to in document after pressing the ‘Page Down’ key.

When interacting with the documents, the experience for the NVDA 2009.1 user was degraded by the lack of core functionality provided by NVDA 2009.1 for PDF files. As a result, the user had to implement alternative strategies on multiple occasions to compensate. However, the user did not find this particularly troublesome and considered the ease and time to accomplish the tasks acceptable.

Table 11: Full test results for user evaluation by task - NVDA 2009.1

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
NVDA 2009.1 A: Read a short document (general navigation) No issues        
NVDA 2009.1 A: Navigate information in a table structure NVDA 2009.1 does not provide a direct association between the header and data cells User has to establish what each column header is then move through the rows and mentally recall the column header NVDA 2009.1 does not provide commands to read tables effectively AT support High
NVDA 2009.1 A: Access and understand information portrayed through an image/alt No issues
NVDA 2009.1 A: Identify and move through a document using structure such as headings No issues        
NVDA 2009.1 A: Navigate through a large document using page numbers 'Page Down' does not act as user expects User did not know what page he was on, or where he had jumped to in document after using 'Page Down'. Does not take him to next screen or document page consistently Document design and AT do not work together for effective and consistent navigation AT support & Document design Moderate
NVDA 2009.1 A: Navigate through a large document using page numbers Page number in document is not read aloud by NVDA 2009.1 Cannot establish what page they are on Author of document has not tagged and included the page number in the reading order. Author has used a spread design meant for printing not online viewing Document design High
NVDA 2009.1 A: Navigate through a large document using page numbers User does not know how to search for a page User relied on page number in document being announced until observer informed him about Adobe Page Navigation toolbar Experience with using PDF files was high, but user did know about Page Navigation toolbar User skill Moderate
NVDA 2009.1 A: Interact and complete a form Field name (postal code) is not read not out File name of document is read out when entering postal field. User has to tab to next field and then shift tab back into the postal code field for field name to be read out NVDA 2009.1 bug with combo boxes in PDF forms AT support Moderate
NVDA 2009.1 B: Read a short document (general navigation) None of the contents of the first page are contained in the structure tree The text on the first page is not read by NVDA 2009.1 so the users cannot access the information Author of document has not tagged and included the first page in the structure Document design High
NVDA 2009.1 B: Read a short document (general navigation) File freezes when moving between pages User has to wait for NVDA 2009.1 and Adobe Reader to reengage with the document Author of document has not correctly tagged all of the document leading to some conflicts with ATs Document design Low
NVDA 2009.1 B: Navigate information in a table structure Tables have not been identified so does not allow quick navigation to them The user could not jump to the next table using the shortcut ('T') as the table had not been tagged Author of document has not tagged and identified tables in the structure Document design Moderate
NVDA 2009.1 B: Navigate information in a table structure Table navigation not supported by document mark–up or by NVDA 2009.1 Users tab through each cells and has to mentally constructs table layout and association between cells NVDA 2009.1 does not provide table navigation, and author of document has not marked up the table to identify header and data cells (<TH> & <TD>) AT support & Document design High
NVDA 2009.1 B: Navigate information in a table structure Adobe Reader crashes when loading file File and reader have to be reopened Author of document has not tagged the document leading to conflicts with ATs Document design High
NVDA 2009.1 B: Access and understand information portrayed through an image/alt The image has no alternative text The users cannot ascertain that an image is present Author of document has not applied an alternative text description to the image Document design High
NVDA 2009.1 B: Identify and move through a document using structure such as headings Headings are not marked up correctly Cannot jump via heading navigation so have to read through all of the document Author of document not marked up headings consistently. <H2 Subhead> and <heading 2> have been used instead of <H2>. NVDA 2009.1 does not recognise either of these tags Document design Moderate
NVDA 2009.1 B: Identify and move through a document using structure such as headings List items not grouped in document structure Users confused by the appearance of the list as one list is read as two Author of document has separated the list in the reading order so the items are not associated Document design Low
NVDA 2009.1 B: Navigate through a large document using page numbers Adobe Reader crashes when loading reading order File and reader have to be reopened Author of document has not tagged the document leading to conflicts with ATs Document design High
NVDA 2009.1 B: Navigate through a large document using page numbers Page number in document is not read aloud by NVDA 2009.1 User bases judgement of page number on Adobe Page Navigation toolbar Author of document has not tagged and included the page number in the reading order Document design High
NVDA 2009.1 B: Interact and complete a form Form fields and labels not associated Users not know what information to enter into a field Author of document not associated labels and form fields correctly Document design High

SATOGO 3.0

One participant attempted the tasks using this assistive technology.

Success rates

Table 12: SATOGO 3.0 test results

Collection Attempts Successes Failures
A 6 4 2
B 6 1 5

The SATOGO 3.0 participant completed four of the six tasks using Collection A with ease. However, as with the other screen reader participants, the SATOGO 3.0 user struggled to complete the tasks associated with Collection B.

Specific issues encountered

Table 13: Issues encountered with Collection A using SATOGO 3.0

Issue Cause Impact
User expected table rows for 'item details' to be numbered Document design Low
SATOGO 3.0 was unable to load document AT support High
Edit field name not read aloud when arrowed to AT support Moderate

For the two tasks in Collection A that were not successfully completed, the participant experienced problems with their AT providing sufficient support for the PDF files:

  1. SATOGO 3.0 had an unresolvable conflict issue with the one document that failed to load. SATOGO 3.0 are aware of the issue and have replicated it in their lab in an attempt to resolve it.
  2. The ‘edit field names’ in the form were not read aloud when selected. The user did not know what the fields related to, but repeated tabbing between the fields enabled the label to be read.

The user experience was hindered by SATOGO 3.0’s conflict with two of the documents (similar conflict occurred for the Collection B documents as well). The user was unable to interact with these documents and therefore was unable to complete the associated tasks.

Table 14: Full test result for user evaluation by task – SATOGO 3.0

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
SATOGO 3.0 A: Read a short document (general navigation) No issues        
SATOGO 3.0 A: Navigate information in a table structure SATOGO 3.0 was unable to load document User could not access the file SATOGO 3.0 bug. SATOGO 3.0 have confirmed the issue and are looking to rectify it AT support High
SATOGO 3.0 A: Access and understand information portrayed through an image/alt No issues
SATOGO 3.0 A: Identify and move through a document using structure such as headings No issues        
SATOGO 3.0 A: Navigate through a large document using page numbers SATOGO 3.0 was unable to load document User could not access the file SATOGO 3.0 bug. SATOGO 3.0 have confirmed the issue and are looking to rectify it AT support High
SATOGO 3.0 A: Interact and complete a form Edit field name not read aloud when arrowed to User did not know what the field related to when arrowed to it, tabbed between the fields and then the label was read SATOGO 3.0 development issue AT support Moderate
SATOGO 3.0 A: Interact and complete a form User expected table rows for 'item details' to be numbered Table rows are not numbered so users not sure how many rows they have to skip Author of document has not provided a column for the row numbers or table summary Document design Low
SATOGO 3.0 B: Read a short document (general navigation) SATOGO 3.0 was unable to load document User could not access the file Author of document has not tagged the document correctly leading to conflicts with ATs. SATOGO 3.0 have confirmed the issue and are looking to see if they need to rectify anything at their end AT support & Document design High
SATOGO 3.0 B: Navigate information in a table structure Adobe Reader crashes when 'Read the entire document' is selected in 'Reading mode options' User could not access the page Author of document has not tagged the document correctly leading to conflicts with ATs. SATOGO 3.0 have confirmed the issue and investigating solutions AT support & Document design High
SATOGO 3.0 B: Access and understand information portrayed through an image/alt SATOGO 3.0 was unable to load document User could not access the file Author of document has not tagged the document correctly leading to conflicts with ATs. SATOGO 3.0 have confirmed the issue and are investigating solutions AT support & Document design High
SATOGO 3.0 B: Identify and move through a document using structure such as headings Headings are not marked up correctly Cannot identify all headings and jump via heading navigation so have to read through all of the document Author of document has not marked up headings consistently. <H2 Subhead> and <heading 2> tags have been used instead of <H2>. SATOGO 3.0 only recognises <H> specific tags. Document design Moderate
SATOGO 3.0 B: Identify and move through a document using structure such as headings List items not identified Users do not know how many items are in a proceeding list SATOGO 3.0 does not recognise list structure AT support Low
SATOGO 3.0 B: Navigate through a large document using page numbers Adobe Reader crashes when 'Read the entire document' is selected in 'Reading mode options' User could not access the page Author of document has not tagged the document correctly leading to conflicts with ATs. SATOGO 3.0 have confirmed the issue and are investigating solutions AT support & Document design High
SATOGO 3.0 B: Interact and complete a form Form fields and labels not associated Users not know what information to enter into a field Author of document not associated labels and form fields correctly Document design High

Window-Eyes 7

Two participants attempted the tasks using this assistive technology.

Success rates

Table 15: Window-Eyes 7 test results

Collection Attempts Successes Failures
A 12 9 3
B 12 3 9

The two Window-Eyes 7 participants succeeded in 50% of the tasks they attempted (12/24) overall in both document collections, with Collection A fairing considerably better at a 66% success rate, compared to only 33% success for Collection B tasks.

The difference in experience/skill level between the two participants was highlighted by their individual success rates. The high experience participant succeeded in seven tasks and failed five. The moderate experience user succeeded in five tasks and failed seven. The participant with a moderate experience/skill level also noted a learning curve as the evaluation progressed.

Specific issues encountered

Table 16: Issues encountered with Collection A using Window-Eyes 7

Issue Cause Impact
Page number in document is not read aloud by Window-Eyes 7 Document design High
Window-Eyes 7 does not accurately associate table containing 2 rows of headers AT support High
Headings not identified by Window-Eyes 7 AT support Moderate
User has to exit 'Browse on' mode to read non editable fields AT support Moderate
User did not know that Window-Eyes 7 does not identify headings AT support & User skill Moderate
User did not know how to navigate table User skill Moderate
Not able to search by page number User skill Moderate

As with the other screen reader participants, the Window-Eyes 7 participants encountered several issues related to the support provided by Window-Eyes 7 for PDF files:

  1. Window-Eyes 7 does not provide comprehensive support for table navigation in PDF files. The users had to tab through each cell and mentally construct the table layout and association between cells.
  2. Headings are not identified in PDF documents. Users had to read all, navigate via paragraphs or ‘arrow down’ the page rather than jump between headings.
  3. When interacting with form fields Window-Eyes 7 does not automatically read edit fields in ‘Browse on’ mode. The user has to jump between ‘Browse on’ and ‘Browse off’ modes.

The first of the three issues above resulted in the user with moderate experience/skill level failing the task, as they could not navigate the table effectively. The other two issues were overcome by both users applying alternative strategies. While the lack of heading navigation had little impact (according to the users) on the ease of task both users stated in the pre–evaluation interview that they navigate in HTML and Word using heading navigation, and this was not available in PDF. As a result, the user with moderate experience/skill level considered the time taken to complete the heading task unacceptable.

Table 17: Full test result for user evaluation by task – Window-Eyes 7

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
Window-Eyes 7 A: Read a short document (general navigation) No issues        
Window-Eyes 7 A: Navigate information in a table structure Window-Eyes 7 does not accurately associate table containing 2 rows of headers Users has to tab through each cell and mentally construct table layout and association between cells and keep going back to verify Window-Eyes 7 does not provide comprehensive support for tables AT support High
Window-Eyes 7 A: Navigate information in a table structure User did not know how to navigate table User did not know if Window- Eyes 7 had any table commands so tried to read the table a cell at a time Moderate experience level of user meant they did not know if Window-Eyes 7 supported table navigation in PDF documents User skill Moderate
Window-Eyes 7 A: Access and understand information portrayed through an image/alt No issues        
Window-Eyes 7 A: Identify and move through a document using structure such as headings Headings not identified by Window-Eyes 7 Users have to read all or navigate via paragraphs, 'arrow down' rather than jump between headings Window-Eyes 7 does not support heading navigation for PDF files AT support Moderate
Window-Eyes 7 A: Identify and move through a document using structure such as headings User did not know that Window-Eyes 7 does not identify headings User tried to find headings using Window-Eyes 7 heading commands for other applications Moderate experience level of user meant they did not know that Window-Eyes 7 does not support heading navigation in PDF documents AT support & User skill Moderate
Window-Eyes 7 A: Navigate through a large document using page numbers Participant not able to search by page number The user did not know how to get to the required page using the Adobe Page Navigation toolbar Moderate experience level meant they were unaware of the Adobe Page Navigation toolbar or how to use it User skill Moderate
Window-Eyes 7 A: Navigate through a large document using page numbers Page number in document is not read aloud by Window-Eyes 7 User cannot establish what page they are on Author of document has not tagged and included the page number in the reading order. Author has used a spread design meant for printing not online viewing Document design High
Window-Eyes 7 A: Interact and complete a form User has to exit 'Browse on' mode to read non editable fields User can access the information but has to jump between 'Browse on' and "Browse off' modes Window-Eyes 7 does not automatically read edit fields in 'Browse on' mode AT support Moderate
Window-Eyes 7 B: Read a short document (general navigation) None of the contents of the first page are contained in the structure tree The text on the first page is not read by Window-Eyes 7 so the users cannot access the information Author of document has not tagged and included the first page in the structure Document design High
Window-Eyes 7 B: Read a short document (general navigation) File freezes when moving between pages User has to wait for Window- Eyes 7 and Adobe Reader to reengage with the document Author of document has not correctly tagged all of the document leading to conflicts with ATs Document design Low
Window-Eyes 7 B: Navigate information in a table structure Table navigation not supported by document mark–up or by Window-Eyes 7 Users tab through each cells and has to mentally construct table layout and association between cells Window-Eyes 7 does not provide comprehensive table navigation, and author of document has not marked up the table to identify header and data cells (<TH> & <TD>) AT support & Document design High
Window-Eyes 7 B: Navigate information in a table structure Adobe Reader crashes when loading reading order File and reader have to be reopened Author of document has not tagged the document leading to some conflicts with ATs Document design High
Window-Eyes 7 B: Access and understand information portrayed through an image/alt The image has no alternative text The users cannot ascertain that an image is present Author of document has not applied an alternative text description to the image Document design High
Window-Eyes 7 B: Identify and move through a document using structure such as headings Headings not identified by Window-Eyes 7 Users have to read all or navigate via paragraphs, 'arrow down' rather than jump between headings Window-Eyes 7 does not support heading navigation AT support Moderate
Window-Eyes 7 B: Identify and move through a document using structure such as headings List items not identified Users do not know how many items are in a proceeding list Window-Eyes 7 does not recognise list structure in PDF documents AT support Low
Window-Eyes 7 B: Navigate through a large document using page numbers Adobe Reader crashes when loading reading order File and reader have to be reopened Author of document has not tagged the document leading to some conflicts with ATs Document design High
Window-Eyes 7 B: Navigate through a large document using page numbers Page number references in the Adobe Page Navigation toolbar are not consistent The page number in the Adobe Page Navigation edit field read 26, but the text beside indicated it was page 32 of 194 confusing the user Author of document has applied non standard page numbering. Adobe Reader reports total number of pages including non standard Adobe Reader & Document design & User skill Low
Window-Eyes 7 B: Interact and complete a form Form fields and labels not associated Users do not know what information to enter into a field Author of document not associated labels and form fields correctly Document design High
Window-Eyes 7 B: Interact and complete a form Users has to exit 'Browse on' to read non editable fields Users can access the information but had to jump between 'Browse' modes (on and off), which caused confusions as to what information was needed Window-Eyes 7 does not automatically read edit fields in 'Browse on' mode AT support Moderate

MAGic 10

One participant attempted the tasks using this assistive technology.

Success rates

Table 18: MAGic 10 test results

Collection Attempts Successes Failures
A 6 6 0
B 4 3 1

The MAGic 10 participant completed all six tasks using the documents in Collection A, but considered the time taken to complete the heading task to be unacceptable and found the task difficult to complete.

Specific issues encountered

Table 19: Issues encountered with Collection A using MAGic 10

Issue Cause Impact
Page number of document does not correspond with the Adobe Page Navigation toolbar Document design Moderate
Hover reading function does not work within content area of document AT support Moderate
Arrowing down functionally not efficient AT support & Document design Moderate
Field name is not read out for postal code field AT support Moderate

Three issues were uncovered that related to support provided by MAGic 10 for PDF files. These related to the speech component of MAGic 10 that was not included in the technical evaluations:

  1. MAGic 10 provides a facility to read text by hovering over it with the mouse. The user was able to access text on the Adobe Reader toolbars and menus using this technique. However they were unable to access text within the document itself. The user had to arrow down through the document instead.
  2. The arrow down functionally, used to read through the document, displayed irregular behaviour. While it read elements of the document, what is highlighted did not align with what is read and sections of text were often repeated.
  3. The user could not get MAGic 10 to read out the field name for the ‘postal code’ field. As a result they had to rely on magnification and visual inspection to complete this task. This issue appears to be similar to that experiences by the NVDA 2009.1 participant and it seemed to be the result of a software bug when entering fields preceded by combo boxes in PDF forms.

The participant stated that MAGic 10 failed to provide the same functionality and efficiency for reading PDF documents that it does for other formats. This made the tasks harder for the user to complete as they had to continually use alternative reading approaches and rely heavily on visual inspection using the magnification component.

Table 20: Full test results for user evaluation by task – MAGic 10

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact

MAGic 10

A: Read a short document (general navigation) 'Hover' reading function does not work with the document User can access the Adobe toolbars and menus but none of the document text when hovering with the mouse. User had to arrow down through the document MAGic 10 does not provide 'Hover' support for PDF files AT support Moderate

MAGic 10

A: Read a short document (general navigation) Arrowing down functionally not efficient While the AT read elements of the document when navigating via arrow keys, what was highlighted and what it read did not align, and the AT often repeated chunks of text AT displays irregular behaviour when arrowing down. Document design may be conflict with AT support AT support & Document design Moderate

MAGic 10

A: Navigate information in a table structure No issues        

MAGic 10

A: Access and understand information portrayed through an image/alt No issues        

MAGic 10

A: Identify and move through a document using structure such as headings 'Hover' reading function does not work with the document User can access the Adobe toolbars and menus but none of the document text when hovering with the mouse. User had to arrow down through the document MAGic 10 does not provide 'Hover' support for PDF files AT support Moderate

MAGic 10

A: Identify and move through a document using structure such as headings Arrowing down functionally not efficient While the AT read elements of the document when navigating via arrow keys, what was highlighted and what it read did not align, and the AT often repeated chunks of text AT displays irregular behaviour when arrowing down. Document design may be conflict with AT support AT support & Document design Moderate

MAGic 10

A: Navigate through a large document using page numbers Page number of document does not correspond with the Adobe Page Navigation toolbar User cannot establish what page they are on Author of document has not tagged and included the page number in the reading order. Author has used a spread design meant for printing not online viewing Document design Moderate

MAGic 10

A: Navigate through a large document using page numbers 'Hover' reading function does not work with the document User can access the Adobe toolbars and menus but none of the document text when hovering with the mouse. User had to arrow down through the document MAGic 10 does not provide 'Hover' support for PDF files AT support Moderate

MAGic 10

A: Interact and complete a form Field name is not read out for postal code field User could not get MAGic 10 to read out the field name for the postal code field so relied on magnification MAGic 10 bug with fields following combo boxes in PDF forms AT support Moderate

MAGic 10

B: Read a short document (general navigation) 'Hover' reading function does not work with the document User can access the Adobe toolbars and menus but none of the document text. User had to arrow down through the document MAGic 10 does not provide 'Hover' support for PDF files AT support Moderate

MAGic 10

B: Read a short document (general navigation) Delay between commands and document response When the document was read by MAGic 10 using the arrow keys it was very slow so user relied on magnification Author of document has not correctly tagged all of the document leading to some conflict with ATs Document design Moderate

MAGic 10

B: Read a short document (general navigation) Arrowing down functionally not efficient While the AT read elements of the document when navigating via arrow keys, what was highlighted and what it read did not align, and the AT often repeated chunks of text MAGic 10 does not provide efficient support for arrowing down and reading PDF files AT support Moderate

MAGic 10

B: Navigate information in a table structure 'Hover' reading function does not work with the document User can access the Adobe toolbars and menus but none of the document text when hovering with the mouse. User had to arrow down through the document MAGic 10 does not provide 'Hover' support for PDF files AT support Moderate

MAGic 10

B: Access and understand information portrayed through an image/alt Not Evaluated        

MAGic 10

B: Identify and move through a document using structure such as headings Not Evaluated        

MAGic 10

B: Navigate through a large document using page numbers Delay between commands and document response When the document was read by MAGic 10 using the arrow keys was very slow, so user relied on magnification Author of document has not tagged the document leading to some conflicts with ATs Document design Moderate

MAGic 10

B: Navigate through a large document using page numbers Poor indication as to loading progress User was not sure if document had loaded MAGic 10 did not provide a clear indicator as to loading progress AT support Low

MAGic 10

B: Interact and complete a form Click to announce field label inconsistent User had to down arrow to read next label then use the mouse to enter field to input data. A mix of audio and visual feedback Author of document not associated labels and form fields correctly Document design Moderate

ZoomText 9.14

Four participants attempted the tasks using this assistive technology.

Success rates

Table 21: Zoom Text 9.14 test results

Collection Attempts Successes Failures
A 24 24 0
B 17 17 0

The four ZoomText participants each attempted the tasks using different customisation and components of the software. as follows:

Even though the participants interacted with the documents in different ways, all four successfully completed all of the tasks that they attempted on both document collections (100% success rate). Overall, the users were satisfied with the time taken to complete the tasks.

Even though the user experience was very positive, the two participants with moderate experienced/skill level, and one with a high experience/skill level, noted a learning curve when working through the tasks. It is likely that this assisted them later in the evaluation when interacting with the representative documents.

Specific issues encountered

Table 22: Issues encountered with Collection A using ZoomText

Issue Cause Impact
Page number of document does not correspond with the Adobe Page Navigation toolbar Document design Moderate

‘Doc Reader’ reads all tables cells as one

AT support

Moderate

Quality of content poor when magnified

User skill

High
Background colours still do not provide sufficient contrast
when inverted, making text hard to read

User skill

Moderate
Table header information not very clear when colour inverted by user

User skill

Moderate
Two column presentation requires horizontal scrolling to read

User skill

Low

In this case, a single issue was uncovered by the participants that related to ZoomText’s support for PDF files. As with MAGic 10 this issue related to the speech component that was not tested in the technical evaluations.

  1. The data cells in the table were not read individually so the user could not move through the table one cell at a time as desired with the ZoomText ‘Doc Reader’.

Interestingly, this group of participants demonstrated the most significant effects of user skill, particularly in relation to their familiarity with using the Adobe Reader interface. Two issues in particular caused the participants some confusion:

  1. When magnified with ZoomText the quality of the document (the Form) was too poor to read. ZoomText advises users to magnify documents in Adobe Reader to 100%, but the users did not know this.
  2. On some documents when the foreground and background colours were inverted they provided insufficient contrast, so the user switched off the inversion and viewed the documents using standard colours. Even though the participant was a highly experienced user of ZoomText, they did not appear to know how to try different colour schemes in ZoomText to find one that provided sufficient contrast.

In both cases, once they had identified how to achieve the required customisation the users quickly adapted to apply these same changes to subsequent documents and tasks.

Table 23: Full test results for user evaluation by task – ZoomText

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
ZoomText 9.14 A: Read a short document (general navigation) No issues        
ZoomText 9.14 A: Navigate information in a table structure 'Doc Reader' reads all tables cells as one The data cells are not read individually so cannot move through the table one cell at a time as wished ZoomText 'Doc Reader' does not provide full support for table structure in PDF AT support Moderate
ZoomText 9.14 A: Navigate information in a table structure Table header information unclear when colour inverted by user User did not find it easy to read table header (black text on grey) when colours inverted Experienced ZoomText user but did not appear to know how to change the colours when the contrast still not sufficient when inverted User skill Moderate
ZoomText 9.14 A: Access and understand information portrayed through an image/alt Two column presentation requires horizontal scrolling to read Users gets a little disorientated after having to horizontal scroll Experience with using PDF files was high but user did not know about the Reflow feature User skill Low
ZoomText 9.14 A: Access and understand information portrayed through an image/alt Background colours still do not provide sufficient contrast when inverted making text hard to read When colours inverted the text became hard to read as the background colour only slightly changed. User turned inverted off The user has high experience, in using ZoomText, but did not appear to know how to change the colours when the contrast was still insufficient when inverted. The user only knew their default setting User skill Moderate
ZoomText 9.14 A: Identify and move through a document using structure such as headings No issues        
ZoomText 9.14 A: Navigate through a large document using page numbers Page number of document does not correspond with the Adobe Page Navigation toolbar Initial confusion as to what page they are on or if page exists. Users search for the correct page using magnification and number in document Complexity of multi page (spread) print presentation Document design Moderate
ZoomText 9.14 A: Interact and complete a form Quality when magnified poor Users magnify the document in ZoomText (x5, x2), but the quality is too poor to read ZoomText advises users to magnify Adobe to 100%, but users did not know this User skill Moderate
ZoomText 9.14 B: Read a short document (general navigation) Table header information in the document not very clear when colours inverted User did not find it easy to read table header (black text on grey) when colours inverted Experienced ZoomText user but did not appear to know how to change the colours when the contrast still not sufficient when inverted User skill Moderate
ZoomText 9.14 B: Read a short document (general navigation) Text not very clear when colours inverted When colours inverted the text became hard to read as the background remained the same – turned inverted off Experienced ZoomText user but did not appear to know how to change the colours when the contrast still not sufficient when inverted User skill Moderate
ZoomText 9.14 B: Navigate information in a table structure Row header read as 2 separate cells so disjointed The row header was split between 2 cells, one read at the start of the row and the other at the start of the next row. User has to rely on magnification to read and understand the table Author of document has placed the row header over 2 cells instead of one merged cell Document design Moderate
ZoomText 9.14 B: Navigate information in a table structure Numerical data separated into 2 cells User was confused as numerical data is read as "2 [space] 380" instead of "2380" Author of document has separated one number into 2 cells Document design Low
ZoomText 9.14 B: Access and understand information portrayed through an image/alt The image has no alternative text Users cannot ascertain the meaning of the image using the 'Reader' so uses magnification Author of document has not applied an alternative text description to the image Document design Moderate
ZoomText 9.14 B: Identify and move through a document using structure such as headings No issues        
ZoomText 9.14 B: Navigate through a large document using page numbers 'Page Thumbnail' does not move with vertical scrollbar User scrolls down the page but the 'Page Thumbnail' remains centre of the page and does not move inline with the position of the vertical scroll bar. The user could not see the thumbnail when scrolling to the top or towards the bottom of the document Adobe centralises the 'Page Thumbnail' and does not allow for it to be fixed/aligned to the position of the scroll bar Adobe Reader Low
ZoomText 9.14 B: Interact and complete a form Form fields and labels not associated Relying on the 'Doc Reader' the user does not know what information to enter into a field. Uses magnification alone Author of document not associated labels and form fields correctly Document design Moderate
ZoomText 9.14 B: Interact and complete a form Print button at top of page When magnified, users do not see whole of page at any one time. User did not notice Print button at top of page Author of document has placed Print button at top of page not bottom – or top and bottom Document design Moderate

Read Out Loud (Adobe Reader 9)

One participant attempted the tasks using this assistive technology.

Success rates

Table 24: Read Out Loud test results

Collection Attempts Successes Failures
A 6 6 0
B 5 2 3

The Read Out Loud Adobe Reader 9 participant successfully completed all of the tasks in the accessible document collection, each in an acceptable amount of time and with ease.

Specific issues encountered

Table 25: Issues encountered with Collection A using Read Out Loud

Issue Cause Impact
Page number in document is not Read Out Loud Document design Moderate

Page number of document does not correspond with the Adobe Page Navigation toolbar

Document design

Moderate

All of first page is read as one block

Document design

Low
Text entered into form fields not automatically read out

AT support

Low

The Read Out Loud participant encountered the same issues relating to 'orientating by page number' task that the screen reader users faced. However, in contrast to screen reader users, the issue did not prevent the participant from completing the task as they were able to adapt and visually inspect the document.

Provided as part of the Adobe Reader program, Read Out Loud provides support for the PDF files as expected. Issues with the document design still present problems for this adaptive strategy. However, in the user evaluations they had no effect on the ease of this participant completing the tasks.

Table 26: Full test results for user evaluation by task – Read Out Loud

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
Adobe Read Out Loud A: Read a short document (general navigation) All of first page is read as one block User has to listen to all of the page as the reading order has not been defined by paragraphs on the first page Author of document grouped all of first page as one block, but separated second page using the headings on the page Document design Low
Adobe Read Out Loud A: Navigate information in a table structure No issues        
Adobe Read Out Loud A: Access and understand information portrayed through an image/alt No issues        
Adobe Read Out Loud A: Identify and move through a document using structure such as headings No issues        
Adobe Read Out Loud A: Navigate through a large document using page numbers Page number of document does not correspond with the Adobe Page Navigation toolbar Initial confusion when page was not found, user visually searches for the correct page using the page number in the document Complexity of multi page (spread) print presentation Document design Moderate
Adobe Read Out Loud A: Navigate through a large document using page numbers Page number in document is not read out by Read Out Loud Page number is read as "blank" so user has to visually read number Author of document has not tagged and included the page number in the reading order. Author has used a spread design meant for printing not online viewing Document design Moderate
Adobe Read Out Loud A: Interact and complete a form Text entered into form fields not automatically read out User has to exit field and 'Shift + Tab' to get data read aloud Read Out Loud does not recognise text as it is typed into edit fields AT support Low
Adobe Read Out Loud B: Read a short document (general navigation) Read Out Loud does not read first page User tries to read first page but Read Out Loud reads "blank, blank" Author of document has not tagged and included the first page in the structure Document design High
Adobe Read Out Loud B: Navigate information in a table structure Adobe Reader crashes when Read Out Load is opened with the document User cannot access file with Read Out Loud Author of document has not tagged the document leading to some conflicts with ATs Document design High
Adobe Read Out Loud B: Access and understand information portrayed through an image/alt The image has no alternative text Users cannot ascertain the meaning of the image via Read Out Loud, has to visually ascertain meaning Author of document has not applied an alternative text description to the image Document design Moderate
Adobe Read Out Loud B: Identify and move through a document using structure such as headings No issues        
Adobe Read Out Loud B: Navigate through a large document using page numbers Adobe Reader crashes when Read Out Load is opened with the document User cannot access file with Read Out Loud Author of document has not tagged the document leading to some conflicts with ATs Document design High
Adobe Read Out Loud B: Interact and complete a form Form fields and labels not associated Users not know what information to enter into a field using Read Out Loud Author of document not associated labels and form fields correctly Document design High

Read & Write Gold 9

One participant attempted the tasks using this assistive technology.

Success rates

Table 27: Read & Write Gold 9 test results

Collection Attempts Successes Failures
A 6 6 0
B 4 4 0

All of the tasks attempted were successfully completed, in an acceptable amount of time and very easily.

Specific issues encountered

Table 28: Issues encountered with Collection A using Read & Write Gold

Issue Cause Impact
Text entered into form fields not automatically read out AT support Low

The participant experienced one issue with the accessible document collection; text entered into the form field was not automatically read out, but required the user to select the 'Click and speak' feature.

Table 29: Full test results for user evaluation by task – Read & Write Gold

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
Read & Write Gold 9 A: Read a short document (general navigation) No issues        
Read & Write Gold 9 A: Navigate information in a table structure No issues        
Read & Write Gold 9 A: Access and understand information portrayed through an image/alt No issues        
Read & Write Gold 9 A: Identify and move through a document using structure such as headings No issues        
Read & Write Gold 9 A: Navigate through a large document using page numbers No issues        
Read & Write Gold 9 A: Interact and complete a form Text entered into form fields not automatically read out User has to select 'Click and speak' feature to get information entered into edit fields read aloud Read&Write 'Reader' does not support information entered into edit fields AT support Low
Read & Write Gold 9 B: Read a short document (general navigation) No issues        
Read & Write Gold 9 B: Navigate information in a table structure Table has not been marked up Header row of the table is read as one cell so all items were read as one sentence Author of document has not marked up the table to identify individual header cells Document design Moderate
Read & Write Gold 9 B: Navigate information in a table structure Numerical data separated into 2 cells User was confused as numerical data is read as "2 [space] 380" instead of "2380" Author of document has separated one number into 2 cells Document design Low
Read & Write Gold 9 B: Access and understand information portrayed through an image/alt No issues        
Read & Write Gold 9 B: Identify and move through a document using structure such as headings Not evaluated        
Read & Write Gold 9 B: Navigate through a large document using page numbers No issues        
Read & Write Gold 9 B: Interact and complete a form Not evaluated        

Dragon professional 10.1

One participant attempted the tasks using this assistive technology.

Success rates

Table 30: Dragon Professional 10.1 test results

Collection Attempts Successes Failures
A 6 6 0
B 4 4 0

The Dragon Professional 10.1 participant successfully completed all tasks attempted. The time taken was acceptable for all but the 'orientate by page number' task. The participant was initially confused about the difference between the page number provided in the Adobe Reader Page Navigation toolbar and the document itself. This confusion led the participant to regard the time as unacceptable.

Specific issues encountered

Table 31: Issues encountered with Collection A using Dragon Professional 10.1

Issue Cause Impact
General Dragon commands do not work AT support Moderate

Dragon does not apply form field numbering overlay for PDF forms

AT support

Low

Text input is not as accurately recorded in PDF documents as it is with other application

AT support

Low

The participant uncovered several AT support issues when using Dragon with PDF files. These issues were based on experiences and expectations the participant had for how Dragon interacts with other file formats such as HTML and Word.

Overall the participant experienced a sense of frustration that Dragon commands were not available or did not function as expected when using PDF files. This included dictated text input into fields on the PDF form where Dragon did not correctly recognise the spoken commands. A comparative test of input text into a Word document or HTML form confirmed that the experience in the PDF file was not equivalent.

Another example was the navigation of forms. In an HTML form, Dragon overlays and assigns a shortcut number to each form field to allow the user to quickly navigate to a particular field in the form. This functionality is not available in PDF forms, and there was no equivalent way for the user to move to a particular form field. This presented a problem when the participant wanted to review the form. To complete such a task, the participant had to exit from the form fields, and if wishing to re–enter form fields, needed to begin from the top of the form. As a result, the participant had to navigate through the form again via continual “press tab” statements to get back to the relevant field. This issue was not highly concerning, but created additional confusion and frustration for the participant, as the assistive technology did not operate in a familiar manner.

Table 32: Full test results for user evaluation by task - Dragon Professional 10.1

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
Dragon Professional 10.1 A: Read a short document (general navigation) No issues        
Dragon Professional 10.1 A: Navigate information in a table structure No issues        
Dragon Professional 10.1 A: Access and understand information portrayed through an image/alt No issues        
Dragon Professional 10.1 A: Identify and move through a document using structure such as headings No issues        
Dragon Professional 10.1 A: Navigate through a large document using page numbers General Dragon commands not work User tries to use common Dragon commands that work with Word (OK/ Cancel) when using the Adobe Page Navigation toolbar dialog box. These did not work and user had to say 'Press Enter' instead of 'OK' Dragon does not provide as much support for Adobe PDF as it does for other applications AT support Moderate
Dragon Professional 10.1 A: Interact and complete a form Text input is not as accurately recorded as with other application User had to repeatedly correct information she had entered via speech as mistakes were recorded Dragon support for PDF and Adobe in particular is less than for other established applications AT support Low
Dragon Professional 10.1 A: Interact and complete a form Dragon does not apply form field numbering overlay for PDF forms In an HTML form Dragon overlays and assigns a shortcut (number) to each field. This does not apply to PDF forms so user has to navigate via continual "press tab" statements to move to the fields Dragon support for PDF and Adobe in particular is less than for other established applications AT support Low
Dragon Professional 10.1 B: Read a short document (general navigation) No issues        
Dragon Professional 10.1 B: Navigate information in a table structure No issues        
Dragon Professional 10.1 B: Access and understand information portrayed through an image/alt No issues        
Dragon Professional 10.1 B: Identify and move through a document using structure such as headings No issues        
Dragon Professional 10.1 B: Navigate through a large document using page numbers No issues        
Dragon Professional 10.1 B: Interact and complete a form Text input is not as accurately recorded as with other application User had to repeatedly correct information she had entered via speech as mistakes were recorded Dragon support for PDF and Adobe in particular is less than for other established applications AT support Low
Dragon Professional 10.1 B: Interact and complete a form Dragon does not apply form field numbering overlay for PDF forms User wanted to scroll back to another place in the form, and was required to exit the field. When wishing to re–enter the form fields, Dragon placed user at beginning of form Dragon support for PDF and Adobe in particular is less than for other established applications. Dragon does not apply a form overlay to enable efficient navigation AT support Low

Keyboard only

Two participants attempted the tasks using this adaptive strategy.

Success rates

Table 33: Keyboard only test results

Collection Attempts Successes Failures
A 11 11 0
B 9 9 0

The two participants who relied on the keyboard to navigate and interact with the document succeeded in all tasks they attempted. Users found the tasks easy to complete.

No specific issues were identified regarding the use of a head pointing device (which operates the keyboard) however, one participant did note that when using a head stick, they are touch typing and not always looking at the screen, therefore navigation between fields is difficult. It is likely that this affected the user’s experience regardless of document type.

Specific issues encountered

Table 34: Issues encountered with Collection A using Keyboard only

Issue Cause Impact
Page number of document does not correspond with the Adobe Page Navigation toolbar Document design Moderate

User reads left column first but information starts on the right

Document design

Low
Table headers not visible when using the Reflow feature in Adobe Reader Adobe Reader Moderate
Page numbers not visible when using the Reflow feature in Adobe Reader Adobe Reader Moderate

User is not aware of the Reflow feature in Adobe Reader

User skill

Low

The participants did not experience any High severity issues, but they did encounter three Moderate issues – two of which related to the ‘Reflow’ feature in Adobe Reader 9.

  1. The participant could not use the Reflow feature in Adobe Reader as the headers in the table were not displayed. Reflow removed the grey background so the white header text was not visible.
  2. The participant had to find the page first via horizontal scrolling, then revert back to the Reflow feature as Reflow in Adobe Reader did not display information in the footer (page numbers) of the document.
  3. The page numbers in the document and those displayed in the Adobe Page Navigation toolbar did not correspond causing confusion. For the user with the head pointing device, excessive movement was required to keep assessing what page they were actually on.

The keyboard only participants successfully interacted with the documents and managed to complete the form in both document collections. The Adobe Reader menu was operational using the keyboard and sticky keys. The Reflow feature, that might have helped one of the users who also increased text size to 200% due to a vision impairment, was not an effective alternative.

Table 35: Full test results for user evaluation by task – Keyboard only

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
Keyboard A: Read a short document (general navigation) No issues        
Keyboard A: Navigate information in a table structure Table headers not visible when using the Reflow feature in Adobe Reader User had to revert back to horizontal scrolling (exit reflow) as headers not displayed Reflow in Adobe Reader did not inherit background colours so any white text was not visible Adobe Reader Moderate
Keyboard A: Access and understand information portrayed through an image/alt User reads left column first but information starts on the right The second column of text was only partially visible when page zoomed to 200%. User has to scroll horizontally to find start (right hand column) of page causing slight confusion Author of document has designed it for print. Has used 2 column presentation with start of document on right hand side of page Document design Low
Keyboard A: Identify and move through a document using structure such as headings No issue        
Keyboard A: Navigate through a large document using page numbers Page number of document does not correspond with the Adobe Page Navigation toolbar Confused when page was not found, user had to search for the correct page using the number on document Complexity of multi page (spread) print presentation Document design Moderate
Keyboard A: Navigate through a large document using page numbers User is not aware of the Reflow feature in Adobe Reader User had to keep moving to the bottom of the page and then horizontally scrolling to find page numbers Experience with using PDF files was moderate and they did not know about the Reflow feature. Adobe does not make the feature obvious and intuitive to new users User skill Low
Keyboard A: Navigate through a large document using page numbers Page numbers not visible when using the Reflow feature in Adobe Reader User had to find the page first via horizontal scrolling then revert back to the Reflow feature Reflow in Adobe Reader did not display information in the footer Adobe Reader Moderate
Keyboard A: Interact and complete a form No issues        
Keyboard B: Read a short document (general navigation) No issues        
Keyboard B: Navigate information in a table structure No issues        
Keyboard B: Access and understand information portrayed through an image/alt No issues        
Keyboard B: Identify and move through a document using structure such as headings No issues        
Keyboard B: Navigate through a large document using page numbers No issues        
Keyboard B: Interact and complete a form Focus automatically jumps to next form field without user control Users confused as upon entering information they moved automatically to the next field without tabbing. Has to keep going back to previous field as they instinctively tabbed after entering information Author of document change the focus to the next control without user operation Document design & User skill Low

Operating System (OS) configuration

One participant attempted the tasks using this adaptive strategy.

Success rates

Table 36: Operating System Configuration test results

Collection Attempts Successes Failures
A 6 6 0
B 6 6 0

The participant with a cognitive disability (perceptual dyslexia) found the default colour scheme made the text hard to read. A change in the operating systems (Windows) background colour to blue/grey was required to provide sufficient contrast. After implementing the colour change the participant successfully completed all twelve tasks.

Specific issues encountered

Table 37: Issues encountered with Collection A using OS configuration

Issue Cause Impact
OS colours not inherited by form fields in document Adobe Reader & Document design Moderate

Colours specified in OS not inherited in document

Adobe Reader & Document design

Moderate
Adobe Reader customisation settings not intuitive Adobe Reader & User skil Moderate

All of the issues the participant encountered (outside those relating to comprehension of information and readability of the document content) related to the specified background colours. For some documents, the design of the document and/or functionality provided by the Adobe Reader meant that the document failed to inherit and render the foreground and background colours specified in the operating system. The participant managed to complete the task in these situations but the tasks were considered difficult.

Table 38: Full test results for user evaluations by task – OS Configuration & Adobe

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
OS & Adobe configuration A: Read a short document (general navigation) No issues        
OS & Adobe configuration A: Navigate information in a table structure No issues        
OS & Adobe configuration A: Access and understand information portrayed through an image/alt Colours specified in OS not inherited with this document User finds it hard to read text on page due to coloured background, only option is to change background colour in Adobe Reader itself to black, but then black text not visible Adobe Reader did not inherit OS colours Adobe Reader & Document design Moderate
OS & Adobe configuration A: Identify and move through a document using structure such as headings No issues        
OS & Adobe configuration A: Navigate through a large document using page numbers No issues        
OS & Adobe configuration A: Interact and complete a form OS colours not inherited by form fields in this document The background colour of edit fields (inactive and active) do not inherit colours specified by the user so caused minor difficulties Adobe Reader does not inherit/ render customised colours for form fields Adobe Reader & Document design Moderate
OS & Adobe configuration A: Interact and complete a form Adobe Reader customisation settings not intuitive User did not know how to override colours in document – had to experiment with menu options Experience level with PDF was high, but user was not aware how to override colour settings using Adobe Reader. Adobe does not make the feature obvious and intuitive to new users Adobe Reader & User skill Moderate
OS & Adobe configuration B: Read a short document (general navigation) No issues        
OS & Adobe configuration B: Navigate information in a table structure No issues        
OS & Adobe configuration B: Access and understand information portrayed through an image/alt No issues        
OS & Adobe configuration B: Identify and move through a document using structure such as headings No issues        
OS & Adobe configuration B: Navigate through a large document using page numbers No issues        
OS & Adobe configuration B: Interact and complete a form OS colours not inherited by form fields in this document The background colour of edit fields (inactive and active) do not inherit colours specified by the user, so caused minor difficulties Adobe Reader does not inherit/ render customised colours for form fields Adobe Reader & Document design Moderate

Auslan

One participant attempted the tasks.

Success rates

Table 39: Auslan test results

Collection Attempts Successes Failures
A 6 6 0
B 6 6 0

The participant with a hearing impairment, for whom Auslan was their first language, completed all of the tasks and considered the time acceptable on all occasions, except for the 'orientating by page number' task.

Specific issues encountered

Table 40: Issues encountered with Collection A using Auslan

Issue Cause Impact
Page number of document does not correspond with the Adobe Page Navigation toolbar Document design Moderate

This participant experienced one significant issue. Like many of the other participants, the Auslan participant was confused when the page numbers in the document did not match the page numbers displayed in the Adobe Reader Page Navigation toolbar. The participant was able to complete the task but deemed the time to be unacceptable.

Table 41: Full test result for user evaluations by task – Auslan

Adaptive Strategy Collection & Task Issues Implications Insights Causes Impact
None (Auslan) A: Read a short document (general navigation) No issues        
None (Auslan) A: Navigate information in a table structure No issues        
None (Auslan) A: Access and understand information portrayed through an image/alt No issues        
None (Auslan) A: Identify and move through a document using structure such as headings No issues        
None (Auslan) A: Navigate through a large document using page numbers Page number of document does not correspond with the Adobe Page Navigation toolbar Confused as to what page they are on, or if page exists. User searches for the correct page using the number in document Complexity of multi page (spread) print presentation Document design Moderate
None (Auslan) A: Interact and complete a form No issues        
None (Auslan) B: Read a short document (general navigation) No issues        
None (Auslan) B: Navigate information in a table structure No issues        
None (Auslan) B: Access and understand information portrayed through an image/alt No issues        
None (Auslan) B: Identify and move through a document using structure such as headings No issues        
None (Auslan) B: Navigate through a large document using page numbers No issues        
None (Auslan) B: Interact and complete a form Focus automatically jumps to next form field without user control Users confused as upon entering information they are moved automatically to the next field without tabbing. Finds it useful once they realise how it operates Author of document change the focus to the next control without user operation Document design & User skill Low

 

Contact for information on this page: wcag2@finance.gov.au


Back to top

Last Modified: 25 November, 2010