Unbiased measurement & assessment

Public Governance, Performance and Accountability Rule 2014 section 16EA(c)

The performance measures meet this requirement when they provide an unbiased basis for the measurement and assessment of the entity's performance.

Performance measures are required for each reporting period covered by the plan (that is, a minimum of 4-years). Entities often demonstrate this coverage through specified targets, where reasonably practicable to set a target, for each of the reporting periods covered by the plan.

Performance measures must provide an unbiased basis for assessment of the entity’s performance.

When designing an approach to collecting and/or analysing information, entities need to consider how bias might be unintentionally introduced.

Common sources of bias

Exclusion bias

Occurs when relevant information is not collected. For example, where information is not collected on activities that make key contributions to fulfilling a purpose.

Sampling bias

Occurs when individuals or groups are disproportionately represented in the sample from which information is collected. For example, when key stakeholders are omitted from the sample group.

Interaction bias

Occurs when the sample group is aware that it is being observed/tested and changes its behaviour either consciously or subconsciously. For example, when people perform differently because they know they are being observed/tested.

Perception bias

Occurs when the people collecting or analysing information have preconceived ideas about how a system should behave or about what the results will be. For example, focusing on information that confirms existing beliefs while ignoing contradictory evidence.

Operational bias

Occurs when the process for collecting information is not followed or when errors are made in the recording and analysis of data. For example, when the manner in which data is gathered or processed introduces systematic errors, skewing the results.

Case studies

Where case studies are used within a performance measure (that is, used to measure an entity's performance), entities should ensure that the performance measures are supported by a clear methodology that explains the basis for selecting case studies.

Case studies selected ‘after the fact’ (that is, after a particular activity has begun or has been completed, or after the entity’s period for performance has ended) introduce the potential for bias.

Case studies should not be relied upon as a stand-alone measurement unless the scope of the case study is predetermined, the activities clearly stated, and the measurement methods determined in advance. This avoids the risk of introducing bias where only favourable case studies that tell ‘success stories’ are selected.

Case studies and details of information to be collected should therefore be decided at the time the corporate plan is developed and before information collection occurs. Sufficient information should be included in the corporate plan to provide confidence to the reader that the selection of case studies and reviews are unbiased.

Better practice principles:

appropriate supports and frameworks should be developed prior to the selection of case study topics
the scope of the case study should be pre-determined, the activities clearly stated and the measurement methods determined in advance
the case study methodology should clearly explain the basis for selecting case studies.

Example

2024–25 Department of Foreign Affairs and Trade Corporate Plan (pages 21-22)

The Department of Foreign Affairs and Trade’s (DFAT) corporate plan includes performance measures which used case studies as the basis for reporting 2 performance measures, including ‘Measure 1: DFAT’s diplomatic efforts support favourable foreign affairs and trade policy outcomes for Australia’. As the Auditor-General Report No. 25 of 2024–25 noted : ‘Diplomacy and foreign affairs and trade policy are inherently difficult to measure. …DFAT’s approach to the use of case studies demonstrated that meaningful reporting of qualitative performance information can be done in an unbiased manner if the appropriate supports and frameworks are developed prior to the selection of topics.’

^Source:^{Commonwealth of Australia, Department of Foreign Affairs and Trade, Department of Foreign Affairs and Trade Corporate Plan 2024–25, November 2024}

Surveys and contracted researchers

Where performance measures are based on surveys, entities should ensure that measures are supported by a clear methodology that identifies how surveys will be conducted.

Where performance measures are based on survey results, documentation should be created which identifies factors including the survey sample population and size, how the survey will be conducted (including controls over the survey process and data) and an acceptable survey response rate.

Where performance measures are based on self-reporting by a third-party provider of services, an assessment of the risk of the provider skewing results to be positive in nature should be undertaken. This would include assessing how this risk will be addressed and disclosed.

Where performance measures are based on assessment and measurement undertaken by those responsible for the management of the program or activity, there should be adequate controls in place to demonstrate that the result was objectively assessed. These might include ensuring the process of selecting which components or client groups the program or activity is targeting is independent and/or involves a random selection process. It might also include ensuring independence of assessment through including individuals not involved in the management of the program or activity in the assessment process or using a totally independent assessment process.

When considering the use of a contracted (independent) research and survey provider the selection process should take account of the qualifications and experience of the provider and the standards under which the provider will conduct work.

This includes ensuring the work is conducted in accordance with:

international quality standards (for example ISO 20252 Market, opinion and social research, including insights and data analytics — Vocabulary and service requirements)
information security standards (for example ISO 27001 Information security management)
Australian Privacy Principles (Privacy Act 1988).

A statement of quality and compliance with standards should be provided in the research report.