Better Practice Checklist - 11. Website Usage Monitoring and Evaluation
May 2004 (contact details updated January 2008)
Introduction
Monitoring and evaluating the usage that target groups make of websites and other e-government initiatives are an important part of the ongoing management of these initiatives. Results can inform future development and, in combination with the evaluation of other aspects of the initiative, such as costs, can help ensure that the initiative meets the business needs of agencies.
A key role of the Australian Government Information Management Office (AGIMO), Department of Finance and Deregulation is to identify and promote 'Better Practice'. This checklist has been created to help agencies to evaluate the use of their websites in the wider context of program output and outcome monitoring and evaluation. The items in the checklist are, however, not mandatory. Agencies will need to consider other issues not covered in this checklist when using channels apart from the Internet to achieve their outputs and outcomes.
This checklist is intended to be a guide for general program managers responsible for the achievement of agency outputs and outcomes. It will also be of interest to webmanagers and business managers responsible for IT areas in agencies. This checklist focuses on non-technical issues.
It should be noted that the checklist is not intended to be comprehensive. Rather, it highlights key issues for agencies. The checklist is iterative and draws on the expertise and experience of practitioners. The subject matter and issues are reviewed and updated to reflect developments.
Complementary documentation
This checklist is intended to complement the Demand and Value Assessment [
] methodologies developed by AGIMO as well as audits undertaken and recommendations made by the Australian National Audit Office on the monitoring and evaluation of Internet services by government agencies www.anao.gov.au [].
Acknowledgments
This checklist was developed with the assistance of Australian Government agencies. In particular, we would like to thank the Australian Customs Service, Australian National Audit Office and Centrelink.
What do monitoring and evaluation of website usage encompass, and why undertake them?
Website usage monitoring involves the collection of information about how a website is being used. Evaluation is the analysis of the usage-monitoring information to assess the success of an online service in fulfilling or meeting business goals. Key tools or techniques in usage monitoring and evaluation include surveys, focus groups, usability testing and website statistics.
The monitoring and evaluation process is an important way of ensuring that the website meets the business needs of departments and agencies as well as the needs of users. As with other government programs, monitoring and evaluation of the usage of websites and other e-government initiatives form part of agencies' corporate governance responsibilities.
The process has a range of potential advantages. These include reducing the risk of budget or scope blow-out by helping to target and refine the scope of website activities, such as redevelopment and site marketing, to ensure that the site delivers maximum benefits at an acceptable cost.
Results can also reveal changes in usage patterns, such as reduced number of visitors. Tips for increasing awareness of websites are available in Better Practice Checklist 20, Marketing E-government [
].
Summary of Checkpoints
Fundamentals
Develop and implement policies on the monitoring and evaluation of websites
When developing business cases and objectives for future services, ensure that they facilitate monitoring and evaluation
Ensure that usage monitoring and evaluation activities are undertaken in accordance with privacy principles
Before starting
Review the objectives of the site or service to be monitored and evaluated
Consider which aspects of the site's usage should be monitored and evaluated
Schedule monitoring and evaluation throughout the product lifecycle
Methods
Consider the use of online surveys
Consider the use of focus groups
Consider user testing
Consider website statistics
Consider web-reporting software or services
Provide a variety of mechanisms for users to give feedback online
Checkpoints
Fundamentals
As the delivery of information and services online is now an integral part of how most agencies conduct their activities, the monitoring and evaluation of website usage should be an integral part of program evaluation. To facilitate this, agencies may need to do the following.
Develop and implement policies on the monitoring and evaluation of websites
Agency-wide policies on monitoring and evaluation can help ensure that these tasks are undertaken consistently to an appropriately high standard. Policies can:
- define roles and responsibilities
- specify a range of monitoring information
- provide for scheduling of periodic evaluation
- clarify reporting and accountability arrangements.
When developing business cases and objectives for future services, ensure that they facilitate monitoring and evaluation
Future usage monitoring and evaluation can be facilitated if clear business cases are documented. These should ensure that initiative has clear objectives and agreed performance indicators. This provides a baseline against which the results of any monitoring can be evaluated.
Business cases and objectives may be particularly useful if they:
- define objectives in relation to the intended clients of services
- specify the information required to monitor and assess the performance of services and to demonstrate their contributions to agency outcomes and outputs.
Ensure that usage monitoring and evaluation activities are undertaken in accordance with privacy principles
Any tracking of individuals, their demographics and personal information, must be done in strict compliance with Commonwealth privacy legislation. This includes informing the individual of what information is being collected, why it is being collected, who will have access to it and where it will be stored.
Agencies are bound by the Privacy Act (1988), and the Privacy Commissioner's Information Privacy Principles (IPPs) www.privacy.gov.au/publications/ipps.html [] provides the definitive guide on the use of personal information collected by government.
The Guidelines for Federal and ACT Government World Wide Websites supplement the IPPs for handling personal information through websites www.privacy.gov.au/internet/web/index.html [].
Before starting
Review the objectives of the site or service to be monitored and evaluated
Results of evaluations are meaningful only if the evaluation has focused on the right things and if the results have been considered in the context of the original goals or objectives of the site or service that has been evaluated. For example, an evaluation of the number of overseas users of a site that was developed to meet the needs of users living in Australia would shed little light on how well the site meets its original goals. Similarly, evaluations based on absolute numbers of users are meaningless unless the numbers are considered in relation to the size and profile of the target group.
Consider which aspects of the site's usage should be monitored and evaluated
The review of the site's objectives or goals will help to inform what is monitored and what level of performance is desirable.
To ensure that services remain relevant and responsive to user needs, it may be useful to broaden monitoring to reveal access to an agency site by different users, the needs of users, and the benefits users gain from the services. Agencies may also monitor how well their sites comply with the Minimum Website Standards to meet the needs of their users.
General areas of monitoring for some websites can also be drawn from industry best practice. The following lists some areas that could be included in the website monitoring as desirable attributes for these areas.
- Home page . This should provide a clear and inviting introduction to the site, showing the services or information available to users. The organisation owner could be identified through 'About us' or 'About this site' buttons available from the home page.
- Branding and design . These should clearly identify the organisation, reflect audience needs/aspirations, integrate with any offline branding (logos, style and colours that represent the organisation) and provide an optimal online experience. Further information is available in Better Practice Checklist 15, Information Architecture for Websites.
In addition, AGIMO has developed The Australian Government Design: Guidance for the Online Environment http://webpublishing.agimo.gov.au/Visual_Design_and_Branding [] to assist departments and agencies to ensure that the Australian Government Design is consistently applied on their websites.
- Navigation. Users should be able to find the information they want and to know exactly where they are in the site at any time. Agencies need to consider 'common user experience', which is part of the Australian Government Design at http://webpublishing.agimo.gov.au/Visual_Design_and_Branding []. Further information on navigation is available in Better Practice Checklist 2, Website Navigation [] and Better Practice Checklist 15, Information Architecture for Websites.
- Discoverability and searching . Content and services should be easy to find on the Internet using popular search engines, such as Google. Once on the site, information should be easy to discover using the site's search function. Further information is available in Better Practice Checklist 16, Implementing an Effective Website Search Facility [].
- Customer focus and communication. The site should be organised around the way users think about the information, rather than take a strictly organisational view. Where a site deals with multiple audiences (with different needs), the site should provide clear pathways for each audience, showing the important and useful things available.
- Content and readability . Information on the site should be comprehensive and well-written and meet the user's needs.
- Technical performance . The site should perform adequately using a dial-up modem and equipment and hardware that are three to five years out of date. This helps ensure that sections of users are not disadvantaged--for example, regional users having poor bandwidth in remote areas, or users having old equipment.
- Accessibility . The site should be able to be used by people with disabilities and those using assistive technologies such as screen readers. Accessibility evaluation usually centres on achieving a rating based on World Wide Web Consortium Website Accessibility Initiative www.w3.org/WAI/ [].
Schedule monitoring and evaluation throughout the product lifecycle
Monitoring and evaluation are not a one-off activity but rather an ongoing process that can be carried out at scheduled times identified by the agency. At each stage, the evaluation process may have a different focus. For example:
- During the planning stage, the objective may be to evaluate the concept for the site and users' needs and preferences.
- During the design stage, the objective may be to check that the scope and purpose of the site are clear and that users can predictably find and access the services and information they need.
- During the pre-launch/development stage, the objective may be to evaluate the site to ensure that it meets corporate and government standards and has acceptable technical performance.
- During the operating life of the website, the objective may be to evaluate that the site is meeting the needs of its users and its business objectives.
Methods
A range of methods can be used to monitor and evaluate sites. To ensure robust results, agencies may find it useful to use more than one method. For example, by supplementing statistical analysis with user testing or focus groups, a more detailed understanding of the users of the website and of their needs can be gained and used to help ensure that the site remains relevant.
Consider the use of online surveys
Online surveys are a useful method to find out information about current users, such as demographics, geographic location, their experience on the website whether good or bad, navigation, suggestions for improvement, website preferences and opinions.
Surveys that take little time for the user to complete offer more incentive for users to participate. Many government websites have implemented '60 second surveys' on their websites.
Surveys may have the following advantages. They can provide:
- quick and cost effective access to the views of a range of users
- quantitative data on the types of users and their preferences
- results that can be automatically tabulated and made immediately available to the website manager.
Responses, however, will be skewed towards people who are already using the website and who are more Internet-literate (because they may feel confident completing an online survey). To find out what potential users and less Internet-literate users may want, it may be useful to use other techniques, such as focus groups, where participants can be selectively recruited.
Agencies will need to ensure that all survey data is collected in a way that conforms with privacy requirements www.privacy.gov.au/internet/web/index.html [] .
Consider the use of focus groups
Directly interacting with small groups of people selected from the site's target audiences, including current users, non-users and potential users of the site, allows in-depth exploration of participants' opinions and preferences through active discussion and hands-on activities. Focus groups can be used to gather data on issues such as website preferences and priorities, use of related websites, purpose and intention of visits to the site, and reaction to visual design (for example, home page mock-ups, colour palettes, logo designs).
An advantage of focus groups is that they provide the opportunity to have broad-ranging and in-depth discussion on issues that impact upon the website. The group interaction can lead to new insights about the website, which can then be taken to the next level by the website team.
A potential disadvantage of focus groups can be that, as they rely on participants' opinions and interpretations rather than hard evidence of actual behaviour, group and individual opinion can be swayed by dominant participants or by the facilitator. This can, however, be countered by careful selection of participants and skilled facilitation.
Consider user testing
User testing is a useful method of determining whether users are able to interact with the website effectively, so it can establish whether the needs of both the users and the agency are met by the website. Further information on user testing is available in Better Practice Checklist 3, Testing Websites with Users .
Consider website statistics
Website statistics measure the activity on the website. Website statistics may be provided by either web server logs or browser-based systems. By setting up servers to record the information, these logs can be used to provide answers to questions such as: How many visitors have come to the site? What content is most visited? What browsers are being used?
Website statistics can be particularly useful when collected and reviewed over a period, so that patterns of use can be seen. Statistics can also be used to assess the impact on site usage from significant events. These could be changes to the website design, a specific marketing campaign for the website or its subject, or changes in users' needs.
While a wide range of website usage statistics may be available, caution is needed in interpreting them. They are most useful to indicate trends rather than provide definitive data/statistics on website usage. Consider the following:
- Hits. This is the number of times all website files (including graphics) have been viewed, and each hit is a single entry in a server log file. A single web page could include 50 files (1 html file, 49 graphic files) and generate 50 hits. Hits are generally not a useful measure of user activity as they are influenced by the design of the website rather than by actual use.
- Page views. This is the number of visits to individual pages on the website. This can be used to identify the most frequently accessed pages and, conversely, which pages are not used. This information can be used to identify pages that should be promoted more often in the site, track the effects of putting particular links on the home page, or identify broken navigation paths.
- Visits and unique visitors. This indicates specific visits to the website and shows the number of users who come to the site, how often they visit and how long the visit lasted. Users may open multiple sessions of the website during a single visit.
Page views and visits can be cross-referenced to find out how many people went to only one page on the site, compared to how many visited two to 10 pages, or 10 pages or more. This information will indicate if people are coming to the site to read a single page or browsing more generally.
- Referring websites. This indicates where users came from if they got to the website by following a link from another website (also known as 'upstream entry points'). This information can indicate what the user was doing before they got to the site and can help in developing online marketing campaigns. This could include liaising more closely with current or potential referring sites to develop reciprocal links and promotion.
- Referral websites. This indicates where users went when they left the website (also known as 'downstream exit points'). This information is particularly useful in portals so that the most popular external links can be identified.
- Top entry pages. This indicates which website pages people entered the site through: the home page or other key pages. Very often the home page is used to alert people to changes to the site, updated services and important news. If a significant percentage of users do not enter through the home page, the agency will need to consider complementary ways of ensuring that these messages are seen--for example, by creating sidebar news columns on all key entry pages.
- Demographics. Most website statistics can only give minimal information concerning user demographics. This is limited to factors such as knowing from what country users are accessing the site, the time of access (which would show whether most access occurs during business hours or after work or on weekends), and whether access is through a government, educational or commercial domain.
Some site usage services have tried to solve this problem by encouraging users to register themselves and then tracking those individual users. On registering, users provide information about themselves, such as age, gender, home address/state, income, level of education, interests and reasons for visiting the site. This enables a much richer set of data to be produced, because individual behaviour can be tracked rather than the anonymous 'unique visitors'.
This information can be misleading, however. For example, usage information may indicate that the user is in the United States , when they are in Australia and accessing the site through a US-based server. Information from demographic statistics is useful in indicating trends only.
- Search results (successful and failed). This indicates the keywords that users choose to search the site, and whether the searches are successful or not. An analysis of these keywords can help refine metadata on the site, so that pages are more easily discovered. For example, users may be misspelling the name of an important document on the site - such as 'McKenzie Report' instead of 'MacKenzie Report' - resulting in a failed search result. This could be remedied by including the misspelling as part of the metadata for the document, as well as in the list of site synonyms. Further information is available in Better Practice Checklist 6, Use of Metadata for Web Resources and Better Practice Checklist 16, Implementing an Effective Website Search Facility [].
- Browser and operating system . This indicates which browsers (Internet Explorer, Netscape, Opera, etc.) and operating systems (Windows, Mac, Linux) people are using to access the site. This information will enable decisions to be made about the need for the website to support a broader or narrower range of types and versions of technology in use.
- Path analysis. This indicates a step-by-step path of the pages that users have taken through the site from their point of entry to their exit. This can be used to optimise website navigation by checking whether the navigation is easy and intuitive for users.
- Internet web-traffic measurement and ranking. This is a comparison of Internet traffic data on similar websites. It is provided in reports by companies that measure web traffic, such as Hitwise.
Consider web-reporting software or services
Site usage statistics can be generated through web-reporting software or site-monitoring services. Some web-reporting software analyses the log file on the web server. The reports can be customised to suit the website and run regularly to provide trend analysis.
Other software analyses client-side usage, which can provide advantages such as being able to differentiate between page views retrieved through proxy servers or browser caches, or even 'back' and 'forward' buttons, where server-based measurement cannot. Such services can also combine tracking with user registration to provide a richer view of site usage.
Provide a variety of mechanisms for users to give feedback online
Many agencies provide an email link to the web manager for providing feedback about websites. Consider that feedback provided to webmanagers may be predominantly about relatively straightforward issues, such as broken links, rather than the user's overall experience of using the service. If a feedback mechanism is provided, someone must be able to provide a timely response.
Links to web managers can be supplemented by links to relevant business unit owners, online surveys, facilities for users to rank pages and other mechanisms to elicit more detailed feedback.
Other Better Practice Checklists
Contact for information on this page: AGIMO Better Practice Team