Special Topics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 234

Table 9.12 Summary of task performance data a .

Original Site Redesigned Site
Average Task Completion Rate 72% 95%
Average Task Time (mins) 2.2 0.84
Average Efficiency 33% 113%
a Average efficiency is the task completion rate per unit of time (Task Completion Rate/Task Time). Adapted from
Withrow et al. (2000); used with permission.

Table 9.12 Summary of task performance data a .
a Average efficiency is the task completion rate per unit of time (Task Completion Rate/Task Time). Adapted from
Withrow et al. (2000); used with permission.

So far, everything is very straightforward and simply illustrates some of the
usability metrics discussed in this topic. But here's where it gets interesting. To
begin calculating ROI from the changes made to the site, Withrow et al. (2000)
made the following assumptions and calculations related to time savings :

• Ofthe2.7millionresidentsofthestate,wemight“conservativelyesti-

mate” a quarter of them use the website at least once per month.

• Ifeachofthemsaved79seconds(aswastheaveragetasksavingsinthis

study), then about 53 million seconds (14,800 hours) are saved per year.

• Convertingthistolaborcosts,wefind370person-weeks(at40hoursper

week) or 7 person-years are saved per month; 84 person-years are saved
each year.

• Onaverage,acitizeninthetargetstatehadanannualsalaryof$14,700.

• This leads to a yearly benefit of $1.2 million based only on the time

savings.

Note that this chain of reasoning had to start with a pretty big assumption: that
a quarter of the residents of the state use the site at least once per month. So that
assumption, which all the rest of the calculations hinge upon, is certainly up for
debate. A better way of generating an appropriate value with which to start these
calculations would have been from actual usage data for the current site.

They went on to calculate an increase in revenue due to the increased task
completion rate for the new site:

1. The task failure rate of the old portal was found to be 28%, whereas the
new site was 5%.
2. We might assume that 100,000 users would pay a service fee on the
orderof$2pertransactionatleastonceamonth.

3. Then the 23% of them who are succeeding on the new site, whereas for-

merlytheywerefailing,aregeneratinganadditional$552,000inrev-
enue per year.

Again, a critical assumption had to be made early in the chain of reasoning:
that100,000userswouldpayaservicefeetothestateontheorderof$2per
transaction at least once a month. A better way of doing this calculation would
have been to use data from the live site specifically about the frequency of fee-
generating transactions (and amounts of the fees). These could then have been
adjusted to reflect the higher task completion rate for the redesigned site. If you