For:
Jeeves Solutions, a division of Ask Jeeves, Inc.
Dr. Max Henrion
&
INFOSYS 271 Final Project
Dr. Rashimi Sinha

Address Inquiries to:
{ jren, emilyl } @sims.berkeley.edu
School of Information Management and Systems
University of California, Berkeley
102 South Hall
Berkeley, CA 94720

APPENDICES
A Experiment Protocol | B Method Details | C More Results | D DataAnalysis | E Raw Data

Executive Summary

Comparative Digital Camera Advisor Study

Usability testing was used in this experiment to assess and compare the usefulness, usability, enjoyability, and trustworthiness of three digital camera advisor systems. 15 participants, who used all three sites to research digital cameras, tested available at AskJeeves.com, MySimon.com, and ActiveBuyersGuide.com. The different systems required users to submit their preferences or parameters in parallel or sequential processes.

This project was conducted for our client, Ask Jeeves, Inc., and the results that we have found are primarily focused on the assessment of the Jeeves Advisor system and how the features of the other two systems compared to those of the Jeeves system. Jeeves has succeed in creating an easy process but needs to improve on users’ perception of overall usefulness.

Pertinent Findings

Participants found that the Jeeves Advisor system was the easiest to use while MySimon’s Decision Guide provided the most useful information. 73% of the participants would consider using Jeeves’ system again in future digital camera research. Though many liked the questions at Jeeves’ site and thought the questions were appropriate, a number of participants wanted more questions to help address their needs.

Users also wanted more information, including more results, more detailed product information and more consumer reviews. MySimon’s Decision Guide put users through a long and sometimes complicated process and had the same underlying information as Active Buyer Guide’s Simple Search system (ABG). Users still felt that MySimon provided better explanation of unfamiliar features, had easier and more useful comparison tables, and overall contained much better information than both Jeeves and ABG.

Recommendations

Simple changes can be made to improve Jeeves Advisor.

· Make more results more easily accessible while reducing the number of results presented when users click for more products. This can improve users’ perception of the system’s usefulness without compromising its ease of use.

· Use text links for additional information on features. Users with limited digital camera knowledge can quickly learn and better understand their options.

· Allow users to access detailed product information individually from the result list instead of forcing users to compare products in order to view details. Many users want to quickly further verify how well Jeeves recommendations meet their needs.

More involved changes include

· Redesign the way users can add more than one item to the comparison table.

· Considermaking more of the decision process more transparent so users feel more in control of their preferences.

Long term considerations include linking products to more detailed product specifications and consumer review information.

Introduction

Product Description

Jeeves Digital Camera Advisor

Ask Jeeves’ Digital Camera Advisor was released on November 16, 2001 on the ask.com website. It aims to help consumers in their decision making when shopping for a digital camera. The entire product was evaluated from the view of a consumer using the advisor to find potential candidates for further consideration. The product solicits users’ preferences in a series of questions and then provides a list of digital cameras that satisfied user requirements.

The intended user population is part of the larger population that uses the ask.com website for various information needs. Ask.com users are general consumers, who tend to have less experience using the Internet, be female, and be younger consumers. This group of users prefers simple, easy to understand terminology and processes. Good system and process guidance is more critical to this group of users when interacting with more complex systems or processes.

The Jeeves Digital Camera Advisor is intended for use on personal computers in settings familiar to the consumer. The systems are generic PC’s at home, work, and locations such as libraries where systems are publicly available. The user would be someone considering the purchase of a digital camera for personal use, and is looking to for some information to help find candidates that meet their requirements and perhaps even make a purchase decision. The user may or may not have much experience with digital cameras. The user would also have various levels of experience researching and shopping for products on the Internet.

Objectives

The goals of this study are to:

· Determine the effectiveness of the Jeeves’s Digital Camera Advisor in meeting consumer needs

· Compare the Jeeves’ Digital Camera Advisor to two other similar systems for relative strengths and weaknesses in overall system usability and usefulness. All three systems should aim to help consumers find digital cameras that meet their requirements by using similar parameters and some way of obtaining user preferences.

· More specifically, evaluate and compare the user experience, overall interface, and satisfaction with results for all three systems

· Find areas of improvement in the Jeeves’ Advisor and ways to improve those areas

Methodology (Complete details can be found in appendix B)

Participants

A total of 15 participants were tested for this study. The participants have the following characteristics:

· 7 female and 8 male

· 6 are 18-24 years old, 3 are 25-34, another 6 are 45-54.

· 5 users have 2-5 years of Internet experience while 10 have more than 5 years of Internet experience.

· Only 4 users use the Internet a few times a week; all others (11) use the Internet on a daily basis.

· The average self rating on digital camera knowledge is 3.33, on a scale of 1-7.

Table 1 contains the demographic information of the study participants. The number of asterisks by each participant’s ID indicates the number of characteristics that matched the characteristics of the likely Ask.com users. The characteristic that is matched is bolded. Only two participants do not match any of the characteristics. One participant (No 2) matches 3 characteristics while three (No 5, No 18, No 10) matches 2.

Participant ID	Age	Gender	Internet Experience	Frequency of Internet Use	Dig. Camera Knowledge
1*	45 - 54	Female	5+ years	Every Day	2
2***	18 - 24	Female	2-5 years	Every Day	3
3*	25 - 34	Female	5+ years	Every Day	5
4	25 - 34	Male	5+ years	Every Day	2
5**	18 - 24	Male	2-5 years	Every Day	3
6*	45 - 54	Female	5+ years	A Few Times a Week	4
7*	45 - 54	Female	5+ years	Every Day	2
8**	18 - 24	Male	2-5 years	A Few Times a Week	4
9*	25 - 34	Male	2-5 years	Every Day	5
10**	18 - 24	Female	5+ years	Every Day	3
11*	18 - 24	Male	5+ years	A Few Times a Week	3
12*	45 - 54	Female	5+ years	A Few Times a Week	4
13*	18 - 24	Male	5+ years	Every Day	5
14	45 - 54	Male	5+ years	Every Day	2
15*	45 - 54	Male	2-5 years	Every Day	3

Table 1: Demographic Information of Study Participants

Experimental Design

Independent Variables

To explore the efficacy of Jeeves Digital Camera Advisor, we designed a comparison study whereby the independent variable is the product advisor system used.

The Jeeves Advisor employs a combination of user preference solicitation and feature based filtering to provide digital camera suggestions meeting user requirements. A number of commercial recommender systems intended to help users with product purchases are available on the Internet. Yet, few operational commercial systems are in the same category as the Jeeves Advisor.

Active Decisions provides two product guide/search interfaces that also solicit users’ preferences before giving users a list of digital cameras meeting their requirements. The Simple Search interface is a typical feature based filtering interface. Users can select a limited number of features all at once for quick results. Users can choose to go into the Advanced Search interface to select their preferences using a much longer list of available features.

The Decision Guide employs a more complex, sequential process for obtaining more detailed user requirements and preferences. After setting price range and selecting main features important to them, users need to complete 5 more steps each involving various number of settings and selections and inputting more information before any results were available.

All three systems express many similarities due to product category. They also contain several differences that were assessed and compared through this experiment. The question (or filter) processes, the research process associated with the result sets, and the quality of the result sets were the primary features studied. Questionnaires were used to capture users’ feelings and opinions about

· The amount and types of questions (or filters) that they were required to answer

· The terminology associated with the questions and results

· The way in which results were displayed

· The methods by which users researched and compared products from the result sets.

A pre-test background questionnaire was also used in order to determine if particular characteristics of participants using the systems affected their views on the systems.

Dependent Variables

The goal of this study was to determine what aspects of the three online advisors that participants liked and disliked and their reasons for liking and disliking the various aspects associated with the advisors.

The dependent variables for this study were:

· How easy each online advisor is to use

· How useful each online advisor is

· How much users trusted each online advisor

· How enjoyable each online advisor is

By obtaining users feelings about each system, this study worked to gain insight about what aspects of online advisors contribute to the system being easy, useful, trustworthy, enjoyable, and further, to determine which of these aspects are more important to users in deciding which system they liked the best.

Control Variables

Both the Active Buyers Guide (http://www.activebuyersguide.com) and MySimon (http://www.mysimon.com) provide Active Decisions’ Simple Search and Decision Guide interfaces to consumers at their site. To help users differentiate the interfaces, the Simple Search at Active Buyers Guide’s site (ABG) and the Decision Guide at MySimon were used for the study.

Each participant completed digital camera searches using all three product advisors. To minimize effect of fatigue, user learning and other bias due to the order in which systems were presented, the order of systems was varied according to the following matrix.

OrderName	System 1	System 2	System 3
A	ABG	Jeeves	MySimon
B	ABG	MySimon	Jeeves
C	Jeeves	ABG	MySimon
D	Jeeves	MySimon	ABG
E	MySimon	ABG	Jeeves
F	MySimon	Jeeves	ABG

Table 2: Experiment Design Matrix

Results

Main findings

Best Systems

Figure 1: Best Systems

Participants were asked to pick the system they felt were the easiest to use and the system that provided the most useful information. An overwhelming majority, 63% of the users tested, felt that Jeeves Digital Camera Advisor was the easiest system to use. However, only 13% thought it provided the most useful information. My Simon received the 53% of votes for the system that provided the best information while only 20% thought it was the easiest to use. The Active Buyers Guide Advisor (ABG) received 33% of participant votes for being the most useful and 13% for being the easiest to use.

Gender and Best Systems

Gender had a significant correlation with the voting on best information system. When picking the system with product information, 86% of women thought MySimon was the best and only 14% preferred Jeeves. 63% of the men thought Active Buyers Guide has the best information and 25% picked MySimon. (See Figure 2 below.)

Though correlation between gender and easiest to use system is not statistically significant, trends based on gender are also observed. Of the 7 women, 57% voted Jeeves Advisor as the easiest to use while 43% voted for MySimon. 75% of the 8 men voted Jeeves Advisor as the easiest to use while 25% voted for ABG.

User interface/interaction can significantly affect users of different gender and their perception of the final results quality. Results of both the MySimon’s preference based solicitation and Active Buyer’s Guide’s filter based solicitation contain exactly the same information (both from Active Decisions), the only difference between the display of information is the initial results page. ABG displays a list of results with shorter descriptions and less information for each item than MySimon’s. Results from MySimon also include percentage fit according to user preference.

None of the women thought ABG had the best product information while 63% of the men thought it had the best product information. Observer logs and users comments indicate that users who took the time to explore a system preferred MySimon’s information while users looking for quick results were more satisfied with the information provided by ABG.

Figure 2: Participant Vote for the Best Sites by Gender

Overall Ratings

Different aspects systems were evaluated on the basis of ease of use and overall usefulness. Trust in the information that a system provides as well as level of enjoyment while using a system were additional metrics by which the systems were evaluated. Figure 3 and its data table show the average user ratings in each of the four categories for the three systems. (Bolded items in the data table indicate the highest score.) Ratings on each aspect by user for each system can be found in Appendix C, Tables C1 - C3.

	Ease of Use	Usefulness	Trust	Enjoyability
ABG	4.67	4.29	4.62	3.89
Jeeves	5.43	4.03	4.43	4.49
My Simon	5.31	4.91	4.82	4.56

Figure 3 & Table 3: User Average Ratings for Each System

Jeeves Advisor again received the highest rating in ease of use at 5.43. Its lowest rating is in usefulness at 4.03. MySimon received the highest average rating in all other categories but ease of use. For the ease of use measure, qualitative results and the participants vote on best system are better indicators. Though only 20% of the participants selected MySimon as the System easiest to use and many complained about various aspects of the long process, the average rating shows that MySimon scored very close to Jeeves. The ratings that make up the ease of use category included ratings on terminology, ease of generating additional results, help needed to use the system, none of which focused on the overall process.

Ease of Use - Question/Filter Process

Most participants felt that the Jeeves online advisor was the easiest system to use. Several participants in this study reported a relatively low level of prior digital camera knowledge. Average knowledge rating was 3.33 on a 1 to 7 scale. Many explained their reasoning for choosing Jeeves as the easiest system as being related to the types and format of the questions it asked. Several participants noted that they liked the narrative format of questions in which certain features or the importance of certain features were briefly explained.

Participants also indicated that they did not have to put a lot of thought or effort into answering the questions asked by Jeeves. They did express frustration with the amount of time that they had to spend answering questions and researching features that they did not understand when using the ABG and the MySimon product advisors.

Ease of Use - Result Viewing Process

Several participants indicated that they felt the Jeeves advisor was the easiest advisor due to the nature of the questions. At the same time they expressed frustration about navigating through the results provided by Jeeves. Users appeared to be satisfied with the relevance of the first three cameras that were recommended to them. A number of users did not recognize that they could look at more pages of cameras and a few were frustrated with not being able to see more results. Still others felt overwhelmed by the much larger number of cameras displayed on the pages following the initial results page.

Participants expressed frustration with the ABG system when receiving zero or few results without being informed which of their selected preferences limited the results. Many users expressed sentiments of wanting the ABG system to return results even when they did not exactly match their preferences, as is done by MySimon. Users also wanted to be able to see which preference(s) they would need to alter in order to receive more results.

Usefulness

The participants in this study had very little previous knowledge about digital cameras. Many felt that the MySimon advisor provided them with the best information because it taught them the most about digital cameras. Users liked being able to access feature information easily. They thought that MySimon offered a lot of feature information, and they found it helpful that My Simon explained the importance of digital camera features. Further, while My Simon provided so many questions with explanations attached to each question, users felt that they had the opportunity to learn a lot more about digital cameras than they did with the other two sites. Many of the users who expressed such strong positive results about MySimon were women. The women participants tended to spend more time investigating camera features that they were unfamiliar with and consequently felt much more informed about digital cameras after having used the My Simon advisor. On the other hand, most of the men did not take the time to access feature information but later expressed frustration about being asked questions that they did not understand.

Input vs. Output

47% of users felt that the Jeeves advisor required an appropriate number of inputs. The rest, 53% felt that it did not provide enough questions to help them obtain desired results. Though the questions were considered appropriate, a number of users wanted to see more. No one thought Jeeves asked too many questions.

The actual number of questions/preferences is not always the number that users perceive. Figures 4 indicates that most users felt the ABG feature filtering system asked just the right amount of question, that Ask Jeeves asked too few questions, and that MySimon asked too many questions. In actuality, Jeeves advisor asks almost twice the number of questions (8) than the number of preferences (5) that users had to set for ABG. Variation between the types of questions/preferences asked by the two sites has likely contributed to the discrepancy.

Figure 4: Number of Results Returned

Significant correlations were seen between how users felt about the number of results returned and the enjoyability, usefulness, and trust factors for both the MySimon Decision Guide and the Jeeves’ Advisor. This indicates that users may tend to judge online product advisors largely based on how they felt about the number of products that the site recommended to them. Though some users do not want to see lots of results at first, they began to want more results and information that can be quickly scanned once they learned a little more about different features.

Wanting more results, however, does not mean users feel they need to answer more questions. Users were asked if they felt that the system that asked more questions, i.e. My Simon, returned better results than the other two systems. 67% said no. Quite a number felt that a number of the questions were not relevant to their needs even though they thought they had to answer something.

Stop Usage

Figure 5: Participants Who Would in Reality Stopped Using a System Without Returning

47% of participants indicated that in reality they would have stopped using Jeeves’ Advisor without returning. 40% and 33% users indicated the same for AGB and MySimon respectively.

Reasons for not continuing with Jeeves’ system include system performance, lack of consumer reviews and more detailed product information, not seeing the connection between preferences and results, and failing to use comparison table for more than 3 products. Number and relevance of results can be an issue for both AGB and MySimon. (See Table C8 in Appendix C for detailed explanations by users.)

More that Systems Can Provide

Participants were asked to provide their opinions about what features the online advisors could have provided, expanded upon, or emphasized more in order for them to make a more informed decision about the products suggested to them. User selections are summarized in Figure 6.

Figure 6: What Else Systems Could Have Provided

More users indicated that Jeeves could have provided more information about camera features, better descriptions of the camera features, and longer camera descriptions. This would help with making a more informed decision about the cameras that Jeeves suggested to them. A number of users also indicated that Jeeves, as well both ABG and MySimon, could have provided more accessible user and expert reviews in order for them to feel more informed about the cameras.

Future Use of System

73% of users indicated that they would use the Jeeves Advisor in the future. Their top 3 reasons for using the advisor include: 1) find products from different manufacturer, 2) find out more about products user knows and uses, and 3) find different products by brand. 73% of users also indicate that they would use ABG in the future for similar purposes. The only difference is that a higher percentage wants to use ABG’s guide to find out more about products user knows and likes. 87% of users said they would use MySimon in the future mostly to find products from different manufacturers.

Figure7: Participants who would use the system again in the future

Design Recommendations

Question Process

Make feature information links more visible to users.

Several participants indicated that they liked being able to view more information about product features in order to gain a better understand of the meaning behind question involving features they were unfamiliar with. While many participants used feature information text links on the My Simon and Active Buyers Guide sites, 3 out of 15 participants used i-button on the Jeeves site, and some even complained that they could not find more feature information.

Give users more control

Considermaking more of the decision process more transparent to users so users feel more in control of their preferences. Users did not feel they had enough control over the decision process, such as making sure that certain features take priority over others.

Result Viewing Process

Make it more clear to users that there are more camera recommendations that just the three visible from the first results page.

Many participants did not view camera information beyond the front page of results. When asked later if they realized that more than three results were returned, several of them said no. Users were unable to recognize that the link to ‘show me other products’ would give them further recommendations. It may be easier for users to have page numbered links at the bottom instead.

Reduce the number of cameras displayed on pages beyond the first page of recommendations.

Some users indicated feeling overwhelmed by moving from the front page of recommendations that contained three cameras to the second page which included a significantly greater number of cameras.

Allow users to view camera feature information for individual cameras rather than solely through comparison tables.

Several participants tried to click on the camera photos in an attempt to view more feature information for individual cameras. Participants indicated that they would like to be able to view more information than appears on the front page for individual cameras.

Allow users to add more than one camera to a comparison table at a time.

Users were frustrated that they had to take the effort to add each camera of interest to the comparison table one at a time.

Make product reviews or links to product review sites more visible.

Several participants indicated that one way in which Ask Jeeves could help them make a more informed decision about digital cameras is by providing user and expert reviews or a quick link that contains user and expert reviews about recommended cameras.