|
Quantitative
Results
Testers
We
tested the interface on thirteen testers. We kept the results
from twelve of those tested. Testers were taken first from
a pool of UC Berkeley students who had expressed interest
in participating in interface testing. The second round of
testers were masters and PhD students from the School of Information
Management and Systems. The testers were all graduate students
and ranged in age from 19 to 49. Half were men, half were
women. There was a range of experience with visualization
software. Three users did not know what visualization software
was, six had used visualization software at least once, and
two had heard of visualization, but had never used a tool.
None of the users considered themselves experienced users
of visualization tools.
TraveLite's
intended audience is tech savvy. They are comfortable researching
and purchasing online. They are interested in downloading
software and content to a portable device. We do not, however,
expect that they will be familiar with visualization tools.
If anything, then, our testers were more experienced and skilled
than we expect our customers to be. If the interfaces we tested
were confusing to our testing audience, then we would certainly
expect them to be as or more confusing to our commercial audience.
Results
Time
Overall,
the differences in task completion time between the Attribute
Explorer and the two forms-based interfaces were significant
- Attribute Explorer took more than twice as long to complete.
As
expected, we note that as task complexity increases within
a particular interface, mean task time increases. We speculate
that the improvement in task time between the Attribute Explorer
three-dimensional and six-dimensional tasks is due primarily
to the extreme variance located in the Attribute Explorer
three-dimensional task mean, variance which lessens in subsequent
tasks. It is important to note that the significance levels
of the difference between the forms-based interfaces and the
Attribute Explorer interface increase as the task complexity
increases, indicating a widening gap in task time. Task complexity
demonstrated a greater impact on task completion time when
using the Attribute Explorer, with the mean task time increasing
by a greater amount as task complexity increases. Measures
of task time on the forms-based interfaces demonstrated less
sensitivity to task complexity.
Confidence
Additional evidence that users were less effective using the
Attribute Explorer is found in the subjective measure of confidence
in the retrieved results for each query. Overall, users were
less confident in the results retrieved using the Attribute
Explorer interface than with either of the forms interfaces.
Task
complexity also demonstrated a significant effect on measures
of confidence in results. We see what may be a learning effect
as users of the Attribute Explorer report greater confidence
in results as they continue to use the interface while task
complexity increases. It is interesting to note that the Dynamic
Queries interface demonstrated practically no variance in
confidence measures, indicating that all testers reported
confidence in their results from the start.
Recall
and Precision
Interface also demonstrated an effect on both recall and
precision, indicating the Attribute Explorer interface impeded
users' ability to specify queries accurately. Using the Attribute
Explorer interface, measures of both recall (ratio of relevant
retrieved items to all relevant items ) and precision (ratio
of relevant retrieved items to all retrieved items) were 13%
lower than with either of forms-based interfaces tested. While
the difference in means between the interfaces for recall
and precision is not statistically significant, it is noteworthy
that there are differences between the interfaces. Because
the tasks were so rigidly controlled (the tasks were designed
to test the specification of queries, rather than the
formulation of queries) problems with recall and precision
are more likely effects of the user's understanding of the
interface than their understanding of the queries. Perhaps
a more telling number than the means of recall and precision
are the raw number of errors committed on each interface.
The number of imperfect scores using the Attribute Explorer
was eight, the number in each other the forms interfaces was
three. We think this signals a real problem for the Attribute
Explorer's ability to assist the user in forming accurate
queries returning useful results.
Effects
of Training
Given that the Attribute Explorer is a less-familiar interface,
we considered whether or not training would have an effect
on users' performance or evaluation of the tool.
While
it appears that training had an effect on the variance of
the three-dimensional task time using Attribute Explorer,
more stringent analysis demonstrated that the differences
in task times between the training and non-training groups
are not significant. It is interesting to note that aggregating
exploration time plus the first 3-dimensional task results
in a measure that is approximately equivalent regardless of
training, indicating that a consistent amount of time using
the interface may be required for initial comprehension. Likewise,
the presence or absence of training demonstrated no effect
on recall, precision, or confidence using the Attribute Explorer.
Post-interface
evaluation
Upon completing three tasks with an interface and prior to
moving on to the next one, users were asked to evaluate the
interface overall by completing a short survey. Results from
these surveys indicate that users preferred the Dynamic Queries
interface overall, giving it the highest marks among the three
interfaces on the measures of ease of understanding, usefulness,
and expected results.
Attribute
Explorer fared particularly poorly when compared to the other
two interfaces on the measure of ease of understanding, supporting
our contention that the Attribute Explorer interface was particularly
confusing for users.
It
is interesting to note that on the question of usefullness,
users rated higher those interfaces that provided immediate
feedback and more direct manipulation (Attribute Explorer
and Dynamic Queries), indicating that these have significant
value to users, despite the complications with the Attribute
Explorer interface.
Qualitative
Feedback
Post
Interface and Made-up Task questionnaires
Attribute
Explorer
On
the Attribute Explorer, most users were very confused about
the distinction between the select and deselect mechanism
(to remove items from the found set, the user clicked on the
criteria to deselect it). This was further compounded by the
fact that most users were also confused by the color coding
employed in the system. All found results appeared in white,
while the deselected items were gray, with items successively
darker as they matched more of the deselected criteria. Because
the users were not initially comfortable with this functionality,
they also spent more time in checking through their selections
to ensure that their results were correct.
In
the post-interface satisfaction survey, most participants
felt that they developed a sense of the entire data set, although
most responded that they were initially confused. For example,
some individuals did not initially realize that the attributes
appeared on more than one screen and others did not initially
understand which graphical elements were important to their
tasks, for example, one person responded that she felt it
was easy to understand but "some details bothered me, such
as the yellow line in the bar so it takes a while to know
what is really important information." Furthermore, most of
the participants responded that they did sense how changing
their constraints affected the queries but again this took
some time to understand, for example, one tester responded,
"yes, once I figured out that you need to click on what you
don't want (sorta counter-intuitive)."
With
the made-up task on Attribute Explorer, again users were somewhat
confused by the tool. In response to the effect the tool had
on creating, expanding or limiting their search, one user
wrote, "I saw I needed to be less restrictive, I knew that
before selecting an attribute because I could see in advance
there were no more white restaurants." However, most users
did not feel that this visualization changed their query significantly.
Most users indicated that they were satisfied with the results
although only half were confident that they had found all
the restaurants that matched their criteria. Basically, most
users were simply confused by this interface and were not
sure how to use it and/or how to understand the results.
Static
Form
With
the Static Form interface, most users could quickly perceive
the available constraints to choose from, however a few found
the long form and the amount of scrolling made it difficult
to track all the possibilities. Most responded that they could
somewhat understand how the constraints affected the query
but would have preferred to have the feedback immediate or
more easily accessible rather than having to return to the
previous page. In fact, one participant responded that "this
kind of task should be simple and quick, lack of immediate
feedback made me less certain about what I'm doing. And lack
of adequate visual cues made the form harder to navigate."
In
the questions following the made-up task, the response to
whether the tool affected their decision to change the query
was fairly divided. Half the participants responded no and
the other half did change their constraints to attempt to
change their results but expressed a desire for more control
while using the interface. One tester responded that 'I almost
wanted to expand the query just so that I would feel like
I was doing something more," while another replied that "This
interface made it much more difficult to see the new results."
Dynamic
Query Form
Participants
liked the Dynamic Query interface the best out of the three.
When first interacting with the tool, most users were pleasantly
surprised and pleased when they realized that changing constraints
immediately resulted in a changed results set on the same
page. All users could sense what they could choose from, although
some felt that it took some getting used to the interface.
One participant replied that 'It took a little getting used
to, but after trying a couple of things - especially combination
check marks in a particular category and comparing answers
with each answer set. I felt comfortable with the answer,"
while another said that "[I] especially like the way each
choice would immediately affect the results in the frame."
In asking whether users could sense how the query constraints
affected the query, all users said yes and one noted that
"I was also looking at - and found helpful - the debugging
notes." [Note: we have since incorporated this facility into
our design as a form of feedback]. Likewise, all users felt
that both of these qualities mattered with the Dynamic Query
interface. In fact, following using this interface, two users
realized the importance of how the constraints affected the
query, one replied that "yes - actually I think in the real
world, I'll put more emphasis on the constraint of things
more while conducting a real search task."
The
Dynamic Query interface also changed how users searched when
they had an opportunity to create their own task. One user,
in a response typical of most users, replied that "it helped
me play around with different possibilities." Another user
explained that, "It made me see clearly what the effect of
my adding or subtracting choices. I can use the information
to know what is the real meaning." All users expressed satisfaction
with the results although not all were as confident about
finding all the possible results.
General
Feedback ~ Post Test Questionnaire
Users
were only somewhat sure about how the interfaces worked. The
Attribute Explorer, as mentioned above, confused most users
and a few also had a difficult time with the other two interfaces
as well. Most users felt that their search did not affect
which interfaces they found most useful although a few did
say that the Dynamic Query form did help in altering their
queries, especially on the more complex searches. One user
felt that "The more constraints, the simpler the interface
should be so I can focus on the search and not the additional
[questions of]: is the tool working, am I making mistakes?"
In
all three interfaces a few participants felt uncertain about
the logic of the flow or placement of the attributes themselves
making the tools difficult to use in a search. Some also felt
that all constraints needed to fit on one page in order for
them to easily understand the range of possibilities and also
to avoid excessive scrolling. Most participants did not like
Attribute Explorer. They felt that the interface had too many
screens, the distinction between the two colors was confusing.
They also disliked selecting attributes they wanted to omit,
finding this action counter-intuitive. Some also disliked
the Static Form interface because it did not show immediate
feedback, how the restrictions affected the results.
In
response to the question of whether the Attribute Explorer
provided a sense of context or was just confusing, many did
respond that it was confusing, "too much information or too
much visual" in the words of one person. One person replied
that "Histograms were ok. They're not very useful for searching
(as opposed to browsing) where they are much more useful,
as they would be for qualifying a search) for just executing
a given task, the dynamic query is better." Another person
felt that "it got easier as I used it." Also, in response
to whether any of the interfaces frustrated them, almost half
responded with the Attribute Explorer and some also felt frustrated
with the Static Form for all the same reasons discussed above.
And
last, one tester wrote that "the AE has real possibilities
for comparison of attributes after results are retrieved,"
while another felt that the Attribute Explorer was useful
"maybe for those who don't really know what they want and
just want to wander around to see what is a good choice."
Overall, 10 participants choose the Dynamic Query as their
favorite interface, with one person choosing each the Attribute
Explorer and Static Form.
|
