A USABILITY COMPARISION OF

Sun StarOffice Writer 5.2 vs. Microsoft Word 2000

Katherine Everitt and Scott Lederer

November – December 2001

December 10, 2001

Katherine Everitt and Scott Lederer

For:

is271: Quantitative Methods
School of Information Management and Systems
University of California, Berkeley

Dr. Rashmi Sinha, Instructor

Address inquiries to: Scott Lederer, Katherine Everitt

Email: {lederer, everitt}@cs.berkeley.edu

Address: UC Berkeley, Computer Science Division, Soda Hall, Berkeley, CA 94720

PDF Version (1.2 MB)

Executive Summary
Introduction
Method
Results
Design Recommendations
Appendicies

Executive Summary

A set of usability comparison tests between the Sun StarOffice Writer and Microsoft Word word processors show most experienced computer users prefer Microsoft Word and find it easier to use. Suggestions for improving StarOffice Writer are provided.

StarOffice Writer 5.2 is the word processor packaged in StarOffice 5.2, Sun Microsystems’s freely available entry into the desktop office suite market. Microsoft Word 2000 is the word processor component of Microsoft Office 2000, the desktop office suite market leader.

Twelve participants completed the usability comparison test. All participants were computer owners and had been using computers longer than five years. 75% has been using Microsoft Word longer than five years, and 92% longer than two years. 17% had used StarOffice Writer before but used it rarely.

Each participant was presented with a hard copy of a short document, the creation of which required the use of a representative set of standard word processor functions. Each participant recreated a reasonable onscreen copy of the document using both of the tested products.

Captured metrics include user interface interactions, errors, indications of frustration, task completion times, and opinions on ease of use.

Participants experienced difficulty inserting tables and pictures into StarOffice Writer documents.

While overall ratings for both products were comparable, participants were more comfortable and satisfied with Microsoft Word and found it easier to use than StarOffice Writer.

Introduction

Full Product Description

The usability of StarOffice Writer 5.2 and Microsoft Word 2000 were evaluated and compared.

The following features of both products were evaluated:

· Text formatting

· Bullet lists

· Tables

· Picture insertion

· Spell check

· Saving and loading files

Both products are intended for use by personal computer users for the composition of letters, reports, and other documents.

Test Objectives

The objectives of the test were:

· to compare the usability of the two products with regard to commonly used word processing functions

· to determine if differences in usability have any bearing on which product someone would be more likely to purchase

· to suggest improvements for future versions of StarOffice Writer.

Users interacted through a mouse and keyboard with the default menus, toolbars, and editing space presented by each product’s user interface. The specific functions accessed by the users were text entry, text formatting (bold, italics, underline, alignment, font style, font size), bullet lists, tables, picture insertion, page breaks, text editing (copy, paste), spell check, save file, load file, help systems, undo, redo. The use of any other functions was statistically insignificant.

The above set of functions was chosen as representative of the word processing functions most commonly used for the creation of simple documents.

Method

Participants

Twelve (12) participants were tested. All participants were computer owners and had been using computers longer than five years. 75% has been using Microsoft Word longer than five years, and 92% longer than two years. 17% had used StarOffice Writer before but used it rarely. 75% of participants (9) were computer science graduate students. 92% (11) use computers as an integral part of their work. Participants were selected based on accessibility rather than particular demographic considerations. Further study would benefit from a more random sample, preferably composed of equal portions of subjects with Microsoft Word and StarOffice Writer experience.

Traditionally, word processors are marketed as tools for office workers, but their true user group includes virtually all personal computer users. According to the traditional criterion, only one participant (8%) was representative of the user population. According to the more liberal criteria, 100% of the sample represented certain demographics of the user population.

Participant characteristics:

ID	Age Range	Gender	Owns a Computer	Years of Computer Experience	Years of MS Word Experience	Years of Star Writer Experience	Frequency of Computer Use	Frequency of MS Word Use	Frequency of Star Writer Use	Usual Word Processor	Preferred Word Processor
1	25-34	F	Y	5+	5+	0	Daily	Weekly	--	Word	Word
2	25-34	M	Y	5+	5+	0	Daily	Daily	--	Word	Word
3	18-24	F	Y	5+	5+	<1	Daily	Weekly	Yearly	Word	Word
4	25-34	M	Y	5+	5+	0	Daily	Weekly	--	Word	Word
5	25-34	M	Y	5+	5+	<1	Daily	Daily	Yearly	Word	Word
6	25-34	F	Y	5+	5+	0	Daily	Weekly	--	Word	Word
7	18-24	F	Y	5+	5+	0	Daily	Weekly	--	Word	Word
8	25-34	M	Y	5+	3-4	0	Daily	Weekly	--	Word	Word
9	18-24	F	Y	5+	5+	0	Daily	Weekly	--	Word	Word
10	18-24	M	Y	5+	3-4	0	Daily	Daily	--	Word	Word
11	25-34	M	Y	5+	5+	0	Daily	Weekly	--	Word	Word
12	55+	M	Y	5+	1-2	0	Daily	Monthly	--	Lotus WordPro	Word

Context of Product Use in the Test

Tests were performed at those locations and times most opportune for participants. The authors have no reason to believe the physical or temporal context of the tests was significantly different from the expected context of the products’ use. However, there is a potentially significant difference in operational context, in that each test was partitioned into a set of discrete, sequential tasks divided by brief question and answer sessions, whereas the normal document creation process is typically less structured and may involve parallel task completion, breaks of different lengths or natures, and other differences.

Tasks

Participants were presented with a hard copy of a short document (see Appendix B), the original creation of which required the use of a representative set of standard word processor features. Each participant created a reasonable copy of the document using each product, saved it to the hard disk, and loaded another document from the hard disk. This entire process consisted of eight (8) tasks and was performed once per product (hence, twice per participant). The tasks were performed sequentially in fixed order. Once a task was complete, participants were not allowed to return to it.

1. Enter a sentence and format it (bold, underline, center alignment).

2. Create a bulleted list of sentences.

3. Create a table of two rows by three columns with short, simply formatted text in the cells.

4. Insert an arbitrary picture file.

5. Select and copy a block of previously entered text from a separate text editor and paste it into a new page of the main document. Format it to look like the hard copy (bold, underline, italics, alignment).

6. Spell check the document.

7. Save the document to the hard drive.

8. Load the document the participant created with the other product and judge this product’s visual presentation thereof.

The authors designed these tasks to require word processor functions commonly used in the process of document creation. Two guides were used in determining such typical functions. First, the authors have extensive experience writing and reading documents and identified functions typically used in document creation. Second, assuming word processor developers populate the default toolbars with widgets for commonly used functions, the tasks were designed to require the use of the functions accessible on the default toolbars of Microsoft Word 2000, the word processor market leader.

Participants were told not to attempt to recreate a perfect copy of the document, but to apply a reasonable effort to its recreation. Each task was considered complete when the participant vocally stated so.

Tasks were clearly defined, both visually and vocally, for participants. 92% of participants had experience using Microsoft Word to perform tasks reasonably similar to those in the test.

Participant’s Computing Environment

Participants performed all tasks using an IBM ThinkPad T21 notebook computer running Windows 2000 Professional. All configuration options for both products were set to their default settings.

Display Devices

The 14-inch LCD display was set to a resolution of 1024 x 768 with 65,535 colors.

The hard copy of the original document was printed on 8.5” x 11” paper at 600 dpi.

All toolbar buttons were set to their default size.

Manual Input Devices

Participants used the computer’s embedded keyboard and a standard mouse.

Test Administrator Tools

Test administrators conducted orderly tests through use of a printed script (see Appendix C) which explained the sequence of instructions and tasks.

Test administrators recorded all data on a printed worksheet (see Appendix D), using a stopwatch to record task completion times.

Experimental Design

Each participant was given a printed copy of a simple document (see Appendix B) and asked to create two onscreen copies of the document, once using StarOffice Writer 5.2 and once using Microsoft Word 2000. The document was divided into a series of five easily identifiable sections:

Page One

1.A line of heavily formatted text

2.A bulleted list of sentences

3.A table

4.An inserted picture

Page Two

5.A formatted document

The first five of the eight tasks composing the test corresponded directly to the copying of the five sections of the document. The sixth test involved spell checking the document. The seventh involved saving the document to the hard drive. The eighth involved loading the document created with the other product and judging the current product’s conversion and visual presentation thereof.

Two (2) testers facilitated the test with twelve (12) participants. To control for tester influence and participants’ tendency toward improved performance on the second product due to familiarity with the task after the first, the following procedure was followed. Half of the participants (6) were tested by one tester, and the other half (6) by the other tester. Half of each tester’s set of participants (3 per tester; 6 total) used StarOffice Writer to create the first copy of the document and Microsoft Word to create the second. The remaining participants (3 per author; 6 total) used Microsoft Word to create the first copy of the document and StarOffice Writer to create the second.

Testers recorded (see Appendix D):

· task completion time

· count of top-level menu accesses

· count of toolbar accesses

· count of keyboard shortcuts used

· count of errors, defined as:

o looking for a function on the wrong menu

o undoing an operation

o canceling a function choice

· count of frustration events, defined as:

o explicit expression of frustration

o obvious bouts of frustration

o giving up on a task

o negative exclamations

· count of help menu accesses

After each task, participants rated the ease with which the task was completed and the accessibility of the functions necessary to complete the task.

Participants filled out a series of questionnaires (see Appendix A):

· Participant background questionnaire

· StarOffice Writer usability questionnaire, after copying the document using StarOffice Writer

· Microsoft Word usability questionnaire, after copying the document using Microsoft Word

· Product comparison questionnaire, after using both products

Independent variables

· Each task

· The functions necessary to complete each task

· The product used to complete each task

Control variables

· The order in which the products were used to complete the tasks

· The tester

Measures

· Task completion time

· Count of interface interactions (not including raw text entry)

· Count of errors

· Count of frustration events

· Product ease-of-use

· Comfort of using product

· Satisfaction of using product

· Frustration of using product

· Boredom of using product

· Product overall rating

· Accuracy with which product translates and displays document created in other product

· Likelihood to replace usual word processor with StarOffice Writer

· Likelihood to buy one product over the other

Procedure

The tasks had no time limit. Participants were allowed to, and sometimes did, give up on tasks they found too frustrating or difficult to complete.

Participants were asked to speak freely and to think aloud throughout the test. Testers answered operational questions about the test, but refrained from answering instructional questions related to completion of the tasks.

Participants were given a $12 gift certificate to a local bookstore as compensation for their participation.

Participant General Instructions

Full instructions given to participants can be found in the script in Appendix C.

While the tests did not involve anyone other than the participants and testers, the former were permitted to freely interact with others in their general vicinity. The few times this happened, the interactions were brief and contributed no significant effect to the conducting of the test.

Participant Task Instructions

For each task, participants were instructed to make a best reasonable effort to create an onscreen copy of the section of the hard copy document corresponding to that task. They were asked to vocally state when they were beginning the task, to think aloud during completion of the task, and to vocally state when they ended the task.

Usability Metrics

Product effectiveness was measured by error counts, frustration event counts, and participant questionnaires (see Appendix A).

Product efficiency was measured by task completion times, interaction counts, and error counts.

Product satisfaction was measured by participant questionnaires (see Appendix A).

Results

Data Analysis

Data Scoring

The data gathered falls into distinct categories.

Background and demographic information was rated according to the demographic variable in question.

Data	Scoring
Age	<18	18-24	25-34	35-44	45-54	>55
Gender	Male	Female
Owns a computer	Yes	No
# years using computers	0	<1	1-2	3-4	>5
# years using MS Word	0	<1	1-2	3-4	>5
# years using StarOffice Writer	0	<1	1-2	3-4	>5
How often use a computer	Never	Daily	Weekly	Monthly	Yearly
How often use Word	Never	Daily	Weekly	Monthly	Yearly
How often uses Star	Never	Daily	Weekly	Monthly	Yearly
Usual word processor	Word	Lotus WordPro
Word processor of choice	Word

Information based on questionnaires administered after each trial were rated on a 5 point Likert scale.

Data	Scoring
Rating of Ease of Use	Extremely Difficult	Somewhat Difficult	Neutral	Somewhat Easy	Extremely Easy
Experience was: Comfortable Satisfying Frustrating Boring	Extremely Inaccurate	Inaccurate	Neutral	Somewhat Accurate	Accurate
Overall Rating	Terrible	Poor	Average	Good	Excellent

Each task was timed in seconds. Each menu item accessed, toolbar button used, keyboard shortcut performed, error made, and frustration voiced were counted. After each task, the subjects were asked if they had ever performed a reasonably similar task before using the same product (True or False), if the actions necessary to complete the task were obvious to them (Extremely, Very, Somewhat, A Bit, Not at All), and how easy the task was to complete (Extremely, Very, Somewhat, A Bit, Not at All).

Statistical Analysis

Our main performance metrics were timing, errors and interaction style. We timed each task for our subjects and compared this metric in seconds between the tasks. We also summed the time taken to complete our entire sample document. For our subjects, the overall time for Word was faster than that for StarOffice Writer (t(10) = -3.545; p <= 0.05). This is a significant. However, it should be noted that our user population was familiar with Word and mostly unfamiliar with StarOffice Writer. The total time to complete the tasks in Word was correlated with how often the subjects use Word outside the context of the test (r(10) = -0.689; p<0.05)

Many of the tasks for StarOffice Writer took comparable time to performing them in Word, while relatively few were significantly slower. The tasks where StarOffice Writer took significantly longer were those requiring the subject to create a table (t(11) = -2.805; p<.05) and insert an image (t(10) = -2.84; p<0.05). In the file-loading task, Word took longer (t(11) = 2.846; p<0.05). In the other tasks, there was no significant difference in speed between the applications. The total time to complete the tasks in Word correlated with that in StarOffice Writer (t(11) = .736; p<=0.01).

Our error metric was composed of errors such as menu misses (selecting the wrong menu for a function), having to undo an action performed by a product, and other mistakes in the document. The number of errors made between the applications was not significant (t(11)= -1.317; p= .215), but there were significantly more frustrations voiced about StarOffice Writer (t(11) = -2.728; p <0.01) than Word. This suggests that our subjects were more frustrated by the actions required to perform the tasks in StarOffice Writer. The time taken to perform all of the tasks in StarOffice Writer correlated with errors made (r(10) = .731; p< 0.05) and with frustration (r(10) = .823;p <0.01).

Our final performance metric was interaction style. We were interested in seeing which way to perform operations were the most efficient. The total time to do all tasks in Word was correlated with the number of menus accessed (r(10) = 0.862; p < 0.01), so the people who used more menus took longer at the task. Total time was also correlated with frustration (r(10) = .875; p<0.01). This makes sense, as frustration with the task could increase over time. The total time in Word to do tasks was negatively correlated with the number of keyboard shortcuts(r(12) = -0.586; p< 0.05), so the more keyboard shortcuts the less time the task took. These trends were not significant in StarOffice Writer.

Another interesting interaction trend in Word was that menu navigation correlated with errors (r(10)= .937; p<0.01) and frustration (r(10) = .875; p<0.01).

As Word and StarOffice Writer appear very similar, we were interested in whether or not subjects would keep the same interaction styles as they moved between products. Word menu navigation correlated with StarOffice Writer menu navigation (r(10) = .695; p <0.05); people who used high numbers of menus in one product did so for both. In addition, Word toolbar button use was strongly correlated with toolbar button use in StarOffice Writer (r(10) = .929; p<0.01). Keyboard shortcut use was also correlated between the two (r(10) = 0.707; p<0.05). This suggests that people tend to employ the same functionality when using the different applications. Another interesting observation is that toolbar button use in Word was correlated with keyboard shortcut use in Word (r(10) = .695; p<0.05 ).

Subjective – opinions

Our subjective measures were surveys taken before and after the survey, after each task and after each application experience.

In general, people felt Word was easier to use (t(11) = 3.023; p < .05). They also felt more comfortable with Word (t(11) = 3.957; p< .01) and they felt more satisfied with Word than StarOffice Writer (t(11) = 3.189; p<.01). However, there was no significant difference in frustration (t(11) = -1.758; p = 0.107), how boring the program was (t(11)= -1.731; p =.111) or overall rating (t = 2.171; p =.053).

The most interesting statistic here is the overall rating for StarOffice Writer was not significantly different from Word. These questions were asked after the subjects had an opportunity to use StarOffice Writer, which 83% had not had not previously done.

A series of questions were asked at the end of the survey about whether the subject would buy Word or StarOffice Writer, given different pricing levels (See Appendix A). Which product people would buy was not significant one way or the other (r(10) = 1.562; ns). We found this interesting, as 11 out of 12 people in our sample used Word as their primary text editor, and 100% selected it as their first choice before the test. The product they would choose was not correlated with their frustration with either program or the time taken.

Presentation of Results

Performance Results

Timing Results (measured in seconds)

Word:

	Subjects
Times	1	2	3	4	5	6	7	8	9	10	11	12
Total Time	620	268	764	375	489	611	494	643	524	615	691	1438
Task 1	56	34	63	41	21	42	56	42	46	47	13	172
Task 2	132	71	139	65	124	190	152	195	207	80	242	180
Task 3	56	36	105	52	84	105	72	82	45	170	236	399
Task 4	42	18	40	38	47	23	50	62	40	51	39	112
Task 5	178	38	100	60	86	79	96	210	70	202	60	283
Task 6	2	9	20	14	15	23	23	12	50	20	43	115
Task 7	37	6	35	12	28	39	32	30	33	25	33	62
Task 8	117	56	262	93	84	110	13	10	33	20	25	115

StarOffice Writer:

	Subjects
Times	1	2	3	4	5	6	7	8	9	10	11	12
Total Time	860	362	591	453	560	780	719	1024	1056	1293	602	1347
Task 1	56	37	28	53	15	84	50	41	62	38	29	86
Task 2	194	97	166	92	101	169	129	248	375	147	156	279
Task 3	304	51	217	122	275	172	242	462	137	784	135	398
Task 4	97	31	60	41	62	108	72	-	170	70	58	309
Task 5	94	87	57	88	57	142	101	170	160	162	102	151
Task 6	18	14	29	18	7	10	23	31	80	32	45	74
Task 7	88	12	26	19	29	53	92	59	60	40	59	25
Task 8	9	33	8	20	14	42	10	13	12	20	18	25

Interactions

Word

	Subjects
# of interactions	1	2	3	4	5	6	7	8	9	10	11	12
Menu	8	5	11	5	8	4	2	8	9	7	16	25
Toolbar Buttons	8	7	15	7	8	16	22	14	33	21	30	13
Keyboard Shortcuts	10	9	13	12	12	2	3	2	1	3	1	0

Star

	Subjects
# of interactions	1	2	3	4	5	6	7	8	9	10	11	12
Menu	16	5	13	9	12	11	6	13	10	7	10	19
Toolbar Buttons	11	10	14	10	11	14	15	14	35	19	25	12
Keyboard Shortcuts	9	10	7	3	13	3	7	5	1	3	1	0

Errors

	Subjects
	1	2	3	4	5	6	7	8	9	10	11	12
Errors In Word	4	5	12	1	2	0	2	5	8	12	16	31
Errors In StarOffice Writer	8	2	10	4	5	7	14	22	15	20	16	14

Subjective Results (1 = low; 5 = high)

Word

	Subjects
	1	2	3	4	5	6	7	8	9	10	11	12
Ease of Use	5	5	5	5	4	5	5	5	5	4	3	3
Comfort Rating	5	5	5	4	5	5	5	5	5	4	4	2
Satisfaction Rating	3	4	5	4	5	4	5	4	4	3	2	3
Frustration Rating	1	2	1	1	2	1	1	2	3	2	4	4
Boredom Rating	4	3	5	3	5	1	3	3	1	3	3	2
Overall Rating	5	4	4	5	4	4	4	4	4	4	3	3
Accuracy: Converting Star Document	5	5	5	1	5	5	5	4	4	4	5	5

Star

	Subjects
	1	2	3	4	5	6	7	8	9	10	11	12
Ease of Use	2	4	4	4	4	3	4	3	2	3	5	2
Comfort Rating	3	4	4	4	3	2	5	2	2	2	4	2
Satisfaction Rating	3	3	2	4	4	1	4	2	2	1	3	3
Frustration Rating	5	2	4	1	2	2	1	4	3	4	2	4
Boredom Rating	4	5	5	3	5	1	3	3	4	3	3	3
Overall Rating	3	4	3	4	4	4	4	3	4	3	4	2
Accuracy: Converting Word Document	5	1	5	5	5	1	5	5	4	2	4	1

Likelihood to replace usual word processor with StarOffice Writer

(1=Extremely Unlikely; 5 = Extremely Likely)

	Subjects
	1	2	3	4	5	6	7	8	9	10	11	12
Likelihood to switch to Star	1	1	1	1	3	2	2	2	2	1	4	1

Direct product comparisons

Subjects

1

2

3

4

5

6

7

8

9

10

11

12

Product more likely to purchase (averaged over various price differences)

Star

Word

Word

Word

(tie)

Word

Word

Word

Star

Word

Star

Word

Product found easier to use (averaged across tasks)

Star

(tie)

Word

Word

(tie)

Word

Word

Word

Word

Word

(tie)

Star

Qualitative Results

Comments and observations of participants reveal the following qualitative information about StarOffice Writer.

· Help Agent (a small floating window that pops up when the application infers the user may benefit from assistance)

o Participants find the Help Agent irritating.

o The button to stop operation of the help agent is too small.

o Textual help is not enough; imagery or examples should also be provided.

· Graphics

o Participants were confused by references to images and pictures as “Graphics”.

· Bullets

o Bullets are presented within a small gray background area. Participants were confused by this, spending considerable effort attempting to discern whether the gray background would appear in printed copies.

· Tables

o When part or all of a table is selected, the alignment buttons on the Object toolbar are replaced with table-related buttons. This caused much consternation for users attempting to align text within a table.

o When accessing the Help system during table composition, the help system provided assistance related to database tables, not standard word processing tables. This appears to be related to the tight integration of the many components of StarOffice, including a database management system.

Suggestions for Future Versions of StarOffice Writer

The following suggestions for improving the StarOffice Writer user interaction process are based on the results of the test.

· Help Agent

o Turn the Help Agent off by default.

o Make the Help Agent simpler to disable.

o Unclutter the interface to the Help Agent.

· Help System

o Provide highly context-sensitive searches when a user access the help system, so, for example, searching for help when editing tables in Writer does not result in help files related to database tables.

· Graphics

o Streamline the process for inserting an image, possibly by assigning this function to a toolbar button. On average, participants took over twice as long to insert an image in StarOffice Writer than in Microsoft Word.

o Rename the Graphics insertion function to “Picture” or “Image”.

o Include a clip art gallery and wizard.

· Bullets

o Disable the gray background for bullets by default.

· Tables

o Streamline the process for adding tables. On average, participants took over 2.25 times as long to insert a table in StarOffice Writer than in Microsoft Word.

o Make text alignment and formatting buttons persist on the Object toolbar when editing tables.

o Provide context-sensitive (right-click) menus populated with complete table editing functions when editing tables.

Appendices

Appendix A – Participant Questionnaires

Background Questionnaire

1.