Augmenting organizational memory: a field study of Answer Garden

Mark S. Ackerman
ACM Transactions on Information Systems, Volume 16 , Issue 3 (1998), Pages 203-224

Summarised by Miriam Walker mwalker@cs.berkeley.edu

This article explores the concept of organisational memory and gives a summary of the importance of organisational memory. It describes the design and testing of the Answer Garden system as a tool for recording and transmitting organisational knowledge. While the study found a number of interesting social phenomena surrounding the asking and answering of questions, it left a number of unanswered questions. Answer Garden also seems to only cover a very small part of the space of organisational memory requirements.

Ackerman defines organisational memory as "organizational memory with persistence". Recording and transmitting of organisational knowledge has many practical short and long term benefits for individuals and organisations including:

Answer Garden provides a system to retrieve knowledge held within a database as well as the capability to ask an expert if the information can't be retrieved. The Answer Garden takes the form of a series of menus where users can narrow down the topic they wish to ask about. If the user doesn't find the answer to their question they can email an expert with a specific question. The responses of experts are used to 'grow' the database with the aim of increasing the accessability of organisational information while decreasing the repetition in questions answered by experts. This paper reports on a field study of Answer Garden using knowledge of XWindows as the domain of the database.

The choice of XWindows as a domain was interesting in that XWindows questions mostly have a single correct answer. XWindows also has very little organisation specific content and hence some of the potential advantages of studying the effect on organisational knowledge were not explored. One of the harder to achieve ideals of recording and communicating organisational knowledge is to preserve decision making processes within context as a source of information for future decisions.

This study found users became frustrated by the emptiness of the knowledge base compared to the size of the problem space. Ackerman suggests that a system like Answer Garden would work best for a small information area but small information areas would reduce the cross-fertilization of ideas across an organisation. In order to grow an Answer Garden it is important to integrate the system into work practices so that people query it more and store more knowledge. If the domain is too specific then the incentive to constantly use Answer Garden will decrease. However, reaching critical mass of information and then navigating that information is a big problem.

Interesting findings from the study include the fear of social stigma by experts. Experts have reputations for accuracy and thoroughness to protect and Ackerman found some expert answers to be more formal and extensive than desired by users. Experts were responsible for expanding the Answer Garden by adding in answers but it was a bit unclear how new categorisations of knowledge were determined. It would have been interesting to know how Answer Garden might have managed conflicting expert opinions and situations when there is no definitive answer.

Questions raised by Answer Garden

Is this menu based interface the most appropriate navigation method?

Answer Garden seems more like a well organised FAQ with the advantage that experts are appointed to update the FAQ as additional information is required. The advantage of the structure of Answer Garden is that it is hierachically organised to provide answers to those who might not be quite sure how to phrase their question but this was not specifically examined. And because navigation is by a choice in each menu there is no risk of a search returning no result. The Answer Garden interface could be compared with natural language queries (such as those used by Ask Jeeves) to see what sort of search mechanism is most successful for answering user queries. Certainly the tree type structure that Answer Garden presented seemed less useful for finding multiple opinions, as the database grew it could become time consuming to narrow down the field of interest.

How is quality and currentness of information monitored? How can historical information be maintained?

Over time some information will be superceded, and someone may have to decide if outdated information should be purged or retained. Who would be qualified to make such decisions if the original experts have moved on? Should information that is outdated be retained for historical interest? If superceded information is retained, how can users place historical information within the correct context and not treat it as current? How can information quality be monitored? Bad data can't easily be detected unless responses are monitored by other experts. Ackerman suggested that overall leaders could prune and maintain the database but this role would require metaexperts who knew about every domain within the Answer Garden. There is a risk that experts would simply try and put a manual for XWindows in each response to reduce incidence of related questions . It is important to monitor the database to keep the answers small, readable and specific so that users find the responses time efficient to use. Ideally every response should be used sufficiently often to warrant a little extra expert time in writing the answer well.

How is expert status assigned, maintained, shared and monitored?

In this study expert status was preassigned, but to preassign expert status requires someone to monitor who is an expert for various fields. For organisations with high staff turnover, the available experts for particular fields constantly change so there is a need for maintainence of a list of areas of expertise or an open forum to which questions can be addressed. Identifying expertise is a hard problem as the relevant knowledge for any project may be distributed amongst members of the project group and the preservation and unification of this knowledge is an important component of augmenting organisational knowledge. Hence any model that presumes as single expert can be queried is likely to run into trouble. Any system that appoints experts formally is going to have to assess the ongoing incentives for experts and there may be a realisation that anonymous queries are less compelling than a visit or email from a named person within an organisation. Inversely, asking anonymously will decrease the incentive to carefully search the Answer Garden before asking an expert. It is likely that the prestige of being able to display expertise will be sufficient incentive and Answer Garden presupposes that experts will provide answers in order to reduce repetition of questions and reduce further workload.

Additional readings:
An article by Franz Lehner detailing theories of organisation memory
What Do We Know About organisational Memories and Can Current Theories Contribute to the Development of Knowledge Management Systems?
http://www.sistemi-informativi.org/kio/cons/Lehner.html

A talk by John Lowe that explores some of the issues with how people might query a system like Answergarden if they could just ask a question in natural language form.
What’s in store for question-answering? Prognostications based on corpus analysis of several hundred million questions
http://www-csli.stanford.edu/~schuetze/lowe.html

An article by E. Jeffrey Conklin that explores requirements for organizational memory with an emphasis on systems for preserving and communicating project based knowledge.
Designing Organizational Memory: Preserving Intellectual Assets in a Knowledge Economy
http://www.gdss.com/wp/DOM.htm