Introduction

In this assignment you will be creating a set of relevance judgements for three queries. These judgements are being recorded at UCSD and stored in a database with judgements created by other students. The system keeping track of all this is called RAVE (Relevance Assessment VEhicle). In an exercise later in the semester, we will use these pooled relevance judgements in order to help us better understand how IR ranking and evaluation work.

This assignment will not be graded, but you will receive credit for doing it, and negative credit for not doing it. You don't have to turn anything in, however, since a record of your judgements will be kept at UCSD.

These particular relevance judgements should not be considered to be sensitive or personal; however, if after looking at the kind of information requested, you object to this information being recorded, please let me know and we'll work out an alternative.

The entire data collection procedure should take no more than 90 minutes. You don't have to do it all in one session.

Basic task

We will be working with three queries in specialized areas of artificial intelligence (AI). You will see a series of "documents" which consist of thesis abstracts restricted to the area of AI. For each document and each query you are to decide if that document is relevant to each query. Assume that the search is restricted to AI documents, and are the documents reflect specific aspects of AI.

The first time you execute RAVE, you will be shown a window with four main panes. The topmost pane contains one document abstract and its metadata (in light blue), the second pane contains two "standard" queries, the third pane contains buttons for navigation and for making relevance judgements, and the bottom-most pane contains the third query, which is an entire abstract and its metadata.

Spend a minute reading these queries and getting them in mind; these will remain constant throughout your session.

Two of the queries are of the form "I want documents ABOUT ..." and appear in the middle of the screen.

The third query is a little different: "I want to find OTHER DOCUMENTS LIKE THIS ONE." That is, the third query is in terms of another document, and you are to find documents you think are similar to it. This third query appears at the bottom of the screen and consists of a longer textual passage and will require more attention at first. It does not change as you proceed.

After having a clear conception of the three queries, read the document at the top of the screen and evaluate its relevance (see below for details). After you've done that WITH RESPECT TO EACH OF THE THREE QUERIES, click on the NEXT DOCUMENT button and evaluate its relevance in the same manner. This cycle will be repeated until you've evaluated all of the documents assigned to you, or you have worked for as long as you are expected.

Your relevance assessment need not be all or nothing. Most of the documents you see will NOT be relevant whatsoever to any of the three queries, and so the default relevance assessment is "NOT" (relevant). However, if you believe a document is relevant, you can respond according to the following three-point scale:

3: Critically (relevant)
2: Relevant
1: Possibly (relevant)

To make these assessments more concrete, CRITICAL documents are those that would be a major omission in a survey article on the query's topic. POSSIBLY relevant documents are those that might be of interest, for example as part of an exhaustive literature review. RELEVANT is between these two extremes, and should be used if you believe it is relevant to the query, but are unsure about qualifying it further.

After you have evaluated the current document WITH RESPECT TO ALL THREE QUERIES, click on the NEXT DOCUMENT button to view the next document.

You need not do your 1.5 hours of relevance assessments all in one sitting. If you click on the QUIT AFTER THIS ONE button, RAVE will remember where you left off and begin there next time.

To login to the system, use your SIMS account name (e.g., kkada) but do not add @sims.berkeley.edu