SIMS 202, Assignment 8:
Computing Precision and Recall using Different Notions of Relevance

Due November 19, 1998

Introduction

In this assignment we are going to bring together many of the concepts and assignments from the second half of the course.

The main idea is to compute and compare precision and recall for two different similarity algorithms according to two different measures of relevance. For more background, see Belew Chapter 4 and the Bellew & Hatton RAVE paper in the reader.

The queries that you assigned relevance judgements to in Assignment 5 have been run through a program called match that ranks the documents according to similarity to the query using a cosine measure and tf.idf weighting (from section 3.4.4 of the Belew book, page 340. Short query weighting is used on short queries, long query weighting on long queries). One version of the algorithm normalizes the document length as part of the ranking, the other doesn't do the normalization.

Relevance judgements for all the students have been pooled to create a single relevance assessment for each query,document pair. Two types of relevance have been assigned:

Conservative relevance is defined as follows:

If the document's average relevance score (on the 4 point scale) is >= 1, it is relevant, otherwise it is not relevant.
Generous relevance is defined as follows:
If the document gets a non-zero score from at least one assessor, it is relevant.

The Data Files

Alison emailed you a file containing the relevance judgements you produced in Assignment 5. Recall that the data looks like the following:

  abrandt  09/15/98  05:52:03   3 4386 (6 1) (1 0) (9 3)

with the format username date time feedback where "feedback" is of the form:

   response# document# (query# rating) (query# rating) (query# rating)

Most people produced relevance judgements for only three queries.

You'll also need to make use of some other sets of files. The file

  /groups/is202/handouts/queries

shows the original queries and their query ids. In the directory

  /groups/is202/handouts/reljudgements/

you'll see two types of files. The files rels.conservative.queryid show all of the documents that were looked at for queryid. They also show the pooled conservative relevance judgement for the query, document pair. Similarly, the files rels.generous.queryid show all of the documents and their judgements according to the generous relevance assumption.

In all these files, 1 means relevant, 0 means nonrelevant.

In the directory

  /groups/is202/handouts/rankings/ 

you'll see four kinds of files. These contain the top 30 documents according to two different ranking algorithms:
• standard match
• standard match with document length normalization
(In a few cases only the top 15 documents are shown.)

The files in this directory correspond to the following information:

• ranking.con.queryid: the top-ranked documents without normalization using conservative relevance judgements.
• ranking.gen.queryid: the top-ranked documents without normalization using generous relevance judgements.
• ranking-norm.gen.queryid: the top-ranked documents with normalization using conservative relevance judgements.
• ranking-norm.gen.queryid: the top-ranked documents with normalization using generous relevance judgements.

The documents (along with their document numbers) can be viewed, as before, in the files:

  /groups/is202/handouts/data/ait/ait*.t

Another way to see the documents that correspond to the results of the query is to type the query into the interface at: http://corea.ucsd.edu:8000 This shows the ranking produced by the match algorithm and a hyperlink to the document itself. The text of the original queries can be found in the queries file.

Computing Precision and Recall

Choose one short query (from queries 1-8) and one long query (from queries 9-11, where an abstract was used as a query) that you produced relevance judgements for.

For both of these queries, do the following:

(1) State which query it is (number and the terms used if one of the short queries).

(2)

(a) What is the minimum number of documents that would have to be retrieved in order to achieve perfect recall in the conservative case (in other words, how many relevant documents are there for this query)?.
(b) What is the minimum number of documents that would have to be retrieved in order to achieve perfect recall in the generous case?

(3)

(a) How many documents did you mark as relevant with a score of 2 or larger? Call this your conservative judgement. (Note this is different than the definition of conservative and generous used for the pooled judgements.)
(b) How many documents did you mark as relevant with a score of 1 or larger? Call this your generous judgement.
(c) How often was the conservative group judgement in disagreement with your positive conservative assessments?
(d) (If applicable) Look at a couple of these cases (look at the documents involved) and try to assess why you disagreed.
(e) How often was the generous group judgement in disagreement with your positive generous assessments?
(f) How often did the conservative group judgement say a document was relevant when you gave it a zero relevance judgement.
(g) (If applicable) Look at a couple of these cases (look at the documents involved) and try to assess why you disagreed.

(4) For this question, choose recall levels that are evenly spread out, ending with the largest recall score possible given that rankings for only the top 30 documents are supplied. For example, it would be interesting to see precision at 20% recall, 40% recall, 60% recall, 80% recall, and 100% recall (but this might not be possible given the data).

(a) Compute the precision at five different recall points for the normalized matching algorithm using the pooled conservative judgements.

(b) Compute the precision at five different recall points for the normalized matching algorithm using the pooled generous judgements.
(c) Compute the precision at five different recall points for the normalized matching algorithm using your own personal generous judgements.
(d) Compute the precision at five different recall points for the non-normalized matching algorithm using the pooled generous judgements.
(e) Plot all four on the same graph. (Simply connect the different points by drawing a line between them; there are methods for smoothing the plots but we won't worry about them here. Pen and paper is fine.)

(5) Discuss how and why the plots in (4e) are similar to or different from one another.

(6) Discuss whether or not you think precision and recall are effective means to evaluate the ranking of this query.

Last modified Nov 10, 1998 MAH