IR and the elusive ground truth..

[Subbarao Kambhampati <rao@asu.edu>]

As I mentioned in the class, it is very important to note that the
ranking problem in information retrieval is fundamentally different 
from the ranking problems you may have come across in traditional CS (and
is quite indicative of the ranking problems that you will come
across 
from here on)

In a traditional CS algorithm, you have a way of testing the correctness
of the output without recourse to an external oracle. You (the computer)
don't need external feedback to decide whether a given sequence is sorted
or not. 

The solution to the ranking problem in IR, on the otherhand, depends on
the user's relevance metric, which is conveniently left out of the
specification ;-) 

So, in order to solve the problem, you first need to
"hypothesize" what is the user's relevance model might be, and
then design a ranking scheme that will follow that hypothesis. Given the
two step process, you can go wrong in either of the steps (you can have
the wrong model of relevance, or the wrong way of operationalizing it
into a ranking metric). 

One way to improve the hypothesis on the relevance model is to tune it by
learning what the user might actually be interested. Techniques such as
relevance feedback are starting points for this.

Rao

ps1:  This type of elusive ground truth is not limited to IR. When
we come to clustering, we will find that what are the right clusters will
depend to a large extent on what are our assumptions/hypthesis about the
generative process that generated the clusters to begin with. We might
decide that the generative process is making spherical clusters,
and  develop a very good clustering algorithm for detecting
spherical clusters (e.g. k-means). However, if our model is wrong and the
clusters are mostly non-spherical, our best algorithms are not likely to
give clusters that the domain experts will have any interest in looking
at. (One of the ways this is handled in clustering is to try to learn
what sorts of clusters the experts seem to like!
An example of such work is this year's ICML best paper award paper:

http://www.cs.cornell.edu/People/tj/publications/joachims_05a.pdf
)

ps2: For some reason, whenever I talk about this issue of having to guess
what the users have in their mind from the imperfect keyword
queries they give, I am reminded of the following little play by Woody
Allen (which I scanned for you). See if you get any insights from it
;-)                             

http://rakaposhi.eas.asu.edu/lincoln-query-woody-allen.pdf
(use the rotate-counter-clock-wise button in acrobat to view it
correctly)



---------------
 We dance round in a ring and suppose
   But the secret sits in the middle and knows
           -Robert
Frost