An evaluation of interactive Boolean and natural language searching with an online medical textbook

Reviewer: Karen Sparck-Jones

The authors describe experiments comparing three systems—natural language Boolean, natural language ranking, and concept ranking—in interactive searches on a large medical textbook. There were 82 requests, searched in a carefully distributed design against 6,623 textbook articles. Performance for retrieved sets did not differ markedly or systematically, and this was also true of batch searches for the two ranking systems. Users did not perceive much difference between the systems. It is not clear what we can learn from these tests, beyond the familiar broad finding that natural language does reasonably well. For example, if comparisons suppress ranking, is this sensible__?__ Perhaps it is, if ranking is seen only as an interactive aid, b<__?__Pub Caret>ut the paper does not discuss the interactive search processes. Also, it does not give enough information about the document file; for example, what is the average document length__?__ At the same time, the median number of documents retrieved was small, and while recall was high, there were few (known) highly relevant documents. Overall, it is impossible to assess the results in relation either to “normal” retrieval or to searching a nonstandard kind of file.