Three Ways of Aggregating Information

Last summer, University of Chicago law professor Cass R. Sunstein[1] published his book titled Infotopia: How Many Minds Produce Knowledge (New York: Oxford University Press, 2006), which includes a substantial discussion of Wikipedia: The Free Encyclopedia (http://en.wikipedia.org). It serves as an example of the third (and allegedly most promising) of three ways to aggregate information that is held by a number of people: 1) surveys, 2) deliberation, and 3) prediction markets.

According to Sunstein, surveys are good for gathering information that is distributed widely across a given population, but they lose validity if many of the people being surveyed do not have any knowledge of the subject. Deliberation, which our society uses very widely, tends to produce the phenomenon that Sunstein calls ideological amplification in groups that are fairly like-minded: they become more uniform and move toward the extreme in their previously held opinions. A given prediction market, such as that which affects stock market prices, relies on the information contributions of people who have some knowledge of the subject. Thus, prediction markets avoid both the potential problem of "pooled ignorance" in surveys and the ideological pitfalls of deliberation. In regard to prediction markets, Sunstein has stated that "the average answer is eerily good."[2]

The basic idea behind prediction markets is not complicated and was stated succinctly at least three years before Sunstein’s book appeared: "The idea is that multiple contributors will . . . balance clashing viewpoints."[3]

The Frustration of Prediction Market Theory in Wikipedia Articles

The overall expectation of using prediction markets to arrive at the good, average answer has appeal; over time, this expectation may in some instances be fulfilled. The problem is not that the theory behind prediction markets was somehow invalid, but that this theory often does not have a chance to come into play in Wikipedia. When one comes down to the level of the individual article in Wikipedia, it is frequently difficult to know whether the range of answers and the range of points of view are both broad enough and inclusive enough to create the hoped-for balance that prediction markets seem to promise. Therefore, to attribute reliability to the "average answer" in a particular Wikipedia article is frequently hazardous. Even an article whose contributors number in the hundreds might simply be a long essay having many sections, each section expressing the views of very few contributors.

An example of the frustration of prediction market theory in articles that have attracted large numbers of contributors is the Wikipedia article titled "David," referring to the biblical king of Israel. Its current version gives fairly lengthy coverage to the biblical presentation of him. The Jewish, Christian, Muslim, Mormon, and Baha’i views of David each receive treatment that varies from brief to scant. The question of historicity regarding David is treated at some length in light of archaeological discoveries. Specific portrayals of him in art (only sculpture), literature (only modern), and cinema (only two films) attempt to round out the article, and they are followed by reference material. The article has hundreds of contributors, but parts of it remain seriously underdeveloped. On the subject of Muslim views of David, with which many readers are not familiar, how can one know what measure of confidence to place in the seven sentences that appear?

In such a situation, reasonable caution would suggest checking the accuracy of Wikipedia’s data against several independent sources that are considered to be reliable. In fact, Wikipedia itself advises readers to do precisely this. One Wikipedia article that presents guidelines for readers begins:

IMPORTANT NOTE: Most educators and professionals do not consider it appropriate to use tertiary sources such as encyclopedias as a sole source for any information — citing an encyclopedia as an important reference in footnotes or bibliographies may result in censure or a failing grade. Wikipedia articles should be used for background information, as a reference for correct terminology and search terms, and as a starting point for further research.

As with any community-built reference, there is a possibility for error in Wikipedia's content — please check your facts against multiple sources and read our disclaimers for more information.[4]

This practical necessity might have given rise to the saying that I heard recently from a graduate student while he was speaking of academic research: "Wikipedia is a good tool but a bad source." Although Wikipedia’s ever-increasing size, variety, and utter mutability make it risky to characterize, I would refine this saying to state: for academic research, Wikipedia is an unreliable source but a frequently useful heuristic tool.

Unreliable Articles as Heuristic Gold Mines

As a reference librarian offering research consultations to graduate and undergraduate history students, I have found that Wikipedia sometimes presents "leads" that are like anonymous "tips" phoned in to police investigations. These bits of information come from unknown sources and may not be accessible elsewhere. They can save time but must be verified.

Last October, an example of its heuristic use presented itself when I was helping a graduate student search for primary historical source material on the pirate Blackbeard, also known as Edward Teach (ca. 1680 - November 22, 1718), who used several aliases. Although the student had found mention of him in one or two primary-source writings of people in authority who knew him, plus some good secondary sources written well after Teach’s lifetime, the research project clearly called for more primary sources. But as one might surmise, criminals are frequently reluctant to leave behind accurate records of their comings and goings.

Having exhausted all the available paper and online scholarly sources that I could think of, I felt I had nothing to lose by consulting Wikipedia. In the middle of the wiki article on Blackbeard was an account of his death, quoted from the Boston News Letter of the period, with a footnoted reference to a book that held promise of a more complete citation. Quickly, I was able to determine that our library holds a microform copy of Boston News Letter from 1704 onward! (Take that, Mr. Teach!)

The online and microform resources of the library contain immense reservoirs of primary source material, but the microforms, especially, are not well indexed. A researcher could literally spend months or even years poring over the microforms, hoping to find some mention of Blackbeard in an early eighteenth century book, magazine, or newspaper. Here in Wikipedia, at the cost of a few clicks, was a precious lead to be investigated that offered a reasonable hope of finding another apparently rare primary source.

Enrichment of Articles

It is a fact of present-day higher education that students are going to use Wikipedia. The evident usefulness of Wikipedia as a heuristic tool leads to the question of how its content can be enriched for greater usefulness (detractors and some realists might say: how its content can be made less bad). A way that stresses subject content is by increasing the per cent of content contributed directly by persons who have subject expertise, if only to correct errors and omissions.[5] A way that emphasizes the prediction market process is to encourage well-informed persons who represent a variety of points of view to become contributors and wait and hope for prediction market forces to become more effective in Wikipedia. The former way is already effective and rapid; the latter way seems to be slow and, in many current instances in Wikipedia, stymied.

Lawrence Mykytiuk, Purdue University

Notes

[1] Professor Sunstein’s university home page is at http://www.law.uchicago.edu/faculty/sunstein/.

[2] This quotation and almost all of the content of the first two paragraphs of this article are from Professor Sunstein’s C-Span 2 Book TV oral presentation, most recently aired at 12:00 a.m. October 16, 2006. See http://www.booktv.org/General/index.asp?segID=7414&schedID=454.

[3] Mitch Laslie, "The People’s Encyclopedia," Science 301 (September 5, 2003): 1299.

[4] "MediaWiki: CiteText," Wikipedia: The Free Encyclopedia [cited December 1, 2006]. Online: http://en.wikipedia.org/wiki/MediaWiki:Cite_text; quoted by George R. Plosker, "Where We Are Now with Content and Technology," Information Strategist column, Online 30/3 (May/June 2006): Research Library Core, 51. Online: www.onlinemag.net.

[5] An appropriate example is Taylor David Halverson, "Wikipedia or Wackipedia? On the Reasoned Use of New Technology," SBL Forum (November 2006): paragraph 5.