How Web Sites Are Leveraging the Minds of the Crowd
by Rick Klau
Everyone knows the phrase "two heads are better than one." On the Internet, the two become legions. Combining the knowledge of the many in a decentralized collaboration can lead to better answers for all.
In 1968 the U.S.S. Scorpion, an American submarine, sank en route to investigate a collection of Soviet ships that had gathered off the coast of the Canary Islands. The Navy had only the most general of ideas of where the sub sank—yet after canvassing a number of individuals and averaging their guesses, it was able to pinpoint the location of the sub's wreckage to within 220 yards. It turned out that the collected guesses, when combined, were nearly perfect.
Years later, Regis Philbin hosted the popular TV show Who Wants to Be a Millionaire?
in which contestants who didn't know the answer to a particular question were given "lifelines." One lifeline option was to "phone a friend" (presumably someone who would know something about the subject matter of the question); another was to "poll the audience." An analysis of lifelines used revealed that the audience got the answer correct more than 90 percent of the time. Friends were right only two out of three times. The expectation was the opposite: Surely friends, who the questioner could pick based in part on their assumed expertise in an area, would be more accurate than a random television studio audience?
Both examples were cited in James Surowiecki's 2004 book The Wisdom of Crowds
—and an entirely new technology is emerging on the Internet that proves Surowiecki's observations yet again. Today, sites like Digg, StumbleUpon and Netscape (not the browser, but the relaunched portal) all give crowds the ability to help identify interesting and relevant content. It's changing how people browse the Web, and it's significantly changing how Web masters are thinking about promoting their sites.
Online Collaboration: Where Every Vote Counts
In this context, "collaboration" means a decentralized cooperation across thousands or even millions of users—as opposed to the traditional "individuals working together" that we usually think of when we talk about collaboration.
At its core, Google's search engine is a kind of collaborative effort between the millions of sites on the Internet and Google's index. When a site links to another site, Google counts that link as a "vote." The more votes a particular site has, the higher its page rank, which in turn helps establish that site's authority in the index. In the process, the "crowd"' of Web masters, bloggers and publishers is creating a kind of intelligence by influencing Google through links. Consequently, end-users benefit by getting smarter search results and better access to information that matters to them.
Another area that has seen tremendous value through crowd activity is spam control. It used to be that when you got a piece of spam in your inbox, you would train your spam filter to understand that this message wasn't legitimate. Over time, the filter would get a better sense of what was spam for you. But as spam filters have moved off individual computers and up to the network level, if not to the Internet as a whole, entire communities of users have been able to indicate what spam is. The filters get smarter because they're receiving more input—and that means that the filters are smarter than they could ever be working only on your behalf. Although it may seem like you've seen every kind of spam imaginable, your colleagues have seen different kinds of spam, spam that you might not see for another week or month—so, by participating in a group filter, you get the benefit of what they've seen, just as they get the benefit of what you've seen.
News Filters: Community-Oriented Editing
Which brings us to today's "news" filters. Leveraging the participation of tens of thousands or even hundreds of thousands of users, this kind of filter upends the traditional approach to presenting news (which is to say, an editor who decides how to lay out a page) and replaces it with the wisdom of the crowd. Digg
, perhaps the most popular of the technology news filters, grew from obscurity just 18 months ago to its millionth registered user this spring. Here's how it works:
- Have an interesting story? You submit it to Digg as a suggested site.
- Your suggestion is then categorized (as news, opinion, product announcement or so forth) and users who are interested in topics in that category get alerted about a new story.
- As users see the link, they're given a chance to "Digg" it, which means to cast their vote in favor of generating more exposure for the story, or to "Bury" it, which is reserved for off-topic suggestions, spam or other unrelated links.
- Links with large numbers of Diggs get promoted out of the category queues and onto Digg's front page. Digg has been known to generate several hundred thousand incremental page views within a few hours—if you get "Dugg", make sure that your server can handle the traffic!
Thus, instead of one editor, Digg has an unlimited number of editors. And users generate reputations along the way, too—as evidenced by the number of stories they submit, the number of Diggs their links receive, and the number of times their links are buried
Perhaps most interesting in this dynamic is that, as with many other community-oriented sites, there are two classes of users. Those who vote—the more participatory users—actively guide the site's evolution and exert tremendous influence over the visibility of specific stories. The less-active users, on the other hand, get a very different experience—yet one that's equally valuable. They get to see interesting content that's been vetted by a community of their peers. They can choose to visit a site with the confidence that hundreds, if not thousands, of individuals have looked at the link in question and found it to be interesting enough for others to see. For a demonstration, check out the Digg Spy
, a real-time presentation of votes as they're received.
While Digg gets the most attention in this space, another interesting site is StumbleUpon
, recently acquired by eBay. Think of StumbleUpon like channel-surfing the Internet: Once you tell it what kind of sites you like, every time you click the "Stumble" button in your browser, you go to another site that's been viewed by other StumbleUpon users who share your interests. You can give each site you see a thumbs up or thumbs down ranking, so votes come into play, which helps provide StumbleUpon with more context for deciding what sites to show its users in the future.
The Potential within Law Firms
As more applications and services come to rely on the wisdom of crowds to infer relevance, interest and value, we'll see more specific uses within corporate and legal organization settings. For example, what if your template library kept track of the number of times it was opened, edited or used in an actual filing? What if each document in the firm's document management system included an ability to "vote" for or against that document? A firm could easily replicate a Digg-like site (an open source version called Pligg
already exists) and adapt it for internal use, which would let lawyers, paralegals and librarians vote for relevant research sites, thereby building a knowledge base internally without requiring any one user to be its custodian. So how much wisdom is your firm gathering from its own crowd? Isn't it time to investigate how the minds of many can work better than one?