Revolving around the ephemeral “web 2.0″ and the future of the internet, Web 2.0 Expo New York wrapped up over the weekend, and already the talks are up at the blip.tv Web2Expo page. (The San Francisco talks from April are also up for viewing.) The expo runs the gamut from technology to business, but most interesting were the talks on web 2.0 structure by Jay Adelson of Digg.com and author Clay Shirky. Shirky’s talk, It’s not Information Overload. It’s Filter Failure., especially titillated my nerd bones, so here it is in all its bald-headed, fast-talking glory:

Basically, Shirky proposes that consequences we normally attribute to an explosion of available information thanks to the internet is actually attributable to a failure of the filters in place to deal with an already abundant amount of information. Since the invention of the printing press, he says, we have lived amidst “information overload”, such that we can no longer look at the phenomenon as a problem, but as a fact. An appropriate response, then, is to build better filters.

Notice that the solution is to build, not to fix. An overload of information has been our oxygen for centuries, but the type and amount of data we deal with now is vastly different than whatever has come before. Shirky ends by separating the types of filters needed into two categories: programming and social. The latter category is pretty nebulous, but “programming” is much more concrete, and already in use today. Digg, Netflix, Google, and every other website run on extrapolating from its users’ actions is utilizing so-called collaborative filters, and these are a pivotal part of the internet’s future.

Collaborative filtering works by ranking content according to prior users’ actions (e.g., Google looks at links, the paragraphs surrounding search terms, other mysterious data), then analyzing your own actions to serve up relevant content. With sites like Digg and Reddit, this involves users upvoting or downvoting submitted sites, and then you seeing the best sites in descending order (with maybe some specializing depending on if you’re at a subpage). With sites like Netflix and Amazon, your own consumption is compared to other consumers in order to serve up recommendations.

As we approach the singularity and the web becomes more ubiquitous, collaborative filtering will become increasingly sophisticated (and accurate). As Jay Adelson mentions, Google’s search rankings have become more powerful simply because of the diversity of its users has increased. Collaborative filtering thrives on multifarious data, and this will come quickly with a larger number of netizens, and more slowly through the growing number of connections between web services. Adelson points to the economic advantages of shared data (read: better advertising targeting), but this is of course of huge value to users and developers as well.

Low-level web curation, from Metafilter to Undress Me Robot, will always have its place online, but the future is definitely in these automated processes which leverage what each user is already doing to provide a highly personalized and more effective experience. However, obviously, collaborative filtering is simply an update of the sort of filter that has been around since the 1500s, moving from the editorial eye of a single person to the gaze of millions. The next big jump in information overload might break down even these strong filters, taking a paradigm shift to get back on top. And who knows where that will take us?

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>