Systems

Everything is data

Lorcan 1 min read

An interesting perspective on Wikipedia is provided by Dan Cohen who notes that the current discussion about authority does not engage with the full potential significance of Wikipedia. He describes how a large, openly available knowledge base like Wikipedia is a valuable resource for emerging data mining and search technologies.

Let me provide a brief example that I hope will show the value of having such a free resource when you are trying to scan, sort, and mine enormous corpora of text. Let’s say you have a billion unstructured, untagged, unsorted documents related to the American presidency in the last twenty years. How would you differentiate between documents that were about George H. W. Bush (Sr.) and George W. Bush (Jr.)? This is a tough information retrieval problem because both presidents are often referred to as just “George Bush” or “Bush.” Using data-mining algorithms such as Yahoo’s remarkable Term Extraction service, you could pull out of the Wikipedia entries for the two Bushes the most common words and phrases that were likely to show up in documents about each (e.g., “Berlin Wall” and “Barbara” vs. “September 11” and “Laura”). You would still run into some disambiguation problems (“Saddam Hussein,” “Iraq,” “Dick Cheney” would show up a lot for both), but this method is actually quite a powerful start to document categorization. [Dan Cohen – Digital Humanities Blog – The Wikipedia Story That’s Being Missed]

This is an interesting example of how what is processable will be processed where it can add value – for somebody.
Via if:book.

Share
Comments
More from LorcanDempsey.net
The technology career ladder
Institutions

The technology career ladder

Library leaders should be drawn from across the organization. Any idea that technology leaders are overly specialised or too distant from general library work is outmoded and counter-productive.
Lorcan 7 min read
icon

Lorcan Dempsey dot Net

The social, cultural and technological contexts of libraries, services and networks

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to LorcanDempsey.net.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.