SuperCrunchers by Ian Ayres

There is a lot of data out there.

This fact has urged a few related notions to propagate from the realm of research into the world of business and the popular consciousness. The first such notion is that the amount of data out there is a bit frightening – there is a tremendous, staggering amount of it residing in the digital storehouses of government and industry, and it is growing bigger every minute. The second notion is that there is a desperate need to harness this data both for the betterment of society and the growth of business. The third is that this desperate need is starting to be met by emerging disciplines, tools, and methods.

I can tick off several books that reside in one or more of the above categories. “Beyond the Tsunami,” an online book from Microsoft Research mentioned in a post in this blog, is one such book. “Visualizing Data,” a title from O'Reilly Press, falls into the the third category.

But the book I want to talk about now is “SuperCrunchers,” a 2007 book by Ian Ayres, a professor at both the Yale School of Law and the Yale School of Management.

SuperCrunchers is a book about how statistical analysis of large datasets (that is, “supercrunching”) is beginning to profoundly affect our lives. The book illustrates this effect with a series of anecdotal case studies. In one example, a computer science professor started the website farecast.com after he was annoyed by learning that the people in the seat next to him on a plane had paid less for their tickets simply by waiting longer before buying them. Farecast is a site that predicts when tickets for your desired flight will be cheapest.

Ayres also describes the application of statistical methods to societal problems in the fields of medicine (evidence based medicine) and education (direct instruction). The sometimes innovative and sometimes creepy use of such methods by large corporations are detailed by Ayres as well.

The anecdotes contained in this book are compellingly written, and since the book is for a popular audience, the statistical methods described are described easily digested by the average reader.

This is an important book—-it clearly describes how the abundance of data and the statistical methods to manipulate it are changing society. This change is sometimes clearly for the better, and sometimes less so.

Google translator phone; CNN interviews Google policy analyst

An article in the Australian Herald Sun (and noted in the Beyond Search blog) states that within a few years Google will have a working translator phone. Yes, you heard right. You speak English, and the guy on the other end hears Mandarin (or French or...). This is, as Beyond Search noted, a very Star Trek-y advance in communications.

In other news, CNN interviewed Derek Slater, a policy analyst at Google about universality of access and the benefits of improving broadband in the U.S.

Google, public data, and visualization

Google offers a recently added feature (discussed on the Google blog in April here) that allows for both searching and visualization of public data.

There is no special doorway to this data; just search on "unemployment rate Philadelphia" (for example) in the plain old Google search box. The first link in the result list will be accompanied by a chart illustrating the latest data for your search.

When you click the link, however, you are taken to a larger interactive chart where you can compare the unemployment rate in Philadelphia with the rate in other cities and states.

The Google blog states that the visualization aspect of the public data search comes through their acquisition of Trendalyzer, a program that enabled the animation of statistical data.

Librarians and Programming Skills

A recent article in Research Information (and referenced on the LISNews blog as well as the ResourceShelf Blog) talks about transforming the role librarians through their acquisition of programming skills.

One point that author David Stuart makes in the article is that librarians are still somewhat bound to the concept of "document," even in an age where the web abounds with all kinds of "structured data" that is not necessarily in a "document" format per se. Stuart believes that if librarians had the right technical skill sets, they could make this data available to their patrons through the use of programming APIs and mashup tools such as Yahoo Pipes. In particular they could integrate it into the result sets of an open source ILS or OPAC.

I think this is a great point. Librarians, especially at academic institutions, are moving away from being guides to stacks of paper books toward being gatekeepers of digital resources, subscription or otherwise. It's exciting and quite fitting to consider librarians as imaginative data aggregators -- using technical skills to combine data from different sources into a result portfolio for a patron or selected audience -- for example, engineers or humanities students.

Check out the article -- it's a great read!

Google Goggles

Google has an image search engine where the input itself is an image, not text as has been customary even in Google itself.

The project is called "Goggles." It's intended for use with your mobile phone as you are out and about. If you, for example, see a bottle of wine, book, or work of art that you would like to know more about, snap a picture of it, submit it to Google and view the results. You can also use Goggles to get more information from a business using a business card. Or take a picture of a store on a street to find out what else is in the neighborhood.

You can find more information about Goggles at:

http://www.google.com/mobile/goggles/#landmark

Happy Holidays from Information Science News

A Merry Christmas to those who celebrate, and a Happy New Year to all! Here's wishing you a productive and rewarding 2010. May you find the things you search for.

I was trying to come up with a clever New Year's wish, and that last sentence was the best I could conjure. Still, it has a nice (George) Lucasian ring to it.

Happy Holidays y'all.

The Fourth Paradigm

There is a new collection of essays from Microsoft Research in honor of Jim Gray, the renowned researcher at Microsoft who was lost at sea in 2007. The collection is titled "The Fourth Paradigm: Data Intensive Scientific Discovery." According to an article by John Markoff of the New York Times, the book's essays are centered around the topic of how to deal with the explosion of data -- particularly scientific data -- in recent years. This particular topic has come up in this blog before -- read about it here.

Some sample essay titles: "Beyond the tsunami: developing the infrastructure
to deal with life sciences data," "Visualization for data-intensive science," and "From web 2.0 to the global database."

Access to this book is free -- it's available at the Microsoft research site at research.microsoft.com.