NYT: Sure, Big Data Is Great. But So Is intuition

Written Jan 27, 2013

Steve Lohr at the New York Times* writes a reflection on the promises of Big Data, citing increasing buzz, yet also its initial big failure:

“Many of the Big Data techniques of math modeling, predictive algorithms and artificial intelligence software were first widely applied on Wall Street.” And what happened there we all know.

He cites various perspectives form experts in the field. A chief scientist from an ad-targeting startup warns:

You can fool yourself with data like you can’t with anything else. I fear a Big Data bubble.

Another expert summarizes the challenges:

A major part of managing Big Data projects […] is asking the right questions: How do you define the problem? What data do you need? Where does it come from? What are the assumptions behind the model that the data is fed into? How is the model different from reality? [emphasis mine]

A skill in the liberal arts and scientific method seems as important as computer and math prowess. This is the perspective of Rachel Schutt, a senior statistician at Google Research who taught a data science class at Columbia. She sees a need for someone “who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data,” according to the article.

And the impact of this work is wide: “Models do not just predict, but they can make things happen,” she says.

*Reference

Steve Lohr, New York Times