The Limits of Big Data: Why We Still Need Humans

Advanced data mining methods like horizon scanning can help companies sift through the billions of bytes of data zipping across the web each second. But is "big data" enough? We take a closer look at the human side of data analytics.

Finding the Needles in the Big Data Haystack

It would be an understatement to say that the Internet is big, and it just continues to grow.

A lot of this data is potentially interesting to companies—customer reviews, social media mentions, scientific publications, regulatory and legislative activities and announcements, breaking news and more. But how do you find the information relevant to your company in this avalanche of data?

Sorting through it all is much more than the human mind can handle. In addition to the volume of data, much of the relevant information is hidden in unstructured formats and sources of indeterminate quality.

Why Data Isn’t Enough: Data vs. Information

Companies need more than raw data. Data must be processed, structured and analyzed in context to be useful for company planning. Data analytics can tell you which topics are trending and where. But by itself, it can’t tell you why.

For example, seeing that interest in a specific ingredient of your product has risen by 500% in two months is a data point. But what does it mean for your business?

Adding Human Insight to Horizon Scanning

Using horizon scanning, companies can now track thousands of relevant topics as they appear in blogs, social media, news sources, scientific journals, government sites and other sources of public information.

But turning masses of data into valuable market intelligence requires a human touch. Horizon scanning must be informed by a deep understanding of the market environment and the issues and topics that are relevant to the business. Human expertise also is needed to interpret the results of the data analysis.