For as long as anyone can remember, the world of predictive analytics has been the exclusive realm of ivory-tower statisticians and data scientists who sit far away from the everyday line of business decision maker. Big data is about to change that.
As more data streams come online and are integrated into existing BI, CRM, ERP and other mission-critical business systems, the ever-elusive (and oh so profitable) single view of the customer may finally come into focus. While most customer service and field sales representatives have yet to feel the impact, companies such as IBM and MicroStrategy are working to see that they do soon.
Big data moves analytics beyond pencil-pushers
Imagine a world in which a CSR sitting at her console can make an independent decision on whether a problem customer is worth keeping or upgrading. Imagine, too, that a field salesman can change a retailer's wine rack on the fly based on the preferences that partiers attending the jazz festival next weekend have contributed on Facebook and Twitter.
Big data is pushing a tool more commonly used for cohort and regression analysis into the hands of line-level managers, who can then use non-transactional data to make strategic, long-term business decisions about, for example, what to put on store shelves and when to put it there.
However, big data is not about to supplant traditional BI tools, says Rita Sallam, Gartner's BI analyst. If anything, big data will make BI more valuable and useful to the business. "We're always going to need to look at the past...and when you have big data, you are going to need to do that even more. BI doesn't go away. It gets enhanced by big data."
How else you will know if what you are seeing in the initial phases of discovery will indeed bear out over time. For example, do red purses really sell better than blue ones in the Midwest? An initial pass through the data may suggest so-more red purses sold last quarter than ever before, therefore, red purses sell better.
But this is a correlation, not a cause. If you look more closely, using historical transaction data gleaned from your BI tools, you may find, say, that it is actually your latest merchandise-positioning-campaign that's paying dividends because the retailers are now putting red purses at eye level.
That's why IBM's Director of Emerging Technologies, David Barnes, is actually more inclined to refer to the resulting output from big data technologies such as Hadoop, map/reduce and R as "insights." You wouldn't want to make mission-critical business decisions based on sentiment analysis of a Twitter stream, for example.
Reviewing unstructured data in social media reaps immediate rewards
There is value in social media, though. What if you learn, as the buyer for a retailer, that Justin Bieber fans really loved the jacket he was wearing at the concert last night-and, oh, by the way, someone tweeted he got it from one of your stores? You could then make a snap decision to stock up on that jacket just in that city since you know it's about to become a very hot item, albeit for a very limited time.
Without a predictive analytics (PA) package looking for patterns in the Twittersphere that correlate your brand with geographic location and factors such as the number of mentions, you could miss out on a great but small window of opportunity to move merchandise.
"In the past, we would have based [our decisions] on historical data-and, by the time we did it, that trend may have already passed us," says Barnes. "So that's PA on steroids, at warp speed."
How this is accomplished is a marriage of open source technologies (where most of the Big Data platforms are coming from these days), Moore's Law, commodity hardware, the cloud and the ability to capture and store huge volumes of non-transactional data that was once discarded because no one knew what to do with it.