adv

3 keys to keep your data lake from becoming a data swamp

3 keys to keep your data lake from becoming a data swampFor years, buoyed by technologies like Apache Hadoop, organizations have been seeking to build data lakes—enterprise-wide data management platforms that allow them to store all of their data in their native format. Data lakes promise to break down information silos by providing a single data repository the entire organization can use for everything from business analytics to data mining. Raw and ungoverned, data lakes have been pitched as a big data catch-all and cure-all.

But Avi Perez, CTO of business intelligence (BI) software specialist Pyramid Analytics said he sees many customers and prospects whose data lakes are deteriorating into data swamps—massive repositories of data that are completely inaccessible to end users.

"Databases are really expensive," Perez said. "The data lake fundamentally answers that problem. Data lakes, and all big data initiatives, come from, one, pressure in the marketplace to have one, and secondly, real-world data generators spitting up gobs of data that you need to find a way to store."

But while a number of the world's most successful companies have built businesses around their data lakes (Google is a prime example), many others are collecting data without any clear way to get value from it.

"They just collect dust," Perez said "You're just collecting junk. I think they'll get abandoned. Eventually you cut the budget for stuff that's big and expensive and not doing anything."

That's not to say the idea behind data lakes is a bad one. Perez is convinced that all companies will need one eventually. But creating a data lake that your end users can actually benefit from requires deliberation.

To avoid drowning in your own data lake, Perez recommends adopting three principles.

1. Collect less data, at least in the beginning

Perez said one of the biggest mistakes organizations make is collecting too much data, simply because they can. Consider your smartphone. If you own one, chances are you've got hundreds or more pictures stored on it.

"You end up with a billion pictures on your phone, and yet 99% of them are probably garbage that you would get rid of in a heartbeat," he said. "It's gotten so easy to take pictures with your phone, it's essentially free. And you probably think, 'One day I'll go and clean it up,' but of course no one ever does. You're collecting an enormous amount of information, but you have no way to work your way through it to use it effectively."

When you inevitably want to show someone a particular photograph, finding it can require scrolling through an enormous volume of junk.

The same thing happens with data lakes, Perez said. Storing data in Hadoop is inexpensive enough that it's often considered free. But the sheer volume of data that accumulates can make it difficult to actually access the data that could provide you with valuable insights.

"I think the way to avoid it is actually to turn the spigot way, way down," Perez said. "Work on the presumption that just because it's cheap to collect the data does not necessarily make it cheap to use it. It could actually be quite expensive. So don't collect information from everywhere and all the time. Keep it focused with a data set where you have a specific plan as to how you're going to mine it."



Comments