Mining gold from smart data lake

IT executives gathered for ideas to build a data irrigation system that nurturesWith an increasing volume and variety of data, enterprises are looking into the implementation of smart data lake to help them extract valuable business insights to nurture market growth.

A panel of Hong Kong IT leaders from different industries gathered recently to discuss the challenges of managing large volumes of data and using smart data lake to gain relevant business insights.

“For the past six to 12 months, there have been a lot of discussions about smart data lake,” said Cally Chan, managing director of Hewlett Packard Enterprise (HPE), Hong Kong. “Enterprises face the challenge of scalability, cost and flexibility with the large volume of structured and unstructured data.”

In a smart data lake, structured and unstructured data is stored in its raw format in a central depository. This approach allows business users to analyze data in novel ways that suit their needs and overcome the inflexibility from a data warehouse that was built originally from relational databases management systems (RDBMS). At the same time, text, audio, images and visual data can be analyzed with speed and flexibility.

From Formula E to insurance sales strategy

One example of applying smart data lake in Hong Kong was during the city’s first electric car racing event FIA Formula E Hong Kong ePrix in October last year. HPE supported the engineers at the DS Virgin Racing team to collect and analyze data obtained from the practice race and refined the engines for the actual race in the afternoon.

“We helped them get valuable insight into vast volumes of data including real-time audio, video and machine data which is critical for the team, allowing the team of engineers to formulate a strategy for the driver,” said Chan.

Similarly, a smart data lake can empower companies to perform different types of analytics with different data formats and effectively compare and correlate data for the specific business user.

“Data generated around business application is less than 20%. Most of the data is generated from machine data and human data. But it is not being analyzed and looked at yet,” said Bradley Mearns, APJ regional big data consulting practice manager at HPE. “Machine data and human data are growing 10 times faster than our traditional data. The integration of different big data technologies in a smart data lake allows assessment of 100% of the data.”

Some companies like AXA Asia are reaping the benefits of a smart data lake. The company used various types of data like customers profile and their related agents to develop a sales strategy.

“We were able to analyze core patterns of consumers, types of contact and selling styles of the agents. In a four-week pilot project, there was 100% increase in sales,” said Ash Shah, regional property and casualty, CIO and chief of staff of AXA Asia. “Now we are doing it across all types of customers with the help of data scientists to bring tangible results.”

Justifying the investment

Participants agree that smart data lake enables organizations to gain valuable business insights from data obtained from multiple sources and formats inside and outside the organization. But creating a business case for the investment is often a challenge for many.

“Technology is enabling so much and now we need to catch up,” said Alison Dack, CIO of Federal Express. “You can store more [data] but the trick is finding the right people to analyze this data. What is the business problem we are trying to solve?”

“Researchers agree that the issue is not collecting the data, rather than why we are doing this. Some may be seduced into investing in the physical architecture,” said Gerrit Bahlman, director of IT of Hong Kong Polytechnic University.

“Choose one use case, start small and determine if it’s of any business value. If you start with 100 use cases, it will be too long and likely to fail,” said Mearns.