How real is the value proposition of Big Data for most businesses

Onboard data or Ingest… Store and Organise it…. Extract Insights or Reporting which all equal value creation….Not to mention all this Data Science and Artificial Intelligence stuff?….Easy right?

Maybe not. With more and more businesses frustrated with their Big Data plays and many Big Data platforms failing to realise value uplift that the stakeholders expected. Something is certainly smelly in the world of Big Data.

Maybe it’s not as easy as we all thought? We all hear about the amazing use cases and satisfied customers, though many industry reports suggest they are in the minority. This series of articles is designed to discuss they pitfalls of Big Data and suggest some key areas to keep in mind to help minimise or mitigate issues early.

Lets breakdown the three distinctive phases, Ingest, Store, Consume. In this first part of three, I’ll discuss Ingest or onboarding data.

Ingest: this area is so often misunderstood and unfortunately the start of some of the basic issues with the effective establishment of a Big Data platform. Many businesses believe they have the data, they have lots of data, but what is realised quickly when starting a Big Data play is that, there are gaps, not only in breadth of data, but depth, accessibility and finally quality (Completeness, Uniqueness, Timeliness, Validity, Accuracy, Consistency), lets save the quality discussion for another time.

Why are there gaps? There are many factors in how this occurs, one of the biggest culprits is that years of projects have created very narrow datasets, designed to address very specific issues and or application outcomes. That has lead to the myriad of ETL (Extract Transform and Load) jobs or Data integrations which are largely point solutions. As such they rarely have had the foresight to bring in as much data and certainly the breadth of data needed for a big data outcome.

To be fair to those who came before (including myself), there were many drivers for this behaviour, time to market or completion deadlines — comments like “We need that project deployed and completed by …”, are common. Project management translation, do whatever it takes to narrow the scope to the absolute minimum viable deliverable. On of the first victims of that thinking, narrow data breadth. Ever heard, “Why do we need all that data?”, “Show me the reporting requirements” or “Do you realise how complicated and costly that is?” and my personal favourite, “Don’t worry we can always do another data extract at a later date”.

How might we overcome these pitfalls?

Firstly, understand the value of your data and convert senior stakeholders into true evangelists. These programs of “Data Transformation” struggle to succeed without strong executive support. Make the case “real”, tie it back to measurements, yep, you need to commit, don’t just work the “Data Hype Cycle”.

Initially it may be lower storage costs, lower environment costs or reduced testing costs, in my experience keep the measurements grounded or in accounting terms “tangible”. People gravitate to the “intangibles”, they sound better, “greater access to data”, “faster provisioning of data” or “better analytics” and though they all have merit, in the early days of a successful Data transformation, you need to ground it in “tangible”, measurable dollars.

A related step is understand you have as much work in change management (roles, structures, skills or people and process) as you do in technology change. Typically we find that companies pursue the Big Data dream and then find they run out of steam, when they haven’t faced into the fact that people need to be re-skilled, re-tooled and processes that may have worked yesterday, are no longer relevant in their new data paradigm. In fact trying to shoe horn old techniques into the new Big Data play typically just don’t work. Rebuild that Target Operating Model, be bold!!

Ask for as much data as you can possibly get!!!! Work that Big Data storage cost advantage. Whatever you do don’t compromise on the depth and breadth of data, use cases are most likely not even in your pipeline yet that will drive revenue and insight into your business tomorrow. That data will drive value, take the leap.

Find experts, don’t wing it. Whether that be a firm that has the experience and skills in deploying these new capabilities or whether you need to find people on the market and insource the skills (typically a mixture of both is a great long term play). Find people that think differently. What do I mean? Big Data has brought with it some fundamental changes to the way we interact with our data, as such new techniques are not only required, but also a new mindset. Flooding people with vast amounts of data, when they have only ever previously been exposed to curated data, is daunting and in many ways creating the environment for failure. The advantage many of the really successful Big Data players today enjoy is that they were built in the new Big Data paradigm (in fact many invented key components of the technology we take for granted today) and have applied the new thinking and technology to drive, the creation of maximum value from their data assets (think Netflix, Uber, Google, Amazon — all data driven companies).

Finally a stretch target, its a tough one, but try nevertheless, as the pay off in my experience is very rewarding, not only in long term cost reductions but also in the establishment of fundamental trust in your data. Cleanse your data as much as possible at source, build integrity checks like postcode check or mobile phone number formatting checks into your web or mobile applications (start with the small and relatively simple changes). Build that mindset into the development of new technologies in your company “Design with Data in mind”, “Deploy knowing someone will use it”.

Doing that very same fixing post load in your Big Data platform is not only expensive in the longer term, but also degrades trust in the data.

In the next article I’ll discuss the key challenges with “Store” and how the shift from a world of structured data to unstructured data, has confused many and again led to sub-optimal outcomes.

About the author:

Related posts:

Amulet is a new DeFi (decentralized finance) coverage protocol for Rust-based ecosystems. It’s simple, Reliable Cover For Everyone in Web3 Amulet is built with the promise of simple, secure insurance…

Bitcoin solo-mining becomes more and more difficult, even if you have some rented hashrates or several ASICs working simultaneously. The reward is the highest among other cryptocurrencies, despite…

Probably not the most asked question on this planet (which would be “Why me?” or alternatively, its near relative: “Why not me?”) but still up there toward the top would be, I guess: “What are we…