When to use Big Data and when not to: it should be guided by thoughtful human expertise

BIG DATA has been on the tip of everyone's tongue for the past several years now, and for good reason. As digital devices and touchpoints proliferate, so too does the amount of data we each create. This information can be used to help us better understand clients and customers, make more effective decisions, and improve our business operations - but only if we can make sense of it all.

By choosing the right Big Data sources and applications, we can put our organisations at a competitive advantage. But to do that, we need to understand Big Data's definition, capabilities, and implications.

Big Data already has widespread applications. From Netflix recommendations to healthcare monitoring, it drives all types of predictive models that improve our daily lives. But the more we depend on it, the more we need to question how it shapes our lives and whether we should be relying on it so much.

While progress is inevitable and something to embrace, Big Data's contribution should not be measured by how many companies apply it, but by how much better off it makes society as a whole.

Defining Big Data and its relationship to AI

Big data is more than just large datasets. It is defined by the three Vs of data management:

• Volume: Big data is often measured in terabytes.
• Variety: It can contain structurally different datasets, such as text, images, audio, and so on.
• Velocity: Big data must be processed quickly because of the increasing speed at which data is generated.

As the volume, variety, and velocity of data expand, they morph into Big Data and become too much for humans to handle without assistance. So we leverage artificial intelligence (AI) and machine learning to help parse it. While the terms Big Data and AI are often used interchangeably and the two go hand-in-hand, they are, in fact, distinct.

Simply put, Big Data powers AI with the fuel it needs to drive automation. But there are risks.

Realising Big Data's business potential

Properly applied, Big Data helps companies make more informed - and therefore better - business decisions.

Melvin Greer, Intel's chief data scientist says in an article: "A few examples include the hyper-personalisation of a retail experience, location sensors that help companies route shipments for greater efficiencies, more accurate and effective fraud detection, and even wearable technologies that provide detailed information about how workers are moving, lifting or their location to reduce injuries and increase safety."

But this crucial competitive advantage is underused because so many companies struggle to sift through all the data and distinguish the signal from the noise.

Five principal challenges keep companies from realising big data's full potential, according to Mr Greer:

• Resources: Not only are data scientists in short supply, the current pool also lacks diversity.
• Data aggregation: Data is constantly being created and it is a challenge to collect and sort it from all the disparate channels.
• Erroneous or missing data: Not all data is good or complete. Data scientists need to know how to separate the misleading from the accurate.
• Unfinished data: Cleaning data is time-consuming and can slow down processing. AI can help manage this.
• Truth seekers: We should not assume data analysis will yield a definitive answer. "Data science leads to the probability that something is correct," Mr Greer writes. "It's a subtle but important nuance."

Addressing the first challenge is of paramount importance. The only way to solve the other issues is to first create the necessary human capital and provide them with the necessary tools.

The true promise of Big Data

Data is a wonderful instrument, but it is not a cure-all. Indeed, "too much of a good thing" is a real phenomenon.

Jacqueline Nolis, Saturn Cloud head of data science, writes in an article: "In my years working with many businesses, I have indeed seen some companies that fell into the situation of not using data enough. However, these occurrences paled in comparison to the number of times I have seen the reverse issue: companies with an over-reliance on data to the point that it was detrimental. The idea that data is needed to make a good decision is a destructive one."

To illustrate her point, Ms Nolis describes Coca-Cola's introduction of Cherry Sprite. What motivated the decision? Data. People were adding cherry-flavoured "shots" to Sprite at self-service soda dispensers. So score one for Big Data.

But as Ms Nolis points out, the very similar-tasting Cherry 7UP already existed - and had since the 1980s. So the data team might have come up with the new flavour more efficiently simply by perusing the soft drink aisle at the local grocery store. The lesson: Too heavy a reliance on data can be a barrier to common sense decision making.

Big Data applications: When and how

So how do we know when to put big data to work for our business? That decision needs to be made on a case-by-case basis according to the demands of each individual project.

The following guidelines can help determine whether it is the right course:

• Consider the desired outcome. If it's to catch up with a competitor, investing in something the competitor has already done may not be a good use of resources. It might be better to let their example serve as guidance or inspiration and reserve Big Data analysis for more complicated projects.
• If disruption is the goal, Big Data can be applied to test new ideas and hypotheses and maybe reveal other possibilities. But we need to beware of the downsides: Data can kill creativity.
• If a business decision is urgent, the "data is still being analysed" is not an excuse to delay it.

Amid a public relations crisis, for example, we won't have the time to mine the available data for insights or guidance.

We have to rely on our existing knowledge of the crisis and our customers and take immediate action.

Of course, sometimes Big Data is not just useful but essential.

Some scenarios call for Big Data applications:

• To determine if a strategy is working as planned, only the data will tell the story. But before we measure whether success has been achieved, we first have to establish our metrics and define the business rules that determine what success looks like.
• Mining Big Data may uncover sales or marketing performance anomalies that would not be otherwise diagnosable. Similarly, AI can help improve energy efficiency and offer insights into customer and employee behaviours.
• Big Data can help process and create models out of vast amounts of information. So as a general rule, the larger and more data-intense the project, the greater the likelihood Big Data could be helpful.

Big Data might be the trendy topic in technology today, but it is more than a buzzword.

Its potential to improve our businesses and our lives over the long term is real.

But that potential needs to be leveraged purposefully and in a targeted fashion. Big Data is not the business equivalent of a wonder drug.

We need to be mindful of where its applications can help and where they are superfluous or harmful.

Indeed, the full promise of Big Data can only be realised when it is guided by thoughtful human expertise.

• Sameer S Somal, CFA, is the CEO and cofounder of Blue Ocean Global Technology. Pablo A. Ruz Salmones is the co-founder and CEO of Grupo Ya Quedó, a software development and artificial intelligence (AI) company.