Big Data vs Small Data: How Less Information Can Deliver More Insights

Big Data holds enormous business potential, but is it making organizations miss the trees for the forest, and does sweating the little things matter as much as the big picture?

The importance of data for business organizations in an era of “data is the new oil” is beyond debate. For a business, big data refers to data sets that are too large or complex to be dealt with by even traditional data-processing application software, such as spreadsheets, much less by human comprehension. On the other hand, small data refers to data that is ‘small’ enough for human understanding, analysis, drawing insights, and creating follow-up action.

Big data analysis involves capturing data, storing it, retrieving its relevant bits, analyzing it, and creating insightful visualizations to create actionable strategies and tactics. The potential of big data is immense, and there is a genuine concern that we may be headed towards a permanently skewed world where large organizations have an unbreakable moat because of their access to big data and the tools to use it. While businesses have been prone to over-optimism in the past, data analytics is living up to its potential and more – e.g. Gartner estimates that 42% of surveyed sales leaders rate the ROI from their sales analytics initiatives as significantly higher than expected. 

 However, a false sense of security stemming from having lots of data and loads of processing power and the genuine ability of data and analysis to overwhelm to the point of indecisiveness is now a widely acknowledged risk of this data dependency. 

Pablo Picasso (apparantly) famously said, “Computers are useless; they can only give you answers.” 

His thought may be old but is far from dated. AI-based content writing apps are a good example of this – they might be able to produce good marketing copies but they also produce gibberish if the questions require curiosity and creativity. In the data-rich age we live in, answers may come more quickly than (the right) kind of questions. Without asking the right questions, we risk not just heading down the wrong path but doing so with great misplaced confidence. The risks grow exponentially in business situations that are inherently dynamic/subject to rapid change or with outcomes that involve subjectivity and uncertainties due to human behaviour. Data analytics follows bounded rules, while humans who make up the real world seldom behave by the book that guides analytics. In such cases, the haystack (big information) obscures the needle (insight), while users walk away with a detailed, great-looking, albeit useless, description of the hay. Such situations may require thinking small rather than big.

Illustration by Ole Kaarsberg

When small is big

Small data is gathered in a less mechanical and more personal way. The exercise often begins with thoughtful observation, inquiry, and questions. For example, LEGO spends time in these brilliant endeavours to obtain clues on what its consumers (kids and adults alike) want. Observing behaviour provides LEGO with insights that any number of quantitative surveys cannot. These observations and insights helped reverse LEGO’s slide into bankruptcy through pivots like its ‘Mindstorms’ models and ramping up the complexity of its regular models to instill a greater sense of achievement and pride in users. Often seen as ‘hiding in plain sight,’ small data offers several advantages:

• Gathering through observation, intuition, and interaction outside the machine world is cost-effective.
• Small data is easier to manage, maintain, and analyze. While some may lament the subjectivity of this analysis, remember that the logic and algorithms of big data analysis also involve subjectivity.
• It can provide actionable insights in particular situations that require a deeper and more nuanced understanding of customers at the human level, making it better suited for customized and personalized solutions and niche markets or products.
• Counterintuitively, small data can allow for quicker decision-making and innovation that is better understood by everyone involved. A side benefit: it enhances collaboration and communication between team members.
• By its very nature, small data doesn’t suffer from the risk of data breaches that plague big data sets.

Touch and feel domains such as education and healthcare are particularly ripe for small data insights. For too long, policymakers have tried to use big data to gain insights that can improve student outcomes. However, these data-driven insights overlook several inherent factors, such as individual needs, the role of emotions, and the influence of human relationships in teaching and learning. Moreover, data analytics can draw correlations between educational variables but cannot conclusively point to causality. Coming to healthcare, Cornell University’s Small Data Lab has developed apps that focus beyond merely gathering data on patients. These apps also track patient sleep, shopping, and exercise habits. Understanding how these habits influence underlying health conditions, particularly unpredictable flareups, helps predict adverse events.

The utility of small data lies in both its directional as well as corroborative power. To make the most of it, insights from small data must be coupled with harder, more conventional analysis. As Martin Lindstrom states in his seminal book ‘Small Data: The Tiny Clues That Uncover Huge Trends,’ “a lone piece of small data is rarely meaningful enough to build a case or create a hypothesis but blended with other insights and observations gathered from around the world, the data eventually comes together to create a solution that forms the foundation of a future brand or business.” This is especially true for domains such as retail financial services, asset management, and fintech, which generate tons of data but are also ripe with human-level insights. These sectors have been leveraging some serious AI/ML capabilities for data analytics but hold immense potential for using readily available small data. Moreover, internet platforms and apps now drive transactions, making collecting small data more manageable and cheaper. Small data-powered AI/ML models can help solve challenges that analytics performed on large data sets may overlook.

Bottomline

Big data certainly has undeniable and proven benefits. One is its utility in handling larger volumes; the cost-effectiveness of small data analysis naturally comes down as the volume of small data increases, while the converse is true for big data.

Moreover, we have only begun to scratch the proverbial surface of the potential of big data analytics. Still, small data should not be overlooked or underestimated, and the filter for situations in which to rely on small data is good old-fashioned common sense. The advantages of small data, such as its simplicity, cost-effectiveness, and ability to provide more focused and personalized insights, make it a valuable tool for businesses and researchers. Ultimately, choosing between big or small data will depend on specific needs and goals. By understanding the strengths and limitations of both approaches and combining insights from both methods, businesses can make better-informed decisions leading to better outcomes.