The Big Data Pyramid | From Data to Wisdom

We have seen big data analytics go from a hip buzzword to a flourishing market accounting for nearly $10 billion, in less than a couple of decades. Moreover, the big data analytics (BDA) market is forecasted to grow at a rate of 29.7% to $40.6 billion by 2023.

Big data is here to stay and, like it or not, big data is bound to envelop every field of human activity, at least to some extent. There is a tremendous amount of content online on the topic of big data and its related fields, such as Business Intelligence (BI), the Internet of Things (IoT) Artificial Intelligence (AI), high-performance cloud computing, and automation.

However, the objective of this article is to assert the premises that allow us to contemplate a data-driven future and the significance of stepping into the brave new world of big data analytics and automation.

More importantly, we are going to explore the fundamentals of big data, to better grasp how big data has been influencing business decision making and how it is going to shape businesses and company cultures throughout the years that lie ahead.

What Is Big Data and Why Do We Care?

If it is digital, it is data. Additionally, if we collect data in large quantities (Volume), at a fast pace (Velocity), and in a large variety of formats (Variety), it is big data.

From the forms we fill in online every week, and the links we click on our social media feeds to our swipes and dwell times on webpages, data is being continuously collected, analyzed, and further used to derive actionable insights and automated actions. Increasingly more sophisticated algorithms are being designed to learn everything about us and our activities online.

The Definition of Big Data

Big data is the expression we use to encompass both a data-centric approach to our cultural, societal, military, political, and business endeavors, and the technologies that we use to enable it.

However, big data is not restricted within the borders of the online world. With the advent of cloud computing and the IoT, our homes, shopping malls, public institutions, our streets, and the cities themselves are getting increasingly smarter.

We are gathering more data than ever, and we are slowly starting to grasp the potential of a smart, interconnected civilization that prospers due to well-informed, big data decision making it relies on.

The Potential of Big Data

For public administrators, politicians, and state legislators, big data and its entailing plethora of data-driven insight can translate into better, less wasteful, more effective public policymaking, along with the means to choose the best possible paths toward our common goals.

For the average citizen, big data potentially translates into better-informed, more transparent democratic elections, better purchasing decisions and, possibly, better life choices altogether.

Envisioning our data-centric, data-driven future is worth a separate article. Until then, let’s dive into the business implications of big data:

Big Data Business | A Constant Hunt for Insights

As companies mature from information-centric to data-driven organizations, they progress in their use of data management and analytics, typically, from a strictly descriptive approach to diagnostic, predictive, and prescriptive analysis models. More and more, organizations act on the basis of data-informed insight and data-driven predictions.

Data-driven organizations share a few key attributes:

• Emphasis on data collection
• Consistent investment in tools and skills to gather and analyze data
• Commitment to making data broadly accessible
• Willingness to consider data-driven ideas arising at every level of the business
• Dedication to ongoing, data-driven progress

Data-centric and data-driven organizations collect copious amounts of data – and many types of data, for that matter. To more accurately grasp how they do it and to what ends, let’s explore a few fundamentals of the big data field:

The 10 V’s of Big Data

Big data is not entirely about volume, velocity, and variety as previously mentioned. Specialists have identified seven additional characteristics that must be considered when attempting to grasp the concept of big data, and when an organization dives into the intricacies of implementing big data projects. Conveniently, all the characteristics of big data start with the same letter. V for Big Data:

1. Volume: no surprise here. More than 90 percent of all the digital data was generated in the past two years. The current rate at which we generate data is astounding. For example, 300 hours of video content is uploaded to YouTube every single minute.
2. Velocity: it refers to the rate at which data is being generated. Here are a couple of baffling figures: Facebook claims 600 terabytes of daily incoming data, and Google processes on average approximately 40,000 queries every second, which translates crudely to 3.5 billion searches every single day.
3. Variety: we will expand on this topic soon since we are developing a content piece that deals with the imperative of data hygiene. Nonetheless, keep in mind that about 90% of all available data is unstructured data or semi-structured data. The good old columns and rows of tabulated data, along with its traditional methods of analysis will not get you very far in the case of unstructured data.
4. Variability: in the context of big data, variability refers to a couple of different things. First of all, variability deals with data pollution, meaning the number of discrepancies in the data itself. Anomaly and outlier detection methods help us iron out discrepancies. Although it is often unfittingly confused with variety, variability does describe a big data characteristic related to variety. Multiple incongruent data types and sources result in a multitude of data key points.
5. Veracity: big data has a lot to suffer thanks to veracity. The veracity (confidence or trust) in the data tends to drop as any or all of the properties mentioned above increase. Veracity mainly refers to the origin and reliability of the data, its context, and its relevance to the analysis it entails. Veracity is a major concern, especially in the context of data-driven, automated functionalities.
6. Validity: similar to veracity, validity raises the question of how correct the data is for its intended use. Data scientists spend about 60% of their time cleansing their data before even considering analysis. Big data analytics is only as reliable as its underlying data, so great data governance is imperative in the field of big data.
7. Vulnerability: privacy and security need to become part of the DNA of any organization that collects, stores, and processes data. There is no way around it. A breach of big data is not just a breach. It is a BIG breach.
8. Volatility: before long, certain data becomes obsolete. Old data is seldom useful data. Moreover, organizations need to establish clear rules for data exchange and availability as well as ensure speedy retrieval of information.
9. Visualization: when people refer to big data as a really hard endeavor, they usually refer to the visualization part of big data. It might not be the most technologically difficult task, but the use of graphs to tell complex stories is extremely difficult. Visualization refers to making the vast amounts of available data comprehensible in such a way that is easy to translate into actions. Visualization is responsible for the elusive insights everybody is on the hunt for.
10. Value: big data means big business. If the return on investment would not cover for the troubles of doing big data analytics, we would all still rely on opinions and best practices.

Understanding customers better and targeting them in view of our understanding, optimizing procedures and workflows, and improving business performance are just a few of the money-making and money-saving benefits of big data.

The Big Data Pyramid (DIKW Pyramid)

Try as hard as you will, you can’t truly manage that which you can’t measure. James Love Barksdale, an American executive who served as the president and CEO of Netscape, is famous for confidently asserting:

If we have data, let’s look at data. If all we have are opinions, let’s go with mine.

There is a lot of wisdom in these assertions. Relying on experience and expertise, rather than on bad data or opinion, is a rational, wise decision. Nonetheless, I am a strong upholder of the conviction that data itself can be derived into wisdom. Let me elaborate.

The DIKW pyramid or DIKW hierarchy refers to a model representing structural and functional relationships between data, information, knowledge, and wisdom. “Typically information is defined in terms of data, knowledge in terms of information, and wisdom in terms of knowledge” (Jennifer Rowley, “The wisdom hierarchy: representations of the DIKW hierarchy”. Journal of Information and Communication Science, 2007)

Data-Driven Wisdom & Wisdom-Driven Decision Making

Based on the DIKW pyramid model, let’s illustrate the data-driven decision process. This classical example describes a military decision-making support system, but the conceptual architecture is easily transferable to data-driven businesses:

1. The chain starts with quality data being fed into the command and control systems.
2. The resulting information is assessed in terms of completeness, correctness, currency, consistency, and precision. If it scores sufficiently on these scales, information is assimilated as knowledge.
3. The quality of the resulting knowledge is assessed against procedural knowledge embedded in the command and control systems such as templates for adversary forces, assumptions about entities such as ranges and weapons, and doctrinal assumptions, often coded as rules.
4. The resulting awareness (or wisdom) is assessed in terms of its degree of usefulness within the knowledge base embedded in the command and control systems.

Swap out the military elements described in the third pointer and replace them with business indicators and instruments, and you find yourself in possession of a powerful decision-making model that fully relies on relevant quality data.

Let’s Wrap Up!

Data-driven organizations are fully aware that data alone is completely useless. Data becomes valuable through analysis by the correct algorithms to retrieve the information necessary to fuel the best possible business decisions.

Companies with great big data strategies have information-centric organizational cultures that encourage employees to make good use of the data-driven processes and resulting insights, every day. Additionally, data-driven companies stay ahead of the curve through innovation.

Once you find better ways, you seldom choose to stay on the old path, right?

No matter where you are with your data-centric endeavors, it will all boil down to gathering quality data. 123FormBuilder can give a big helping hand with your big data:

 

 

Load more...