Thought Leadership, Innovation, Business Growth, AI
“In God we trust. All others must bring data.”
This quote by W. Edwards Deming has never been more relevant. As enterprises rush to adopt GenAI, many are hitting a hard wall, not because of a lack of tools, but because of something far more fundamental: the quality and usability of their data.
According to Gartner, 30% of GenAI projects will be abandoned by the end of 2025 due to poor data quality, unclear business value, and lack of risk controls.
Technology in IT is constantly changing. We’ve moved from mainframes to the cloud, and now to AI and GenAI. But what’s different today is the speed. New tools emerge almost daily, including AI models, LLMs, chips, and others.
Here’s another interesting stat from Gartner: 39% of people surveyed said that a lack of data is one of the biggest barriers to using AI.
Isn’t it ironic that a lack of data is one of the top barriers to technology advancements in an era where we find a lot of data everywhere in our enterprises? Problem of plenty!!
When every organization produces and consumes vast amounts of data every minute, it brings us to this situation. Let’s zoom in to find out.
Is a Lack of Data the Real Reason?
The simple answer is no. While data is everywhere, the real challenge is finding data that’s actually useful for a specific purpose.
GenAI breaks through those walls. It does more than follow rules; it interprets them, adapts to new data, and reasons through scenarios with human-like flexibility. I saw it firsthand.
Before we explore what “fit-for-purpose data” really means, let’s start with a few simple questions:
- Do we know all the data that exists across our IT and business systems?
- Are we actually using the data we already know about?
- Are we getting real value from it?
- And finally, what more could we solve if we had better or more useful data?
How many of us know about “Dark Data”?
Gartner defines dark data as “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.” Surprisingly, 80% to 90% of enterprise data is considered “dark data,” or information that organizations are unaware they have. As a result, it often goes unused and untapped.
This stat shows the real issue—it’s not that we lack data, but that we lack usable data. Most of it is hidden, unstructured, and never put to use.
Let’s come back to the idea of “fit-for-purpose” data. Just like one size doesn’t fit all, the same is true for data. To make data truly useful, we need to understand its key attributes, such as volume, quality, variety, security, and more.
We need to build a culture where data is treated like any other product. Like a product, data is an asset—meant to be trusted, well-managed, and ready to meet the needs of different users.
Engineering Mindset for Data
It’s time to rethink how we view data. Like any asset, data has a lifecycle from creation to deletion. To unlock its full value, we need to apply solid engineering principles at every stage. That’s how we make data more meaningful and useful.
Data engineering has been a well-established practice, widely used across many industries. Today, numerous frameworks, platforms, and technologies are available to support every stage of the data lifecycle.
But the key challenge now isn’t the lack of tools, but how we view and treat data. Making the most of these resources requires a shift in mindset that must start at the executive level and extend to operations teams and field support engineers.
As mentioned earlier, everything starts with awareness: knowing what data exists. Only then can we take meaningful next steps. One of the best ways to uncover existing data is by creating a Data Dictionary. This tool lists all available data sources along with their key attributes and dimensions. Building one from scratch can be a surprisingly insightful exercise.
Some organizations already have data dictionaries tailored to specific industries, services, or domains. Partnering with them can help surface your own “dark data” and bring it into active use. Ultimately, the organizations that bring the most data out of the dark—and utilize it strategically—will be the ones that thrive.
Data: The Key Driver of Decisions
Now that we know what data exists in the landscape, it’s time to make the most of it. But how?
Again, there should be a systematic approach to this. The traditional Data → Information → Knowledge → Insights → Wisdom (DIKIW) framework comes to our rescue.
A single data point can be powerful in solving business problems—but correlating multiple data sources is a whole different beast.
The DIKIW framework helps shift our perspective on data. Applying it allows us to uncover the knowledge and insights that lead to better decision-making and the wisdom to tackle new challenges.
This approach leads us to the next step: building a catalog of individual and correlated data sources, along with the knowledge and insights they provide.
The Real Work Begins Now
A good beginning is only half the journey. The other half requires careful planning and precise execution to achieve success. Each one of us is a catalyst in making that happen. When the basics are solid and the foundation is strong, we can move forward with confidence knowing the path ahead is built to last.
Insights are how data speaks. To unlock its value, we must listen, understand, and act, especially when driving innovation through automation or AI/GenAI solutions. A mature data assessment helps uncover business problems that can be solved using the data you already have.
At Astreya, our Data Maturity Assessment helps IT leaders uncover hidden data sources, prioritize high-impact use cases, and build a roadmap to smarter, AI-ready operations. Want to know where your data stands?
Data engineers, data scientists, and domain experts play a critical role in data assessments. They help identify high-impact ideas and solutions to solve problems today and prepare for tomorrow.
There are many platforms and tools available to speed up assessments, build solutions, and manage data governance. The right expert can guide you to the best-fit tools for your unique journey.
Throughout the entire process, one thing is non-negotiable: strong data security, privacy, regulatory compliance, and governance.
These are not optional; they are mandatory.
When people, data, and technology come together, we become future-ready. Every innovation sparks new challenges. And businesses can tackle any challenge with their secret weapon: data.
On a lighter note: In today’s world, it might be more accurate to say, “In Data We Trust. All others must bring ‘God’ mode.”
Karthikeyan Gangatharan
Director – AI Automation and Data Engineering
Visionary leader with nearly 20 years of experience delivering customer value through AI-led automation and data analytics. Recognized thought leader in driving business transformation through data engineering and AI automation. Specialized in delivering data engineering services through catalog-based due diligence and solving complex business problems using data insights and intelligent solutions. Passionate about leveraging technology to create meaningful value and solve client challenges at scale.