Data Quality: Know before you load

 

Have you ever made a decision that you knew was right, just to watch it blow up in your face?  After the fallout out settled down, were you able to look back and see where you went wrong?  In most cases, the common flaw was not enough information or wrong information.  At some point, you either believed what you were told or you made an assumption about something without taking time to validate what you were thinking. 

When this happened to you, it was devastating, but the impact of this bad data was localized to you.  Have you ever wondered what would happen when bad data makes its way into a large decision?

I thought you checked that..

Let’s talk about a situation where data quality mattered…to the tune of $327.6 million.

In 1999, the NASA Mars Climate Orbiter was lost during its mission due to a navigation error. One engineering team was using metric units, while another was using imperial units for crucial navigation calculations. The discrepancy went unnoticed, leading to the spacecraft flying too close to the Martian atmosphere and disintegrating. This incident serves as a stark reminder of the mission-critical nature of data quality and the devastating consequences of neglecting it.

Simple Steps, Big Impact: Embedding Data Quality in Your Pipelines

What went wrong?  Frankly, there was no system of checks and balances in the data they were looking at.  And I don’t mean manual checks.  I am sure there were people looking at numbers, but humans cannot look at every number.  However, by integrating data quality practices throughout your data lifecycle, you can systematically eradicate bugs like this before you experience their impact.  Here are three simple, yet powerful, ways to ensure data quality at every step:

  1. Standardized Data Formats and Units: Establishing and enforcing consistent data formats and units across all data sources is fundamental. Clear data quality metrics help understand overall data health.
  2. Automated Data Validation and Monitoring: Regularly validate data against predefined rules and standards. This helps detect and correct errors and inconsistencies early. Data quality tools can automate these processes, streamlining the process of identifying and resolving quality issues.
  3. Comprehensive Data Governance: Implementing a robust data governance framework is key. This includes defining data ownership, establishing roles and responsibilities, and documenting data processes and systems.

The Cost of Neglect and the Power of Prevention

The Mars Climate Orbiter disaster demonstrates that investing in data quality is not an option; it's a necessity. The cost of neglecting data quality includes financial losses, missed opportunities, and erosion of trust. By contrast, proactively addressing data quality from data inception to decision-making fosters a culture of reliability and confidence in your data.

This proactive approach builds a strong foundation for your data strategy, enabling the organization to:

  • Make informed decisions: Ensuring data accuracy, consistency, and completeness empowers decision-makers with trustworthy insights.
  • Boost operational efficiency: Streamlined data processes and reduced errors lead to increased productivity and reduced costs.
  • Enhance customer experiences: Personalized services and targeted marketing campaigns built on high-quality data improve customer satisfaction.
  • Gain a competitive advantage: Leveraging accurate and reliable data provides a competitive edge in today's data-driven landscape.

Ultimately, investing in data quality is an investment in the future of your organization. It ensures that your data platform delivers reliable insights that drive informed decisions and business success.

The Mars Climate Orbiter is not the only project that has ended in failure because of bad data.  The list can go on and on; however, one thing remains.  Bad data has real-world impacts.  Take the steps to embed data quality into your culture and your pipelines so that you can avoid crashing and burning like the orbiter.

If this post helped you see the value of prioritizing data quality, share it with your network!


Comments

Post a Comment

Popular posts from this blog

Data Strategy: Guiding Principles

Image
  It has been a long time since my last post, and a lot of things have changed.  I have a new job, I have moved to the other side of the state, and I even have a new dog.  But one thing that has stayed with me is my focus on data and how much potential it has.  All of this change has also solidified my understanding of why a data strategy is critical to any success in the data world.  Without a strategy, your effort will be as useless as trying to strike the wind.  You have to be able to aim your blows if you intend to hit anything.  In our case, we want to "hit" that business objective, the one thing that the business must have to progress to the next phase in its development.   The thing I have come to understand is that when the term "Data Strategy" is used, most people in management jump right to the end and focus on things like cost or "vendor lock-in".  Some people in management focus so much on what they perceive the outcome will ...
Read more

Data Principles: The Power of Naming Standards

Image
In my last blog, I highlighted five data principles that I have used in the past to create a solid data strategy.  The idea behind a principle is that it is flexible to adjust to changing needs but rigid enough to maintain order.   A good example of this is the golden rule, "do unto others as you would have them do to you".  When you apply this principle, you are free to do whatever you want, as long as it is something you want done to yourself.  It is a very flexible principle that can have serious consequences if broken.   One crucial principle that forms a solid foundation for your data initiatives is  naming standards . It might seem like a simple concept, but consistent naming conventions throughout your data environment can significantly impact efficiency, governance, and understanding for everyone involved.  With the right naming standards, looking into the data environment as a DBA, developer, or user, there will be no question about ...
Read more

Data Governance: Check and Check Again

Image
We are progressing in our data governance by building out the framework.  We are moving data into areas that the business can use it but how do we know they should use it?  More importantly how do they know that they CAN use it? So much of what we do relies on trust.  The business trust that we will understand what they need so that we can deliver reliable data.  The first time they find an issue that trust is broken.  Part of our framework builds out data validation processes but how do we put that into practice? One of the first things we will need to do is check ourselves. Y our data governance framework can be used to create natural data quality checkpoints. Is your data pipeline ready to migrate? Did you check it? When your process captured the requirements did you ask who in the business would validate the work? Your processes are the first place to start checking your data. Some specific examples of how you can use your data governance framework to cr...
Read more