Data Governance: Check and Check Again


We are progressing in our data governance by building out the framework.  We are moving data into areas that the business can use it but how do we know they should use it?  More importantly how do they know that they CAN use it?

So much of what we do relies on trust.  The business trust that we will understand what they need so that we can deliver reliable data.  The first time they find an issue that trust is broken.  Part of our framework builds out data validation processes but how do we put that into practice?

One of the first things we will need to do is check ourselves. Your data governance framework can be used to create natural data quality checkpoints. Is your data pipeline ready to migrate? Did you check it? When your process captured the requirements did you ask who in the business would validate the work? Your processes are the first place to start checking your data.

Some specific examples of how you can use your data governance framework to create natural data quality checkpoints are:

  • Require that all data be validated before it is loaded into a data warehouse. This could be done using data validation rules or by running the data through a data quality tool.
  • Require that data be audited on a regular basis. Even long running and stable data feeds can fall victim to bad data so you have to make sure that it is correct and complete periodically. This could be done using a data audit tool or by manually reviewing the data.
  • Set up alerts to notify you when data quality issues are detected. This will help you to identify and address data quality issues quickly.
  • Create a data quality dashboard to track key data quality metrics. This will help you to identify trends in data quality and to identify any areas where improvement is needed. In the long run this is probably one of the best ways to automate known issue detection using a set of standard KPI's
Data quality and validation is important but most organizations do not have the resources to devote an entire team to the process. This is where automation really shines. There are a number of ways to automate data quality validations. One way is to use data validation rules. Data validation rules are sets of criteria that data must meet in order to be considered valid. These rules can be baked into ETL/ELT and fail the data before it even gets loaded. With a good set of processes to deal with the errors you could even "self heal" the data by alerting and presenting the business with ways to fix the data at the source.

Another way to automate data quality validations is to use a data quality tool. Data quality tools can be used to identify and correct a variety of data quality issues, such as missing values, duplicate records, and invalid data formats.

Here are some specific examples of how you can automate data quality validations:

  • Use data validation rules to validate data before it is loaded into a data warehouse.
  • Use a data quality tool to identify and correct data quality issues on a regular basis.
  • Set up automated alerts to notify you when data quality issues are detected.
  • Create a data quality dashboard to track key data quality metrics and to identify any areas where automation could be used to improve data quality.
Look for these traits when you are investigating a data quality tool. One other thing to look into would be how easy it would be for the business to implement rules. The closer you can get that step to the business the better off your data will be.

There are many different tools out there that you can use to automate this task. Just a few are:

  • Informatica: Informatica offers a range of data quality and data governance solutions, including Informatica PowerCenter, which can be used to automate a variety of data validation tasks.
  • Talend: Talend offers a cloud-based data integration and data quality platform that includes a variety of data validation features.
  • Trillium Software: Trillium Software offers a range of data quality and data governance solutions, including Trillium Quality Center, which can be used to automate a variety of data validation tasks.
  • IBM: IBM offers a range of data quality and data governance solutions, including IBM Information Governance Catalog, which can be used to automate a variety of data validation tasks.
  • SAP: SAP offers a range of data quality and data governance solutions, including SAP Information Steward, which can be used to automate a variety of data validation tasks.
Remember all of these tools need to tie back into the overall data strategy.  In some cases a data quality tool can pass rules directly to an ETL tool.  If you are in an ecosystem using a suite of tools check to see if it covers data quality.  You may already have this problem resolved.  

The reality is that even if you think your data is right, you need to check it again.  Your reputation is only as good as your data.  Remember to build out the processes in your framework, get the right tool to automate and then put in the work to check and recheck the output.  That hard work will pay off when the business stops asking if your data is right and starts making informed decisions with it.

Comments

Post a Comment

Popular posts from this blog

Data Strategy: Guiding Principles

Image
  It has been a long time since my last post, and a lot of things have changed.  I have a new job, I have moved to the other side of the state, and I even have a new dog.  But one thing that has stayed with me is my focus on data and how much potential it has.  All of this change has also solidified my understanding of why a data strategy is critical to any success in the data world.  Without a strategy, your effort will be as useless as trying to strike the wind.  You have to be able to aim your blows if you intend to hit anything.  In our case, we want to "hit" that business objective, the one thing that the business must have to progress to the next phase in its development.   The thing I have come to understand is that when the term "Data Strategy" is used, most people in management jump right to the end and focus on things like cost or "vendor lock-in".  Some people in management focus so much on what they perceive the outcome will ...
Read more

Data Principles: The Power of Naming Standards

Image
In my last blog, I highlighted five data principles that I have used in the past to create a solid data strategy.  The idea behind a principle is that it is flexible to adjust to changing needs but rigid enough to maintain order.   A good example of this is the golden rule, "do unto others as you would have them do to you".  When you apply this principle, you are free to do whatever you want, as long as it is something you want done to yourself.  It is a very flexible principle that can have serious consequences if broken.   One crucial principle that forms a solid foundation for your data initiatives is  naming standards . It might seem like a simple concept, but consistent naming conventions throughout your data environment can significantly impact efficiency, governance, and understanding for everyone involved.  With the right naming standards, looking into the data environment as a DBA, developer, or user, there will be no question about ...
Read more

Data Governance: Building good bones

Image
The next step in our data governance journey will be setting up the data governance framework. In some ways this is just like the frame that your home is built around. In a home the frame lays out the exterior and interior walls, provides a place for electrical and network wiring, a place for insulation, and more importantly exterior and interior wall coverings like brick and sheetrock. The data governance framework works in the same way. By using principles and guidelines it provides structure around your data so that you can implement things like data cleansing rules, roles around data, and the tools that you allow the business to use. The end goal is to make sure that your data is being used in a consistent and compliant way so that everyone using it gets the same answers. You can see why this is a task that is easier said than done. Your data governance framework should include the following components: Data governance roles and responsibilities : This section should de...
Read more