Data Principles: Sharing Knowledge with Column Metadata


Did you have a sibling growing up?  Maybe an older brother or sister who did not like sharing?  Or maybe there was a neighborhood kid who was not good about being a team player?  Do you remember that frustration when all you wanted to have was a chance with the football, but because it was not yours, you could never touch it?

Do you still get that angry, anxious feeling when you open your data exploration tool of choice, knowing that there are secrets hidden in that data that you have no idea about?  Does the customer on the sales order always have to be the party that pays the invoice?  Should the bill-to address be the ship-to address?  What impact do the shipping terms have when we consider a late payment?

Wouldn't it be nice if that type of information were kept somewhere, besides in the head of the person who has worked with the data the longest?  How mad would you be if I told you there has been a place for this data for years, but it just was not used?

The common data term that describes this type of information is metadata.  In some systems, it is called "extended data".  Other data environments just label it as "description".  No matter how it is labeled, once these areas of your data environment are populated, you will start to unlock the potential of your data.        

Beyond Just Names: The Richness of Column Metadata

Column metadata goes beyond simply assigning a name to a column. It involves documenting vital details about each data element, including its data type, definition, relationships, and any constraints or dependencies. This richness provides context that enables deeper data understanding and facilitates data-driven processes.
Empowering Data Engineers and Analytics
For data engineers:
  • Improved data pipeline building: Well-defined metadata allows data engineers to construct more accurate and efficient data pipelines. Knowing the meaning and characteristics of each column helps them design transformations, mappings, and integrations with greater precision.
  • Faster root cause analysis: When issues arise in data pipelines, metadata provides a detailed roadmap for tracing the origin and transformations of data, enabling quicker identification and resolution of problems.
  • Enhanced data lineage: Metadata facilitates the tracking of data lineage, allowing data engineers to understand how data moves through various systems and processes. This ensures data integrity and supports auditing and compliance requirements.
For data analysts and data scientists:
  • Faster data understanding: Comprehensive metadata allows data analysts to quickly grasp the context and meaning of the data they are working with. This eliminates the need for extensive data exploration and reduces the time spent on data preparation.
  • Improved data quality: Metadata can indicate data quality metrics, enabling analysts to assess the reliability and accuracy of datasets. This builds trust in the data and supports informed decision-making.
  • Streamlined data discovery: Detailed metadata enhances data search and discovery, making it easier for users to locate relevant datasets for analysis or reporting.
Overcoming the Tribal Knowledge Problem
Unfortunately, there are some people who think holding on to how data works makes them important.  In some places, this may not be a power play.  Instead, it is a result of priorities and resources.  The result is the same: a small group of people who understand the data enough to be the ones called on to fix or build anything.  
When column metadata is required, that dynamic is changed.  With the data knowledge democratized, the ability to use and troubleshoot data is given to a much larger group of people. Instead of relying on key individuals who possess undocumented tribal knowledge, the metadata captures this vital information and makes it accessible to the entire organization. This reduces bottlenecks, improves collaboration, and mitigates the risk of losing valuable insights when key personnel depart.  In essence, you give everyone a chance to use the football.
The Effort and the Payoff
While the initial effort of documenting column metadata can be substantial, the long-term benefits far outweigh the costs. Not having proper metadata leads to data littering, delays, wasted resources, and ultimately, a significant hindrance to data reusability and innovation.
Metadata and Data Governance
Metadata management is a key component of effective data governance. It provides the necessary context and information to implement data policies, enforce standards, manage data access, and ensure compliance with regulations. By capturing and tracking metadata, organizations can streamline data governance processes, leading to faster audits, reduced risks, and lower compliance costs.
At the end of the day, your data strategy will not have the impact you need it to have without this core data principle.  Infusing your data model with vibrant column metadata creates a solid foundation for everything that is built on top of it.  The impact is truly limitless.  If we think back to our childhood when we were playing football in the front yard with the neighborhood kids, it is the difference between everyone working together and learning to be a team, and the one kid running home shouting, "it's my ball".  So use this data principle to build a better data platform and a better data team.

Comments

Post a Comment

Popular posts from this blog

Data Strategy: Guiding Principles

Image
  It has been a long time since my last post, and a lot of things have changed.  I have a new job, I have moved to the other side of the state, and I even have a new dog.  But one thing that has stayed with me is my focus on data and how much potential it has.  All of this change has also solidified my understanding of why a data strategy is critical to any success in the data world.  Without a strategy, your effort will be as useless as trying to strike the wind.  You have to be able to aim your blows if you intend to hit anything.  In our case, we want to "hit" that business objective, the one thing that the business must have to progress to the next phase in its development.   The thing I have come to understand is that when the term "Data Strategy" is used, most people in management jump right to the end and focus on things like cost or "vendor lock-in".  Some people in management focus so much on what they perceive the outcome will ...
Read more

Data Principles: The Power of Naming Standards

Image
In my last blog, I highlighted five data principles that I have used in the past to create a solid data strategy.  The idea behind a principle is that it is flexible to adjust to changing needs but rigid enough to maintain order.   A good example of this is the golden rule, "do unto others as you would have them do to you".  When you apply this principle, you are free to do whatever you want, as long as it is something you want done to yourself.  It is a very flexible principle that can have serious consequences if broken.   One crucial principle that forms a solid foundation for your data initiatives is  naming standards . It might seem like a simple concept, but consistent naming conventions throughout your data environment can significantly impact efficiency, governance, and understanding for everyone involved.  With the right naming standards, looking into the data environment as a DBA, developer, or user, there will be no question about ...
Read more

Data Governance: Building good bones

Image
The next step in our data governance journey will be setting up the data governance framework. In some ways this is just like the frame that your home is built around. In a home the frame lays out the exterior and interior walls, provides a place for electrical and network wiring, a place for insulation, and more importantly exterior and interior wall coverings like brick and sheetrock. The data governance framework works in the same way. By using principles and guidelines it provides structure around your data so that you can implement things like data cleansing rules, roles around data, and the tools that you allow the business to use. The end goal is to make sure that your data is being used in a consistent and compliant way so that everyone using it gets the same answers. You can see why this is a task that is easier said than done. Your data governance framework should include the following components: Data governance roles and responsibilities : This section should de...
Read more