Integrating the Dataverse

 Have you ever gone into a meeting with key business stakeholders and left more confused than you were when you went in?  When you think back about that meeting, did it seem like the two groups in the room were not speaking the same language?  You are probably not all wrong if you thought that.  You see, even if both parties were talking the same language, how they defined their words was probably different.   So, do you want the decoder ring? By The Power Of The Data Glossary! When you are facing this problem, you are missing a data glossary.  This is a data governance component that is very special.  Do not confuse it with the data dictionary.  A data dictionary is a metadata store about columns and the data in the columns.  More factual verbiage like: Column Color holds a list of colors available for a product.  A data glossary is a business-managed list of terms that provides direction or helps people to understand a busin...

  Whenever you are triaging a problem, you have to focus on what is in front of you, but you cannot forget about how your actions will impact your overall desired outcome.   For example, let’s imagine you are managing a thriving new software company.   Your software has gone viral, and you have to grow ASAP to handle support and new development.   What is your knee-jerk reaction?   Get as many bodies as you can.   You bring in a wave of contractors, from project managers to developers.   Your short-term needs are met, but in 3 months, what happens?   You grew too fast, and customer complaints are starting to expose the weakness of not having a fully functional organization.   A data governance practice usually has the same problem.   There is always one pillar that needs more immediate attention at the beginning: data quality.   Even if that is the case, you cannot pour all of your gas on that fire; you still have to plan for the ...

  Have you ever made a decision that you knew was right, just to watch it blow up in your face?   After the fallout out settled down, were you able to look back and see where you went wrong?   In most cases, the common flaw was not enough information or wrong information.   At some point, you either believed what you were told or you made an assumption about something without taking time to validate what you were thinking.   When this happened to you, it was devastating, but the impact of this bad data was localized to you.   Have you ever wondered what would happen when bad data makes its way into a large decision? I thought you checked that.. Let’s talk about a situation where data quality mattered…to the tune of $327.6 million. In 1999, the NASA Mars Climate Orbiter was lost during its mission due to a navigation error. One engineering team was using metric units, while another was using imperial units for crucial navigation calculations. The dis...

  Have you ever watched a nature documentary where animals hunt as a pack?  They all work together to take down the prey, but what happens after there is no more need for teamwork?  Suddenly, the animals break into smaller groups and start fighting amongst themselves for the food.  Usually, the strongest gets its fill and then the rest take the scraps.  I can already hear you asking yourself,   “What does the animal kingdom have to do with data?”.   In some organizations, what I just described plays out every day; the only difference is that the animals are wearing very expensive suits.   You see, the organization may have a façade of unity, but internally, they are all trying to drive their own agendas using their own version of the data.   Most of this time, the intent is not malicious.   Each department really feels they are looking at the data the right way and are seeing the right patterns to give them justification to move the ...

 Data governance is the cornerstone of a healthy and effective data strategy. It's the framework that ensures your data is reliable, secure, and valuable across the entire organization. Think of it like a three-legged stool. Each leg – Data Quality, Data Sharing (with Data Catalogs & Glossaries), and Data Lineage – is crucial for stability. If one leg is shorter than the others, the stool becomes wobbly, just as neglecting one area of data governance weakens the entire framework. Data Quality : This leg ensures data is accurate, consistent, complete, and reliable. Data Sharing : This leg promotes discoverability and understanding through tools like Data Catalogs (inventories of data assets) and Data Glossaries (standardized business term definitions). Data Lineage : This leg tracks the data's journey, providing transparency into its origins and transformations. These three pillars work together to creat...

  As we continue to remove the mystery around data strategies, we will spend the next few weeks diving into the components and principles that make up data strategies.  My hope is that as you start putting the pieces of this puzzle together in your own mind, you will start to see that with a methodical and measured approach, a solid data strategy is within your grasp.  Today, we are going to start focusing on data governance.   Data governance is a key component of a successful data strategy. It ensures data reliability, security, and usability throughout an organization. Data governance manages data from creation to deletion. It involves defining roles, responsibilities, processes, policies, and standards that guide data management and protection. But don’t worry.   We are going to take our time.   Today, we will cover data governance at a high level, and as the days go by, we will get into the details.   When we are done, you will all have greate...

  Building and executing a data strategy is hard, and if anyone tells you they can do it for you in 3 easy steps, they are lying.  The fact is, there are quite a few steps involved that can seem overwhelming.  Some of these steps can take an organization years to cultivate, like building a culture that not only relies on the data but demands to have it to make any decision.  A key thing to remember, however, is that a data strategy does not need to be fully baked when you start down the road of implementing it.  When you build using strong data principles, you will be able to move forward providing business value, and your data strategy can flex as you fill in the details.  The Main Parts of Your Data Strategy: So what are the parts of a data strategy, which ones do you need right away, and how can your data principles help fill in the gaps?  Below are 10 core parts of a data strategy that you will need to at least consider before starting. ...

  At first glance, the value of nothing is not significant.   In the world around us, we literally walk past “nothing” all the time and don’t even notice it.      But at times, “nothing” is just as important as something.   For example, if you were walking on a city street and someone removed a manhole cover.   That “nothing” would certainly have a huge impact if you fell into the manhole. Another way “nothing” can impact you is that it allows others to make assumptions.     Have you ever been in a conversation with a friend when they asked you a question you did not want to answer?   Instead of telling them, “I would prefer not to comment,” did you just leave your comment unsaid, just hanging there in the air?   You were hoping they would infer that you did not want to comment, but is that what happened?   You see, when you did not specifically answer the question, you allowed your friend to answer for you, and that is whe...

When designing your data model, especially in data warehousing or analytical systems, you might opt for identity columns as primary keys. This is a common and often advantageous practice due to the inherent benefits of identity columns. However, relying solely on these generated keys can create a disconnect with how the business views and interacts with data. This is where the data principle of utilizing auxiliary keys becomes crucial. Identity Columns: The Physical Key to Optimization Before we dive into how the principle of using auxiliary keys works, we have to define what we would use as a primary key.  In the world of data warehousing, it is common practice to use identity columns as the primary key of every table.  These auto-incrementing integer columns offer several advantages in database design, but as we will see, they do have flaws.  Some advantages are: Compact Storage: Identity columns are generally small, fixed-size data types (like integers), which makes t...

As a child, did you enjoy doing your chores, or did you think your parents were just being mean?  As a parent, even though you know your child is not happy with you when you enforce the chore list, why do you do it?  Certainly, in the current moment, enforcing chores is not easy.  Making your child comply takes effort and consistent oversight.   Who has time for that, right? But after 18 years of that effort, what do you end up with?  Isn't it true that these chores build a solid foundation for your child to stand on as they enter the adult world?  Didn't all of that constant oversight and hard work pay off as you proudly watch your very capable child move on to the next challenge, knowing they would be able to conquer it?  When implementing a data strategy, you are faced with a similar question.  Where do you put your time, effort, and energy?  Do you go quickly, bypassing things you know will be problems in the future for the quick win...

Did you have a sibling growing up?  Maybe an older brother or sister who did not like sharing?  Or maybe there was a neighborhood kid who was not good about being a team player?  Do you remember that frustration when all you wanted to have was a chance with the football, but because it was not yours, you could never touch it? Do you still get that angry, anxious feeling when you open your data exploration tool of choice, knowing that there are secrets hidden in that data that you have no idea about?  Does the customer on the sales order always have to be the party that pays the invoice?  Should the bill-to address be the ship-to address?  What impact do the shipping terms have when we consider a late payment? Wouldn't it be nice if that type of information were kept somewhere, besides in the head of the person who has worked with the data the longest?  How mad would you be if I told you there has been a place for this data for years, but it just was no...

In my last blog, I highlighted five data principles that I have used in the past to create a solid data strategy.  The idea behind a principle is that it is flexible to adjust to changing needs but rigid enough to maintain order.   A good example of this is the golden rule, "do unto others as you would have them do to you".  When you apply this principle, you are free to do whatever you want, as long as it is something you want done to yourself.  It is a very flexible principle that can have serious consequences if broken.   One crucial principle that forms a solid foundation for your data initiatives is  naming standards . It might seem like a simple concept, but consistent naming conventions throughout your data environment can significantly impact efficiency, governance, and understanding for everyone involved.  With the right naming standards, looking into the data environment as a DBA, developer, or user, there will be no question about ...