Integrating the Dataverse

We have checked off all of our tasks on the road map.  The capabilities that we wanted to implement are up and running and our staff is fully imbedded in the new technologies.  Reports are going out the door so now sit back and enjoy all of the fruits of your hard work.  The reality of business is a totally different story.  Business objectives change, markets change, regulations change.  Change is the name of the game so how do we know if our data strategy was a success? Well how did we track against the road map? Did we stay in communication with the business and the IT department? Can we see a measurable improvement in services sold or products that went to market? A successful business data strategy is one that helps the organization achieve its goals. If those goals change it does not mean that you did not succeed. So what are the characteristics of success? Improved decision-making : A good data strategy should help the organization make better de...

  The  first part of our data strategy journey is picking the destination.  What business objectives are we trying to satisfy?  Once we know where we are trying to go we have to decide how we are going to get there.  Its time to chart out path!  So how do we do that?   Just as when you are trying to get to a physical location we will not only need something that we can use to help track our path but it has to have a clear and understandable path. We need something called a data strategy road map. This is going to be a document that outlines the steps an organization will take to achieve its data goals. It typically includes a timeline, milestones, and resources needed to implement the strategy. So how will this road map help us accomplish our data strategy? The following four outcomes will come from using a good road map. Align data goals with overall business goals . This ensures that the organization is using data in a way that...

We talked some about what a data strategy is and how it needs to line up with business outcomes. Now lets talk about the largest "soft" skill that you will need to successfully execute the strategy. Communication When you are executing on a data strategy you become the focal point of expectation and outcomes. At this point your job is going to be as much a facilitator as it is a technical leader. As a facilitator you will have to step up your communication game. Notice I did not say speaker. In this role you will spend more time listening than talking. The business will have certain goals that you have to accomplish. How do you understand what they want without asking probing questions. Just because you hear that the business wants to improve sales is not enough. Sales of what? An underperforming brand, a product that has a high margin, maybe something that is made at an "at risk" plant that the business is trying to save. You have to ask the probing ques...

As a data architect I am tasked all the time to move data or enable reporting or investigate new data sources. There are so many different ways to do these tasks that if I did not have an over reaching strategy my work would become disconnected quickly. Doing these tasks without standards or direction would leave a mess of technical debt in my wake. I would certainty not be an effective architect or leader. But what formulates that strategy? How do we use it to validate our work and make sure we are working towards it? That is what I will spend the next few days discussing with you. Where to start? The beginning: To have a strategy first we have to know what a strategy is. Google will tell you that a business data strategy is a long-term plan that defines how an organization will collect, store, manage, and use its data to achieve its business goals. It is a critical component of any organization's overall strategy, as it can help to improve decision-making, increase effi...

When I was contemplating how to end this week I realized that instead of diving into another technical topic I personally would get more benefit by looking back over the week.  It's always important to move down the road when you are working on goals, but if you do not stop to look behind you how can you appreciate where you have been?  For that reason, I plan to use Friday as a moment to reflect on where we came from. This last week was a week of change.  It was the first week I started blogging.  It was the first week I was truly serious about changing the direction I will go professionally.  This week I decided it was time to truly forge a path and lead.  It is time to make new patterns in how I think and act and hopefully in how I inspire all of you. Just like some of the topics this week circled around making decisions about what tools to use or even on what overall data strategy to pick, everything starts with creating a new pattern.  But new pat...

  An organizations data strategy lays out a path and direction for their future data state, but what is the best way to implement it?  There are many technology choices that i hope to cover in later blogs, but today we will review Azure Data Factory as one tool to help implement your strategy.   Azure Data Factory is a cloud-based data integration service that helps you to automate the movement and transformation of data. It can be used to connect to a variety of data sources, including on-premises data warehouses, cloud data lakes, and SaaS applications. Data Factory can contribute to your overall data strategy in different ways. How your architect the solution will make it lean towards a data fabric or data mesh. But in either case it can be a good choice for more than data movement. In fact Azure Data Factory is a powerful tool that can be used for a variety of purposes. Here are the top 10 common uses for Azure Data Factory: Data integration : Azure Data F...

ETL is a complicated game to play.  Sometimes you have to work around data quality issues, data structure issues, and even business processes that create data exceptions that could not be accounted for in the requirement gathering phase of your sprint.   With all of the unknowns that your data creates deciding on the right architecture for your ETL environment is important. The devil is in the details but at a high level you have just a few choices. You can centralize your ETL using a tool that creates a one stop shop for development and execution or you can decentralize and run you ETL where the data sits. The first method uses tools like Informatica or Datastage to consolidate metadata and transformation logic into one repository. Tools like this can orchestrate moving data from different types of sources to targets of any type and provide a great place for an administration team to monitor the processes. When you decentralize your ETL platform you spread scripts t...

We all know the basic definition of a data lakehouse: a unified system that combines the best of data lakes and data warehouses. It provides a centralized repository for all of your data, regardless of its format or structure. This makes it easy to store, manage, and analyze all of your data in one place. This sounds like a slam dunk. Lets just put everything in one place and let the magic happen. But is it? The reality of a system like this is under the covers the administration is tangled at best. Sure you can access files and parquet systems and columnar tables but all of those things live in different places and are managed different ways depending on the underlaying architecture. That is where Databricks found one of its sweet spots. By creating a common architecture and standardizing on parquet files Databricks started to streamline administration. Databricks makes it easy to build and manage data lakehouses. It provides a unified environment for data engineering, data scie...

  In today's data-driven world, organizations are increasingly turning to data fabrics and data meshes to manage their data. Both approaches offer advantages and disadvantages, so it's important to understand the differences between them before choosing the right one for your organization. What is a data fabric? A data fabric is a software architecture that integrates data from disparate sources into a single, unified view. This allows users to access and analyze data from across the organization without having to know where it is located or how it is structured. Data fabrics are often used to support real-time analytics and machine learning applications. What is a data mesh? A data mesh is a data management approach that distributes data ownership and responsibility across the organization. This means that each team or domain is responsible for managing its own data, including its governance, quality, and security. Data meshes are often used to support agile development and se...