It’s The Data Quality

April 2, 2014   //   Business Intelligence, , , , ,

Success in business intelligence usually boils down to doing the simple things well. One area we never lose sight of at SWC is the focus and passion for “Data Quality” in all of our projects. Quality is not a feature you add in at the end of the project or mention in a design artifact. Quality has to be pervasive through the entire project and exist even after delivery has been wrapped up.

Embedding Data Quality

A few years ago the business intelligence team at SWC was taking stock of our approach to BI and looking for an opportunity to inject SWC IP into our projects. We were looking to do something that would separate us from our competitors and leave a long lasting impression on our customer base. We came to the realization that “Data Quality” on our BI projects was a substantial manual effort and seemed to be re-invented each time we did a new project. And so the advent of “Data Sentinel” came to fruition. The “Data Sentinel” or DS tool/framework was our answer to ensuring a repeatable process. The tool would ensure quality on all of our projects, be easy to use/configure and be extensible for those clients that desired to enhance the tool at their discretion. The tool was an immediate success! All of our customer’s using Data Sentinel continue to use, embrace and enhance its adoption to this day. We even have customers using Data Sentinel to monitor business KPIs for organizational health.

Keeping It Simple

One of core requirements when building Data Sentinel was that it had to be easy to configure/install. We envisioned that business analysts, data analysts, test engineers and developers of all skill levels would use the tool. Each Data Quality configuration has the following components:

  • Regression tests need to live in a container or group so that they can be better categorized. Tests that are mission critical go in the mission critical test group and tests that are important to a particular division or group can defined in groups along the organizational structure. Any grouping structure needed can be defined.
  • Within a group there will be one or more tests. Typically on a BI project, you will have a measure or KPI that is important to the organization and ensuring the quality of that measure means checking data at several junctures in the process. For example, if we had a measure for “Order Amount” we would need to check the aggregate at the semantic layer/cube, data warehouse, staging database and source system.
  • Within a given test there will be one or more steps. Building on the example from #2, the test steps would be to get counts from each data retention point and require that the count for all the steps be equal. If the source system says MTD “Order Amount” is $7 million, then each of the data points after the source system also have to be $7 million as well. As you can imagine, if there is a breakdown early in the process, everything downstream is going to be inaccurate.

Making Quality Pervasive

We use Data Sentinel on our BI projects along the entire promotion path starting at the local development machine and from there depending on our customer’s SDLC we will have additional checkpoints along the way. A typical BI project assumes: Local Machine, Development Server, Smoke Test Server, QA, Stage and Production. The automated regression testing is always ongoing and because it is automated, we also tie it into the software build process so that we know immediately if a software build has introduced data quality problems. So in closing, by making data quality a major focus to our practice area and building tools that automate this time consuming test we get to spend more of our time solving the harder problems and let our framework ensure the quality of the solution.

To learn more about SWC’s business intelligence solutions, please join us for our next informative Business Intelligence event or follow Chad Dotzenrod on LinkedIn.

Additional Business Intelligence Posts

If you enjoyed this post from Chad, you may be interested in reading a few of his past posts on Business Intelligence:
Do Tableau And MDS Make Strange Bedfellows?
If They Only Had Tableau
My Search For The Business Intelligence Chupacabra
Have We Forgotten How to Fix Problems?
Technology Meetups Make A Difference
What’s the R in ROI (for BI)?
Microsoft BI: Catching the Deadliest Dashboard
Tech Smart vs. Business Smart: Which Hat Does Your IT Consultant Wear?

Recommended Business Intelligence Posts

You might also be interested in reading some additional posts from SWC’s Business Intelligence experts:
How to Fast Track Business Intelligence
OAuth 2.0 – Google API Business Intelligence Implementation
What’s New with MDS In SQL Server 2012?
Can’t afford BI? Try the BI Analytics Tools in Everyday Software
How to Break Business Intelligence Users’ Excel Addiction
An Agile Approach to Business Intelligence
No Biggie: Data As A No Brainer
Marry your database. Date your dashboard
Ask SWC: What is the New R Feature for Tableau 8.1?
What’s So Great About Tableau?