Five reasons why few companies are data driven

 For years now, we've heard of the importance of becoming a "data driven" organization.  Being data-driven, we are told, means making decisions not based on gut instinct or what managers believe is true, but based on evidence, on hard data.  This, it would seem, should be simple.  After all, businesses have had robust IT systems and teams for years.  The advent of ERP in the 1990s and beyond created sweeping systems that automated many sectors of the business.  Even smaller companies that did not need the larger ERP systems like SAP could find solutions in Salesforce, Workday, Netsuite and others.

So, why is it that so many companies cannot make decisions based on their data?  If we've known about this need for decades, if we expound upon the importance of data and its use in decision-making, if robust systems exist to allow us to create, gather and use data effectively, why are so many companies no closer to becoming a data-driven organization.  There are several reasons:

1. The executive team does not like what the data tells them.  In many cases, data contradicts what management wants to do, so in those cases the data must be "wrong".  The data is inconvenient to current thinking or perspectives.
2. The executive team does not trust the data, and has good reason not to trust the data because it is out of date or incomplete.  The data is full or errors, untrustworthy, incomplete.
3. The executive team trusts the data and the data is accurate, but the data does not inform current or future decisions.  It is good data, just not the necessary data.
4. There is too much data and not enough information to inform executives.  The team is inundated with data and has too little information.
5. The executives have accurate and up to date data that is meaningful and easy to understand but they do not know how to interpret or act on the data.  The decision makers do not have the perspectives or tools to interpret the data and make decisions.

In some cases, all of the above is also true.

From these five reasons, there are three categories of issues:  problems with data itself, problems with policies about data, and problems with people.  The first two are the easiest to solve.

Problems with data

Every company has lots of data, often spreadsheets full of data.  And therein lies the problem:  there is no single source of truth, and data is highly compartmentalized and curated for each team or department.  When decisions need to be made at an enterprise level, but all data is managed and manipulated at a team or departmental level, obtaining a realistic view of the data or information a company has, and what that data means, is difficult because when aggregated, the data has passed through so many filters or lenses that it is often contradictory or meaningless, or so out of date as to be useless.

The problem is also part of the solution:  many managers and executives grew up in a time when IT managed all the data, and IT can be monolithic and slow to respond, full of data governance rules and policies.  So, many companies have gone "free range" with their software selections and data management.  Every department selects its chosen software, deploys how they care to, and does not worry about enterprise data or data management.  The problems with this approach will emerge within 6 to 12 months of the third or fourth departmental deployment, when executives request a single view of the business on one sheet, and no one can provide it.

Problems with policies

Face it, no one likes being told what to do, and no one likes policies, until those policies actually benefit them.  Managers and executives don't like data lexicon or nomenclature, or rules about data usage or data governance, because they impede speed and deployment and impose rules.  However, months or years later, they inevitably look back wistfully and wish they had had the coaching to implement data governance early, because imposing these rules and policies later in a company's life is difficult.  My advice:  start defining your data policies and rules early.  Keep your definitions consistent and clear, your calculations the same across teams and divisions.  Trying to impose data policies, data governance and data quality on a company years later is like trying to fit a grown adult into the clothes they wore in their youth.  What you need is a good suit of clothes that can be upgraded and tailored from youth through adulthood.

Problems with People

Even if your data is correct and your policies are strong and the data is timely, if managers or executives cannot interpret the data or cannot describe the data or information they need to run a business, all the other infrastructure work will fail.  There are plenty of people who are experts at a very narrow range of business metrics - sales people know sales metrics, financial people know financial metrics, but few of us get education or experience in managing metrics across a business - we are often simply too siloed.  But in a small but growing business, understanding all of the metrics is vital.  

As much as we'd like to believe that we educated humans are rational actors, we are attracted to data that reinforces our biases and reject data that calls into question our wants and goals.  A data driven organization needs a Spock-like character who can make decisions not based on desires or emotion but on pure logic, and most of us aren't able to do this reliably.  When you realize that NONE of us are logical or Spock-like, and we all have our own perspectives and biases, you can see how difficult decision making influenced by data becomes.

That's the data, what does it mean?

Finally, being able to understand the data is one thing, understanding what it means, what the data is telling you is another thing entirely.  Here, we can't simply wish our way to intelligent AI bots that will tell us which actions to take based on the information we are receiving.  Good managers and executives will need to combine experience, history, data and information, emerging trends and some gut instincts in order to evaluate data and make determinative decisions and actions.  Most of the existing management cadre in business today are familiar with backward looking data (reporting) and historical trends and precedents but are not familiar with the existing pace of change, or the amount of data and the requirements for interpretation of the data, much less trying to determine future actions based on the volume of data they are receiving.  

Solving for the three big data issues

There are two approaches to solving for these issues - bottom up or top down.  My recommendation is top-down.

The bottom-up strategy will focus first on getting the policies correct, then correcting the systems, how the systems are integrated and the validation of the data.  It's not really all that helpful to start improving the data and systems unless there are policies that will sustain the changes you make.  Once the data, systems and policies are in place, then you can ensure your leadership team can interpret and use the data.

The top-down strategy requires that the executive team and senior users and managers of the data can describe what they need in order to make better decisions, and how they need the data presented to them.  Then, they can create or reinforce policies around data strategy, data quality and data governance, which will influence how systems are implemented and how data is exchanged and aggregated.

Understand that all of these things need to change - the systems and their integration to get better, faster data; the policies to ensure that they data is useful and meaningful; and the users, to ensure that they are educated in how to interpret the data and how to make decisions based on data.  This is yet another holistic change that needs to occur.  Changing policies without changing systems, or educating the executive team without improving the quality of the data leaves the solution incomplete.