401 Percent: An Introduction

A couple of people have asked me "why a blog titled '401 Percent'?". So, here's the story:

Cast your mind back to 1996, the UK was struck with an outbreak of bovine spongiform encephalopathy, gangsta rappar Tupak Shukur was gunned down in Las Vegas, and International Data Corp (IDC) published the largest study of data warehousing return on investment (ROI). Bells ringing? Tupak? No?..I'll move on..

IDC presented a 'Foundations of Wisdom' study, which revealed, on average, a 401 percent three-year return on investment for 62 companies possessing active data warehousing deployments. Vendors have used this 401 percent ROI figure as a carrot to wave in front of hungry investors for a number of years, and, given my research interests, I felt it a fitting title for this new blog.

While we're looking at the IDC study, and because I have editorial control over this blog, it may be worth delving a little deeper into IDC's Foundations of Wisdom. 401 percent, pretty good right? Why wouldn't we invest? Well, 401 percent isn't the whole story, in fact, far from it. Andrew Lang, once said (or words to this effect), that statistics are often used like a drunken man uses lamp-posts, for support, rather than for illumination. Rather than bore you with my very limited knowledge of Scottish literature, let's ponder the mysterious '401 percent'. One of the most interesting parts of the IDC study was the quite significant range of ROI's that were found in the report. In fact, ROI's disturbingly as low as -1,857 percent! (and also as high as 16,000%, to be fair). These figures already place the 401 percent in a different light. At best, the record for data warehousing success is patchy. It's not hard with a quick Google search to find failure rates for data warehousing projects quoted from 10 to 90 percent! Figures such as these put the DW decision-maker in a precarious situation, the BI industry is predominately industry-driven, and largely self-evaluating. Those with most to gain out of a 'research' finding, are usually those in control of reporting their results. Those of us absorbing that information should be wary. There are very few completely independent research analyses in the BI field (three cheers for Nigel Pendse's wonderful OLAP report), and we should take what we read with a grain of salt. Have you ever read a vendor white paper that does anything but preach positive outcomes and significant returns on investment? It's understandable their reluctance to disseminate stories of failure and underperforming implementations. Scratch a little below the surface of many research reports and you often find those organizations that rank the highest, whatever the focus, are often clients of the analyst firm. Even organizations such as the data warehousing institute (TDWI), a leading data warehouse research body, admits that almost 70% of a recent surveys participants were IT vendors or consultants - those with the most to benefit from their findings (rather than stakeholder organizations making up the majority).

If you look at the nature of benefits from DW and BI, it's a little easier to understand why it's so difficult to arrive at reliable ROI figures. Much of the benefits to be gained from these systems are in the form of what I would call "BI-enabled business change". Much of the success of BI goes far beyond simple operational cost savings, these are relatively easy to quantify, but come through business transformation and improvement through BI technologies. These benefits are much more difficult to quantify, are more numerous, and intangible. Traditional accounting methods do not perform well when applied to strategic, soft benefits. Much of the benefits from DW and BI are filtered through other areas of the business. One of the biggest challenges in obtaining consensus for BI and DW ROI is concerned with the accountabilities associated with using qualitative measures of success. In organizations where quantitative measures play the biggest role in determining business performance, justifying DW or BI with ROI will be challenging.

I'm not saying that industry research findings are useless, but what I am asking is for us to take what we read in context. Look at who is publishing the research findings, what do they have to gain? How was the research conducted? What is the researcher/analysts' experience? Is there a sponsoring organization? Armed with a little more knowledge, and perhaps a tiny bit of skepticism, we give ourselves more opportunity to gain the most out of what we read; "are these findings useful for me? Or is the research methodology a little contrived?" Happy reading.