Big Data is doing great things in fighting cancer, fighting crime, winning (and losing) elections but in the corporate world, not so much. Why is this?

We have designed and built a good number of Big Data projects in our careers.  Some were hugely successful, some not. What happened? We have a number of reasons:

  • Ego
  • Existing relationships
  • Vanity projects
  • Mergers and acquisitions

Some years ago, we offered one of our products to a company to license it to access their big data systems. The product was ready to go, proven to do the job. But no, a key player at this company had a budget to build her own. Millions of dollars and 5 years later their product is failing to gain traction among their users. They have recently decided to allow third party products to work with their data. Ego cost this company millions.

At another, now former client, we had a product shoot out. Our big data system was up and running. We scored 6.4 out of 10, not great but the nearest competitor scored 4.9. This competitor’s product did not even exist at the time. We lost. We found out a year later that a key executive had a close relationship with the competitor. He is no longer with the former client.

IT management sees the potential of Big Data but does not know how to calculate a return on investment on Big Data projects. By their nature they are expensive. Often, a Big Data project is a vanity project for IT management. The ultimate goal and ROI are not part of the equation. “Build it and they probably will not come”. Big Data systems must be designed and driven by end users.

The biggest reason for failure of Big Data systems is mergers and acquisitions. Because it is hard to put an ROI on Big Data, new owners of a company see it as a top candidate for saving IT expense. This is especially true when a Private Equity company takes over the business. The onus is on IT management to have a clear vision.  Vanity projects are easy to spot!

By: Michael Dickenson and Steve Hultman