Is the EDW Dead?

“The King is dead. Long live the King” is a traditional proclamation made following the accession of a new monarch. But what happens when the original king was highly revered and his successor brings foreign ways to the throne?

Bloor Research Raises the Issue

Philip Howard, noted IT researcher at Bloor Research, recently published an IT-Director article entitled The EDW is dead where he declares the well regarded monarch of data integration deceased. In this article he identifies two major flaws in the traditional enterprise data warehouse approach that support his controversial stand.

Flaw One – Multiple Data Types
The first flaw in the enterprise data warehouse concept relates to how data is stored and processed.  Not all data is structured.  Unstructured (for example text and images) and streaming (for example web clicks and sensor data) are also critical and growing elements in the enterprise data landscape.  According to Howard, “It is just about possible to conceive of a platform that supports relational, Hadoop-like and streaming data at some point in the future but it’s not going to happen soon-if it ever does.  So, certainly for the time being, there is no prospect of an EDW supporting all of these different types of query processing.”

Flaw Two – Long Time to Solution, High Operating Costs
The second flaw relates to the narrower field of structured data which in and of itself presents significant challenges at enterprise scale.  Here Howard states, “I don’t think the concept of a single monolithic EDW was ever the right one.  It was too time consuming and expensive to set up and run. But even if you think that it was conceptually the right approach it has been overtaken by practical considerations: in particular, data marts and application-specific appliances have proliferated throughout large enterprises and there is no way that that is going to change.  Further, data virtualisation and federation are now mature technologies that allow federated query access across data marts and warehouses so there is no longer any incentive to try and centralise everything.”

Forrester Report on Data Virtualization Raises Similar EDW Concerns

Data Virtualization Reaches Critical Mass: Technology Advancements, New Patterns, And Customer Successes Make This Enterprise Technology Both A Short- And Long-Term Solution by Forrester Research raises similar EDW concerns including:

• Shortcomings in ETL-based data integration approaches cause degradation of data quality and delayed information.
• Traditional data integration approaches that consolidate multiple databases into a single warehouse are too slow, expensive and risky in today’s dynamic business environment.
• Cloud-based data sources add new integration complexity
• Business cannot wait for long running initiatives such as MDM and will take the integration problem into their own hands if IT cannot provide better short term alternatives.

Will Data Virtualization Be the New King?
The Forrester report predicts accelerated adoption over the next 18-36 months. Six technology advancements are accelerating data virtualization’s ascension to the data management throne.  These include:

• Cost-based query optimization performance improvements that open the door for additional use cases and wider adoption;
• Distributed caching which sets the framework for global deployments;
• Improved discovery tools that simplify understanding of data and accelerate new development;
• Data masking and other security enhancements to better govern data delivery;
• Third party technology integration such as CDC and ESB to extend data virtualization tooling; and
• Big data integration to create new opportunities for analysis and insight.

Is the Kingdom Large Enough for Two Kings?

One of the most important services that analyst firms such as Bloor and Forrester provide to their enterprise IT clients is the prediction of future trends. But enterprises move more slowly and large EDW investments have and will continue to pay off for the foreseeable future.

Given this reality, maybe we should anticipate a long period of shared rule where data virtualization and data warehousing team to provide a benevolent reign.