Enterprise adoption of data virtualization continues to accelerate driven by organization demands for greater business agility and lower IT costs.

My professional services team has deployed Composite data virtualization offerings in support of hundreds of implementations. From this work a number of best practices have emerged. 

In compiling the ten case studies described in the recently published Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility co-author Judith R. Davis and Robert Eve also had a chance to closely examine a number of data virtualization best practices, synthesizing them in to  five best practices areas. 

In this series of three blog posts, I will pass along the lessons that these ten organizations learned along the way, as they are a valuable to help other organizations avoid common pitfalls and realize the benefits of data virtualization as quickly as possible.

1. Centralize responsibility for implementing data virtualization

Several organizations stressed the need to centralize the initial design, development and deployment responsibility for data virtualization into a focused data virtualization team. The key benefit here is the ability to advance the effort quickly and to take on the bigger concepts, such as defining common canonicals and implementing an intelligent storage component to speed development, reduce time to solution and deliver a more powerful and complete data virtualization environment.

Relative to data virtualization development, Northern Trust added that centralization provides economies of scale and enable the company to accelerate up the best practices learning curve.

Agreement on a common data model was also identified with the centralization best practice.  Several of the users commented that this will ensure consistent, high quality data, make business users more confident in the data and make IT staff more agile and productive.

According to Qualcomm, establishing a clearly defined, centralized data virtualization development and management environment was important. Key issues are who is responsible for the shared infrastructure and for shared services.

Both the Global 50 Energy Company and Qualcomm described the importance of management support for the data virtualization effort. This may be easier if responsibility for data virtualization is centralized.

Performance! Performance! Performance!

Composite 6.1 advances our industry-leading query optimization algorithms and techniques and with improved caching performance, expanded caching targets, data ship join for Teradata, and Hadoop MapReduce extensions. 

• Improved Cache Loading Performance: Improves the speed of cache loads and refreshes using native cache loading and parallel cache loading techniques. This enables organizations to cache more data, more often and flexibly improve the timeliness of information delivery.

• Expanded Caching Targets: Greater caching target flexibility enables developers and administrators to optimize both query performance and operating costs. Composite 6.1 adds SQL Server 2008 and Netezza 6.0 as caching targets to a list already including IBM DB2 v8; Microsoft SQL Server 2000, 2005; MySQL 4.1, 5.0; Netezza 5.0; Oracle 9i, 10g, 11g; Sybase 12.5; Teradata V12.

• Hadoop MapReduce Extensions: Enables MapReduce programs and developers to easily query Composite as a data source, on-demand with high performance. Allows enterprises to extend MapReduce analyses beyond Hadoop stores to include diverse enterprise data managed by Composite.

Higher Productivity and Greater Ease of Use

Composite 6.1 also provides significant improvements to our data services development environment with an enhanced data services editor and new publishing options for Representational State Transfer (REST) and Open Data Protocol (OData) data services. Our all-in-one relational and data services development environment simplifies development of data services and/or mixed relational and data service objects enabling high productivity development of data services.  This accelerates time-to-solution for new data services and expands the development community to a wider set of users.

Leading Data Virtualization Analyst Likes What He Sees

“Many of today’s BI architectures are too complex and inflexible and are encumbered with too many data layers resulting in data that is often not up-to-date,” saidRickF. van der Lans, managing director of R20/Consultancy B.V., a leading authority on data virtualization and the author of the upcoming book “Data Virtualization in Business Intelligence Architectures: Revolutionizing Data Integration for Data Warehouses.  “Data virtualization platforms—such as Composite 6.1—enable more timely and cost-effective integration with disparate operational systems enabling more effective analytics and reporting.” 

Available Now

Composite 6.1 is available immediately and is a free upgrade for Composite customers with current support agreements.

RainStor is Really Versatile

RainStor is used as a primary database to ingest, retain and manage high volumes of multi-structured data such as logs, call data, financial service transactions, smart meter data and other forms of automated machine-generated data. In addition RainStor runs natively on Hadoop providing query and analytics through both SQL and MapReduce. RainStor can also be used as a secondary database for holding historical data from retired applications as well as enterprise data warehouses such as Teradata.

A Powerful Combination

In all of these scenarios, Composite allows the data retained within RainStor to be easily combined with other enterprise data scattered throughout the organization.

“The combination of the two technologies provides a cost effective and high performance way to retain and access data on demand without the need to create multiple copies and deal with the complexities of moving data between environments,” said Claudia Imhoff, president of Intelligent Solutions, Inc., a leading authority on BI and data warehousing.

“RainStor allows enterprises to retain Big Data for compliance and historical business analysis,” said RainStor VP Product Management Ramon Chen. “We are often asked for the best way to combine the data stored in RainStor with information from its originating source or across other related content. We are delighted to partner with Composite Software to offer this capability as they represent the gold standard for data virtualization.”

According to Peter Tran, Composite Software vice president of product marketing, “To maintain our success as the market’s leading enterprise data virtualization solution, we must continuously add the latest, most innovative data sources. RainStor’s compelling economics for Big Data management and our joint philosophy of simplicity in enterprise data access makes our products a natural fit.”

Data virtualization middleware platforms provide the functionality described above within integrated offerings that support the full software development life cycle, high-performance run-time execution and reliable, 24x7x365 operation.

These include:

• Integrated Development Environment
• Data Virtualization Server Environment
• Management Environment

In the Part 1 of this blog, I described the key concepts behind data virtualization. In Part 2, I will drill down into data virtualization middleware’s three key environments:

Integrated Development Environment

Data virtualization technology includes an integrated development environment (IDE) that can be used by a range of people, from business analysts to application developers, to define and implement the appropriate view and data service objects.

The foundation of these views and services is an underlying logical data model that is, in turn, based on either a tabular or hierarchical schema. Data quality requirements, such as standards conformance, enrichment, augmentation, validation and masking; and security controls (e.g., authentication and authorization) can also be also implemented within these object definitions.

The IDE includes profiling-like introspection and relationship discovery capabilities designed to simplify each developer’s understanding of existing data sources and jump-start the modeling process.

To limit the coding required and save development time, drag and- drop modeling techniques and a rich set of pre-built, any-to-any transformations automatically generate view or data service objects. Multiple languages (SQL, XQuery, Java, etc.) can extend these capabilities to address more advanced data virtualization needs.

Standard source and consumer APIs, based on ODBC, JDBC, SOAP, REST, etc., simplify source data access and consumer delivery development activities.

Integrated data governance, including lineage and where used, metadata asset management and versioning provide needed controls.

Data Virtualization Server Environment

In data virtualization, run-time activities are typically triggered by queries, or requests for data, from a consuming application. The data virtualization server is the component that executes these queries.

The query engine within the server, which is specifically designed to process federated queries across multiple sources in a wide-area network, optimizes and executes queries across one or more data sources as defined by the view or data service.

Cost- and rule-based optimizers automatically calculate the best query plan for each individual query from a wide variety of supported join techniques. Parallel processing, predicate push-down, scan multiplexing and constraint propagation techniques optimize database and network resources.

The data virtualization server also does the following:

• Transforms query results sets to ensure that the data is complete, high quality and consumable by the user.
• Executes authentication and authorization security functions to protect data from improper use.
• Caches appropriate data sets to enhance both performance and availability.

To complete the query, the server delivers the results directly to the consuming application and logs all activities.

Management Environment

Data virtualization servers are configured for development, testing, staging, production, back-up

and failover operations. To manage this topology, meet service-level agreements (SLAs) and ensure reliable 24x7x365 operations, the data virtualization platform also includes a complete set of integrated management tools.

These integrated tools support all the activities required to set up the data virtualization middleware and users, including provisioning the software, granting access to sources, integrating with LDAP and other security tools, etc.

System management tools manage server sessions and resources.

Monitoring tools log activities, monitor memory and CPU usage, as well as display key health indicators in dashboards.

Optional clustering tools improve workload sharing and synchronization across servers.

Data virtualization is a data integration approach and technology used by innovative organizations to achieve greater business agility and reduce costs.

Data virtualization technology is a form of middleware that leverages high-performance software and an advanced computing architecture to integrate and deliver to both internal and external consumers data from multiple, disparate sources in a loosely coupled, logically-federated manner.

In this blog, I will describe the key concepts behind data virtualization.  In Part 2, I will drill down into data virtualization middleware’s three key environments.

Key Concepts Behind Data Virtualization

Views and Services – The primary objects created and used in data virtualization are views and data services. These objects model and encapsulate the logic necessary to access, federate, transform and abstract source data and deliver the data to consumers.

Virtual Layer – By implementing a virtual data integration layer between data consumers and existing data sources, the organization avoids the need for physical data consolidation and replicated data storage. Standard APIs and an open architecture simplify the consumer-to-middleware-to-data source connections.

On-demand Consumption – Most front-end business applications, including BI, analytics and transaction systems, can access data through the data virtualization layer. Consumption is on demand from the original data sources, including transaction systems, operational data stores, data warehouses and marts, big data, external data sources and more.

Optimization – High performance query algorithms and other optimization techniques ensure timely, up-to-the-minute data delivery.

Middleware – Data virtualization middleware platforms provide the functionality described above within integrated offerings that support the full software development life cycle, high-performance run-time execution and reliable, 24x7x365 operation. These include

• Integrated Development Environment
• Data Virtualization Server Environment
• Management Environment

In Part 2, I will describe each of these more fully.

How Data Virtualization Paid Off In 2011

Enterprise adoption of data virtualization accelerated in 2011 propelled by organizations growing need for greater business agility, lower costs and better performance.

I wrote ten Data Virtualization Leadership Series blogs on business value throughout 2011 as I tried to communicate a number of these successes.

In these blogs I highlighted the significant business value derived at real enterprises including NYSE Euronext, Pfizer, Putnam, Qualcomm, Telus and more. 

Further, in What Would You Do With A Million Dollars? , I summarized the typical business case value propositions we see across our customer base include some combination of:

• Sales/Productivity increase in the business
• Risk reduction (either company-wide, or on a project basis), Staff productivity increase (IT and/or business staff)
• Technology cost reduction (especially storage and software costs),
• Time reduction (which translates especially into increased time-to-market… thus gaining the advantages sooner).

Others Showcase Data Virtualization’s Value

But I wasn’t the only one out there describing this value.

Data virtualization benefits were fully described in a series of Virtualization Magazine articles by my colleague Robert Eve:

• The Agile Business – Why Data Virtualization Is Needed
• How Data Virtualization Delivers Business Agility – Part 1
• How Data Virtualization Improves Business Agility – Part 2
• How Data Virtualization Improves Business Agility – Part 3

And data virtualization’s value is clearly evident in the ten case studies described in Judith R. Davis and Robert’s recently published Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility.

Data Virtualization Value In 2012

As I prepare for 2012, I think back to how I described my mission in Data Virtualization Can Be Rewarding. At that time, I said “What is the best job in the enterprise software industry?  Imagine a job where every day you are:

• Exposed to a diverse set of industries and challenges.
• Discovering the value drivers and putting together business cases to move your projects forward faster.
• Seeing the actual results from successful data virtualization implementations … how incredibly rewarding!

That is my job every day! I have the pleasure of working with Composite Software prospects and customers to help them understand,measure, realize and communicate the business and IT value of their data virtualization investments.”

One look at my January and February travel schedule show lots of data virtualization value assessments already underway. 

The data virtualization value bottom line for 2012 will be huge.

I can’t wait!

This week, Forrester Research, Inc., delivered their first Wave report on Data Virtualization.

The January 2012 report: “The Forrester Wave™: Data Virtualization, Q1 2012” provided a number of insights about the data virtualization market and the key vendors in the market, including leader Composite Software. 

Successor to Information-as-a-Service 

Previously Forrester has used the Information-as-a-Service to describe this market segment, publishing waves in 2008 and 2010 under this terminology. The report is available from Forrester at www.forrester.com.

Data Virtualization A Large and Growing Market

According to the report, authored by Forrester’s Principal Analyst Noel Yuhanna and Mike Gilpin, vice president and research director, “the data virtualization market continues to grow strongly, with more mature solutions supporting large deployments. Forrester predicts that total software revenues (licenses, maintenance and services) in data virtualization will grow to $8 billion by 2014”. 

“In the early years of this market, most data virtualization was focused primarily on the financial services, telecom, and government sectors,” according to the Forrester report. “However, in the past 24 months, we have seen a significant increase in adoption in the healthcare, insurance, retail, manufacturing, eCommerce, and media/entertainment sectors. Regardless of industry, all firms can benefit from data virtualization, especially when integrating larger numbers of data sources, delivering real time data, and integrating with a wide range of data types.”

Successful Customers Use of Composite Is a Key Evaluation Factor

Composite Software was among the select companies that Forrester invited to participate in its data virtualization research study. The report concludes that “customers like Composite Software’s easy-to-use solution, tight integration to support most uses cases, and its lower cost, compared with larger software providers.” In this evaluation, Composite Software received top scores for data services performance and data services integration. 

Composite in Data Virtualization Selection Short Lists

“Composite Software has done well over the years, improving its solution dramatically, adding more advanced data virtualization capabilities, and delivering innovative products,” the report states. “Among pure-play data virtualization vendors, Composite Software is often shortlisted by customers largely because of its visibility and strong product offering. Composite Software supports some of the most complex data virtualization deployments in existence, in part because it has been active in this market as long as or longer than any other player.”  

If you want to read more about customer use cases, Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility, provides details on how ten enterprises successfully adopted Composite’s data virtualization offerings. Or if you are more video oriented, check out the many Composite data virtualization use cases on YouTube http://www.youtube.com/user/Compositesw.

At Composite Customers Are Number One

This is Composite Software’s fourth opportunity to present at the BBBT.  Following the pattern established with Compassion International in 2010 and NYSE Euronext in 2011, in 2012 Composite will again include a customer speaker in our session.  This year it will be Pfizer.

BI Agility at Pfizer

Pfizer’s Michael Linhares, Ph.D. and Research Fellow and leader of the Business Operations Group within Pharmaceutical Sciences, will present the Pfizer case study. 

Linhares was instrumental in development of Pfizer’s Research Information Factory (RIF), global reference data implementation, introduction of data federation and other enterprise data integration projects. 

“We started our data virtualization journey five years ago and it has made all the difference in the world in enabling our innovative enterprise data integration and agile business intelligence projects,” said Linhares. “With a traditional data warehouse approach, an organization takes source data, moves everything to a staging area, then moves a portion of it again into the operational data store, and then moves data a third time to a data store.”

Data Virtualization Pays Off

Linhares explained. “In the first stage of our federated implementation, instead of moving the data, we built a representation of the data in the Composite Data Virtualization Platform. If a source is ever changed by the owner, which often happens, we can update the representation in the information abstraction layer quickly. This allows control of all downstream data in one location.”

Data Abstraction Adds Flexibility

Linhares added that with data virtualization, “we don’t have to be concerned about the technical solutions chosen by our data providers. Data virtualization enables a level of abstraction for us that separates us from the source systems and gives us automatic access to all of them.  Data virtualization has enabled us to combine isolated silos of data with a data delivery platform that integrates different types and sources of data into a comprehensive package of value-added information.”

Wayne Eckerson, veteran consultant and industry analyst in analytics, business intelligence, performance management and data warehousing at the BeyeNETWORK  just published a new eBook entitled “Federating Architectures for BI Professionals.” Composite Software sponsored it’s publication.

It is a Federated World

In the eBook, which can be downloaded here, Wayne writes that BI professionals currently operate in a federated world where critical data and analytical logic is spread across multiple databases and applications in various parts of an organization. A federated architecture knits these disparate environments together virtually rather than physically.

According to Wayne, It is neither centralized nor decentralized, but a hybrid of the two, maximizing the benefits of both options, while minimizing their downsides. Moreover, a federated architecture is fluid, changing shape as an organization reinvents itself to better respond to new market realities.

Helping BI Managers and Architects

“Contrary to popular opinion, a BI architecture is a dynamic environment, not a blueprint written in stone,” Eckerson said. ”BI managers must define an architecture based on prevailing corporate strategies and then be ready to deviate from the plan when the business changes due to an unanticipated circumstance, such as a merger, acquisition, or new CEO.”

The BeyeNETWORK eBook describes how to create a federated BI architecture that aligns with organizational imperatives. 

Data Virtualization is Critical

Wayne believes that data virtualization is a missing layer in most BI architectures.  In the ebook he notes that “thanks to the popularity of data center virtualization, many organizations are exploring the possibility of virtualizing their data as well to help manage these dynamic environments.”

“More importantly, advances in network speeds, CPU performance, and available memory have significantly increased the performance and scalability of data virtualization tools, expanding the range of applications they can support.  Moreover, data virtualization vendors continue to enhance their query optimizers to handle more complex queries and larger data volumes.”

Making effective business decisions requires knowledge and insight that can only be developed from access to and analysis of complete, high-quality actionable information.  Data virtualization enables the organization to deliver this information in several ways.

Delivering Complete Information

Understanding the complete picture is the first step in any decision process. Large enterprises today have thousands of data sources that span multiple transaction systems of record, complementary applications, consolidated data stores, external data sources and more. Each source is a silo with its own unique metadata and data model, data access toolset, and underlying architecture.

The challenge is integrate data across these traditional silos in order to provide the business user with a single, complete and high-level view of whatever information is needed for analysis and decision making.

Taking this concept to the enterprise level, data virtualization has the ability to provide an organization with a unified view of information across the entire business.

One traditional solution is to consolidate all of the data in a unified, enterprise data warehouse. However, this does not always prove feasible in practice for a number of reasons, including the ongoing proliferation of new data sources and types that must be incorporated into the warehouse.

As an alternative, data virtualization offers virtual data federation functions that enable an organization to integrate its extensive range of internal and external data sources without moving any data. Accessing the data in a new source, for example, simply requires establishing a single connection – from the data virtualization layer to the source – and the creation of virtual views and data services to access, transform, abstract and represent the source data in an appropriate format for consuming applications.

Delivering High-Quality Information

Ensuring that the information guiding business decisions is high quality and fit-for-purpose is a second major business decision agility requirement.

It is rare that as-is, raw data in original source systems is an exact match for the consuming application and business user. At a minimum, some degree of format and syntax transformation is required to bridge the gap between source and consumer data models and technologies. In many cases, additional validation and standards-conformance processing is also needed to improve the integrity and consistency of the data delivered to consumers.

Data virtualization supports multiple techniques to ensure delivery of high-quality information. For example, when providing up-to-the-minute, institution-wide views of equity, option, futures, derivative and debt positions for risk managers in the financial services industry, data virtualization transforms the data as structured in the various trading platform sources into a consistent form and format for consumption by the risk management applications.

Delivering Actionable Information

Finally, the information decision makers require must be actionable. Time-to-action is an additive function that combines time from event-to-insight, insight-to-decision, decision-to-implementation and implementation-to- results. 

Yesterday’s data, summarized in the warehouse, is not sufficient when having the most current information is the first step in a time-to-action path. For example, understanding the current location and availability of maintenance staff and repair gear is a critical first step in understanding how to respond to an equipment failure in process industries.

Data virtualization provides high performance query engines and flexible caching to query and deliver source data in near-real time whenever it is requested. This ensures decisions are made based on most up-to-date information available when appropriate.