Why Make a Change?

Shawn Rogers of Enterprise Management Associates, in his April 16, 2012, EMA Blog Embracing a Hybrid Data Ecosystem succinctly addressed the need to take a more hybrid approach to data management.

“The EDW has served us well and created a foundation for today’s sophisticated analytics. A paradigm shift driven by a maturing user community, new technology, economics and valuable data types is moving us towards a hybrid data ecosystem that strives to match the workload and the data with the best possible platform.

Many of us have spent years defending EDW best practices and work process that have taxed our infrastructures to the breaking point. It’s time for us to embrace a wider view of our data landscape and strategically integrate these new solutions.”

Drilling Down

Shawn drills down to help us understand each of the drivers in more detail:

• Maturing User Community – over the past decade the demands of simple reporting have morphed into highly complex analytics. The consumer of business intelligence has changed dramatically from a few highly technical users within a company to democratized culture where more and more employees are gaining access to information and driving the business with it.
• New Technology – Technology advancements are driving the adoption of Hybrid Data Ecosystems. Analytic platforms were the earliest to exist alongside the EDW providing a better platform for analytic analysis. Cloud platforms and Big Data frameworks are the latest to match new workloads and data on the best possible platforms.
• Economics – the overall cost of analytics plays a critical role in the adoption of a Hybrid Data Ecosystem. The lower capital costs associated with Big Data, cloud and analytic platforms have contributed to the enterprise adopting them.
• Valuable Data Types – Until recently the enterprise has primarily focused on structured information best leveraged by SQL and stored in relational databases. The task of analyzing high volume, high velocity multi-structured information has often been too complex or expensive for most companies to address. New technologies have made it possible for us to incorporate new and highly valuable data (social, machine, sensor data) into our analytic processes providing greater real-time insights and predictive outcomes.

What is the Strategy Impact?

I believe the key strategic impact will be a change in approach. No longer will you assume that the EDW is the default solution to each new data management challenge.

Instead you will need to think first about SLAs and workloads, before deciding how to implement the solution.

Then you can select from fit-for-purpose analytic appliances, new distributed data stores such as Hadoop, data virtualization, data marts or traditional EDWs in a mix and match mode as appropriate.

Data Virtualization and Business Agility

In his talk, “Data Virtualization: Going Beyond Traditional Data Integration To Achieve Business Agility,” David addressed how the rise of big data, mobility and cloud computing are breaking traditional data integration paradigms using research from Forrester, Gartner, IBM and IDC. He then described data virtualization, leveraging Forrester’s definition below and top ten characteristics of data virtualization.

“Data virtualization delivers a simplified, integrated view of trusted data about specific business entities, such as information about customers, products, or sales. Information is delivered in real time or near real time as needed by the consuming applications, processes, or business users.”¹

Proven Path to Data Virtualization Success

Next he showed how organizations such as Comcast, NYSE and Pfizer are using data virtualization to successfully provide their business users with one virtual place to go and a business view of the data in spite of these new business and technology challenges.

Practical Advice for Getting Started

David closed his formal presentation with advice on how others can succeed including:

• Use a proven data virtualization vendor (based on successful customer implementations)
• Seek out data virtualization domain experts (Composite wrote the book on data virtualization)
• Self-fund your data virtualization investment (using data virtualization’s cost savings and rapid payback)

Q&A

At the end of the presentation David answered multiple questions from the audience; addressing a range of topics including how data virtualization works with big data, when to consolidate versus when to virtualize, how query optimization works and more.

If you have any questions, just respond to this blog below. David and I will be sure to address them.

1. Source: Forrester Webinar – Your Enterprise Data Virtualization Strategy 2012 February 21, 2012

However, judging by the buzz, one might have thought it was a Data Virtualization conference, wrongly named.  Here is what I heard.

Rick van der Lans Keynote

Rick van der Lans, noted analyst, filled the room for his The Impact of Data Virtualization on Data Governance and Information Management presentation. He started by reminding everyone that virtualization is not a new concept in the world of IT.  Memory, storage and network virtualization have been around for some time. 

Then he described the essence of data virtualization as the decoupling of the applications from their data sources, leading to more flexible architectures. But data virtualization goes beyond technology; he also described how it can improve information management, data governance and MDM.

If you want a better sense of Rick’s message and presentation style, you can see him in the following three short videos”

• Thoughts From Rick van der Lans of R20 on Data Virtualization
• Composite Chalk Talk – Increasing BI Agility

Jill Dyche Keynote

DataFlux’s Vice President of Thought Leadership and noted IT expert Jill Dyche also addressed data virtualization in her keynote address Data Governance: What Your CEO Needs to Know.

To paraphrase, “folks, I don’t know how many you of you attended Rick’s presentation on data virtualization earlier, but you have to get with the program, because were all moving to data virtualization and we moving there fast.”

Chris Bradley, IPL Statoil Case Study

Chris Bradley, whose consultancy IPL has guided several of the most advanced data virtualization implementations in the world, also spoke about data virtualization in his presentation A Model Driven Data Governance Framework for MDM: A case study from Statoil. In particular, Chris described the many ways organizations can flexibly leverage data virtualization to meet their master data management objectives.

Data Virtualization Users

Composite had a number of customers attending the event and related activities including Barclays, Betfair, GSK, Sky and The World Bank.

Data Virtualization Book Giveaways

Data Virtualization’s big two books were also hot.

Every conference attendee was given a free copy of  Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility. I saw lots of people reading it during the breaks. And as co-author of this one, it was fun to share “the story behind the stories” with them.   

Further, our friends from Denodo were registering delegates for a free copy Rick’s upcoming book when published later this year.  

To learn more about these books, take a look at these videos:

• Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility
• Data Virtualization Book – Business Intelligence Architectures: Revolutionizing Data Integration for Data Warehouses

No Agility without Agile Architecture

Enterprise architects know the score.  According to Brian, only 21% of enterprise architects in Forrester’s 2011 State of EA Survey reported being even modestly agile.

Brian identifies three barriers can get in the way of agility in architecture:

• “Brittle processes and legacy systems – we all know it this one; the current state mess of processes that cannot adapt to change and legacy systems where everything is connected to everything else, so even the smallest changes have broad impacts.Techniques to overcome this barrier include partitioning the problem into digestible pieces to show incremental progress and short term payoff.
• Victim mentality – frequently when I talk to IT personnel I hear, “the business doesn’t know what it wants”, or “the business doesn’t understand what it takes to build solutions.” The story is the same when I talk to the business, “IT is too slow. I don’t understand why they take so long” or “I don’t deal with IT if I don’t have to.” People get too busy pointing fingers to act quickly, or too focused on their own pain to address it organizationally. Frequently this is coupled with a feeling of accountability without visibility or authority. Techniques to overcome this barrier include developing joint accountability, and simple shared principles that align stakeholders.
• The quest for bullet proof solutions – as EAs, we are paid to create enterprise solutions which we design to have all kinds of qualities such a recoverability, robustness, etc. Often the driving quality requirement is agility, yet we only know one trick – make it bullet proof. Frequently this is caused by an aversion to risk, and conversation about failure avoidance rather than risk and reward. Overcoming this barrier includes techniques for managing risk including use of architecture zones and empowering decision rights and accountability to promote better risk taking.”

I have been architecting enterprise systems for over 20 years. This list of barriers is a good as any.

So Now What?

As architects, the key question is what are we going to do about it?

Do we continue to accept these excuses or do we find ways to move ahead in spite of them?

I know where I stand.  And I know what I recommend to help enterprises overcome these barriers; data virtualization. That’s because business agility is the primary business value data virtualization provides.

You can see for yourself by reviewing the ten enterprises profiled in the recently published Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility and learning about their agility improvements.

Brian’s Data Virtualization Reaches Critical Mass provides similar successes.

Make it so!

On April 12, 1912, the maiden voyage of the seemingly unsinkable RMS Titanic ended in disaster. 

Now, one hundred years later to the day, IT is on course for a similar collision, with similar catastrophic fate. 

Business’s Icy Relationship with IT

Business dissatisfaction with IT is well-chronicled by TDWI and others. IT responsiveness surveys¹ show an average time to add a data source of nearly eight weeks, with another seven weeks added-on to create a new report or dashboard. What was it that Adam Smith said about the long run?

TDWI research on the top five drivers for self-service BI² show desultory results of similar magnitude: 

• Constantly changing business needs (65%)
• IT’s inability to satisfy new requests in a timely manner (57%)
• The need to be a more analytics-driven organization (54%)
• Slow and untimely access to information (47%)
• Business user dissatisfaction with IT-delivered BI capabilities (34%)

Approaching the Iceberg, Full Steam Ahead

IT’s full-steam ahead mentality, seemingly with little care for their business customer, is not a course for long-term survival. Like the proverbial iceberg, business leaders today are revealing just ten percent of their future plans in discussions covering analytics, mobile and cloud adoption. 

Over the next five years, what lies beneath will be fully revealed.  I envision…

• Need analytics?  There’s an app for that! Apple-like apps stores, full of easy to learn, personalize and use business applications, will serve business users’ many and varied needs.
• Need data?  Google it! Information access will be ubiquitous.
• Need to visualize?  Play it! Location awareness and 3D visual reality will extend human sense and response in unprecedented ways. 
• Need computing?  Charge it! Massive cloud computing factories will address complexity and provide compelling economics at the expense of on-premise craftsmen and computing resources. 
• Need advice?  Twitter it! Mavens and super users will guide solutions by influence rather than IT standards.  Enjoy the wisdom of crowds!

What Becomes of IT?

And what becomes of IT as we know it?  That too is easy to project.  At their retirement clubs, someone will need to rearrange the deck chairs.

1. Self-Service Business Intelligence: TDWI Best Practices Report, @TDWI July 2011
2. 2011 TDWI BI Benchmark Report:  Organizational and Performance Metrics for Business Intelligence Teams, @TDWI December 2011

Last week I attended the Gartner BI Summit in Los Angeles.  While I am an avid follower of Gartner’s Information Management, Analytics and BI research, it was great to hear this content directly from the analysts and gauge the reaction of the thousand plus attendees.

Data Virtualization’s Capabilities are Critical Enablers

I was pleased to see data virtualization’s rising prominence in the Gartner analyst presentations.  In his presentation entitled Gartner’s Information Capability Framework: Creating Value from Information Assets, Ted Friedman described the breadth of enabling infrastructure functions required by today’s agile businesses and complex big data environments.

In this framework, decoupling information assets from consuming applications is a key tenet.  This is something data virtualization is especially adept at.  Further data virtualization can provide a number of the key functions including discovery, modeling, abstraction, semantic reconciliation, aggregation, enrichment, standardization, publishing and more, in an integrated solution from development through operation.

Data Virtualization Core to Logical Data Warehouse

Data virtualization was also a core element in the Gartner’s Logical Data Warehouse research.  In his presentation entitled How Will We Survive Without the EDW, Mark Beyer described the major forces impacting the traditional enterprise data warehouse.   Volume, variety, velocity and complexity are but a few of the extremes that are causing a major reassessment of earlier information management paradigms.

Based on my interpretation of the Logical Data Warehouse, data virtualization will play a key role.  Data virtualization along with physical repositories such as Teradata and the Oracle database, and distributed processing approaches such as Hadoop will be deployed in a mix and match approach based on service level needs.  For example, use data virtualization to service up to the minute, smaller queries and distributed processes for complex, large scale analytics.

What Do You Think?

If you attended the Gartner BI Summit and/or have been following their Information Capability Framework or Logical Data Warehouse research, I would enjoy seeing if you felt the same level of data virtualization heat.   I’ll start a discussion at the The DV (Data Virtualization) Café LinkedIn group where we can explore this further.

Data virtualization is a hot topic today as business and IT realize data virtualization’s business agility, time-to-solution and resource savings benefits. As a result, numerous new articles, research reports, white papers, videos and more become available every day.  

If you are interested in data virtualization, how do you keep up?

Announcing the Data Virtualization Café

The Data Virtualization Café^TM or DV Café^TM is a microsite 100% dedicated to the topic of data virtualization. You can think of the DV Cafe™ as your destination for all things data virtualization.

With the DV Café, staying abreast of key data virtualization trends and insights is now much easier. Not only is the freshest data virtualization content consolidated in a single place, the DV Café’s format make this content enjoyable to consume. Further, the DV Café’s “best of data virtualization” compilation approach, along with links to additional sources, ensures your time is used wisely. 

What’s Brewing at the DV Café?

The DV Café includes a wide range of data virtualization information including:

• The latest data virtualization news articles from leading media such as the BeyeNETWORK, DBTA, Information Management, ITBusinessEdge, SearchDataManagement, Virtualization Magazine, and more.
• The most relevant data virtualization research reports from thought leaders at analysts firms such as Gartner, Forrester, TDWI and EMA as well as data virtualization white papers from independent analysts such as Claudia Imhoff, Rick van der Lans and Colin White. 
• A diverse set of videos that address key data virtualization topics from the point of view of users, analysts, vendors and more.
• Blog content from the Data Virtualization Leadership Blog^TM, data virtualization’s most complete and far-ranging blog site covering data virtualization strategy, architecture, best practices, products, business value, market trends and more.
• Direct links to books about data virtualization.
• Invitations to upcoming data virtualization events and web seminars.
• Interactive discussions on diverse data virtualization topics with members of the DV (Data Virtualization) Café LinkedIn group.

Is the DV Café for Me?

The DV Café is designed for multiple audiences. If you are new to data virtualization, the DV Café can accelerate your learning curve. If you already know about data virtualization, the DV Café can help you become an expert. And if you are a data virtualization advocate, the DV Café can help you become a true data virtualization champion.

Can I contribute to the DV Café site?

The editors at the DV Café value high quality, high impact data virtualization content from any source.  If you have something to contribute, please send an email to info@datavirtualizationcafe.com.

In addition, you join the DV (Data Virtualization) LinkedIn group.  Focused exclusively on data virtualization, this community contributes discussions, polls and other insights and is linked directly to the DV Café site.

So how is small and mid-sized organization adoption proceeding? And how are these firms benefiting?

Typical Technology Adoption Trends

In general, large enterprises tend to be early adopters for all new technologies.  For example, they were the first to deploy data warehouses fifteen years ago, data warehouse appliances five years ago, and NoSQL data stores today. In many instances, they adopted data virtualization to help them extend and gain more value from these earlier investments. 

Further, larger enterprises have more silos of data. And thus they can more easily justify new data integration investments.

That said, the good news is data virtualization works equally well for smaller enterprises or at the project level in a large enterprise.   

And the further good news is that larger enterprises’ earlier investments are now paying off in stronger data virtualization offerings from everyone – small, medium or large.

Small and Medium Enterprise Adoption Is Accelerating

At Composite Software, we have a number of small and medium enterprise customers. Let me talk about just a few.

The first is a non-profit, Compassion International. They provide charitable support services to children in developing countries such asHaiti where poverty is a huge problem. By 2020, the organization wants to quadruple the number of its beneficiaries. To achieve this goal, the IT team needed to modernize the organization’s information infrastructure.

Data virtualization has enabled Compassion to both scale their BI infrastructure and meet demanding new information requirements much more easily and quickly than before. Time-to-solution for new information requests has improved by more than 50%.  And data quality and integrity have improved as well.

Additional Examples

Other small and mid-sized data virtualization adopters include Brown University, Carfax, Cricket Wireless, IEEE, Kaplan, Massachusetts Department of Education, Partnerships for Schools, Scripps Networks and many more. 

Like the larger enterprises, these organizations and numerous other small and medium-sized enterprises share substantive business pressures and face significant technology transformations. 

In every case, they were willing to look beyond traditional data integration methods and try something new, data virtualization. And as a result, all are seeing significant business decision, time-to-solution and resource agility benefits.   

Lyndsay Wise, the BI analyst best known for her small and mid-sized enterprise research and I examine this adoption further in Understanding Data Virtualization.

How Slow Is BI Today?

According to the TDWI Benchmark Report: Organizational and Performance Metrics for Business Intelligence Teams, on average it takes eight weeks to add a new data source and another seven to develop a new complex report.

• How can BI become more agile?
• Will agility come from new BI tools?
• Or will the answer come for the new approaches to data integration such as data virtualization?

Look to the Data, not to the BI Tools

In a recent white paper, “Data Virtualization for Business Intelligence Agility”, Rick advises IT organizations to think broadly about their BI agility challenge. “It’s not simple to pinpoint why most of the current business intelligence systems are not that agile. It’s not one aspect that makes them static. But undoubtedly one of the dominant reasons is the database-centric solution that forms the heart of so many business intelligence systems.”

Long Chain of Data Stores Reduces Agility

The architectures of most business intelligence systems are based on a complex chain of data stores starting with production databases, data staging areas, a data warehouse, dependent data marts, and personal data stores. Simply maintaining this complexity is overwhelming IT today. In addition to Rick’s description of this chain in the white paper, he also addresses the same subject in a video chalk talk.

According to van der Lans, “While these classic systems have served business well for the last twenty years. However, considering the need for more agility, they have some disadvantages:

• Duplication of data
• Non-shared meta data specifications
• Limited flexibility
• Decrease of data quality
• Limited support for operational reporting:
• Limited support for reporting on unstructured and external data”

Data Virtualization Delivers BI Agility

Rick van der Lans’ white paper discusses how data virtualization products can help to make business intelligence (BI) systems more agile. According to Rick, data virtualization simplifies and thus improves BI time-to-solution in through aspects such as:

• Unified data access
• Data store independence
• Centralized data integration
• Transformation and cleansing
• Consistent reporting results
• Data language translation
• Minimal data store interference
• Simplified data structures
• Efficient distributed data access

The financial benefits of these data virtualization capabilities are significant. And they can be applied flexibly to fund additional data integration activities and/or other business and IT projects.

Learn More from Rick

Rick will headline the March 20 broadcast of The Briefing Room discussing “Stay Flexible: Five Ways to Harness Information Assets” at 4pm ET. Go here to register for the March 20 Briefing Room broadcast.

You can also watch Rick’s three data virtualization videos at http://www.compositesw.com/resources/videos/.

Broader Deployment Requires Larger Scale Systems

Patterns of data virtualization deployment are evolving rapidly. Five years ago, most new Composite Software customers purchased its data virtualization platform to integrate data at the project level. Last year, half of them purchased us to serve as the enterprise-scale data layer from the start, with over 60 percent of the earlier buyers now on the same journey. With many organizations already supporting thousands of users, this trend will continue with user counts rising to the tens of thousands on a global basis.

Data virtualization’s ultimate deployment architecture is a low-latency, globally distributed data fabric, operating 24×365, with full source and consumer location transparency that spans both on-premise and cloud.  Operating at this scale will significantly impact to the Composite Software’s roadmap for its development, run-time and management environments. 

Expanding User Base Drives Ease-of-Use Agenda

To date, most data virtualization users have been skilled developers and administrators.  This is reflected in the user tools provided by today’s data virtualization vendors.  As data virtualization deployments expand, the range of business and IT staff using this technology becomes larger and more diverse.  Enabling these new communities to learn data virtualization quickly, and then use it productively, is critical to data virtualization’s expansion.

To ensure this happens, Composite Software’s data virtualization roadmap implements two strategies. The first is to continue pushing the bounds of automation to eliminate work wherever possible.  The second is to continue to provide more powerful and intuitive tools to maximize user productivity for the work that remains.

Better Governance for Consistency, Compliance and Control

As data virtualization deployments evolve from projects to layers to global data fabrics, the need for greater data governance expands as well.  Control over data access, assurance of data quality, sharing of common specification and compliance with standards and regulations are but a few examples of governance requirements.

The Composite Software data virtualization roadmap includes capabilities that expand internal visibility over and support of governance practices and policies. Closer alignment with adjacent technologies and processes will simplify governance as well.

Conclusion

Just as business and technology continuously evolves, so too will the scope and impact of data virtualization adoption and Composite Software’s offerings. 

By meeting the markets increasing access, optimization, deployment, ease-of-use, and governance requirements, Composite and our customers will thrive.  

Enabled by a solid roadmap, this will be an exhilarating and rewarding journey.