I am often asked how to managing query performance of frequently-accessed data sources, in order to minimize impact on operational systems or to support service level agreements.

While this can be a challenge in large scale data virtualization environments, caching, also known as materialized views, provides an excellent performance adjunct to query optimization.

Caching Flexibly Persists Data to Meet Service Level Needs

Mature data virtualization platforms provide multiple caching options and techniques.

These let you flexibly persist queried data to meet data delivery service level agreements and protect source system performance.

• Any View, Any Service, Any Procedure – Any view, service or procedure may be cached for future use, and all caches may be periodically and automatically refreshed to stay synchronized with their systems of record. Queries are processed against caches just as if you were querying the original data source.
• Multiple Cache Repository Options – It’s a good idea to cache data with other frequently accessed sources. Composite for instance can cache on DB2 , Greenplum, Microsoft SQL Server, MySQL, Netezza, Oracle, Sybase, Teradata and Vertica.
• Event-driven Refresh – Updating a cache based on defined business rules provides significant flexibility based on events and activities.
• Scheduled Refresh – Updating a cache based on set times is useful in more schedule-driven environments.
• Manual Refresh – Updating a cache on demand, for example when a report is run, provides an additional option.
• Incremental Refresh – Updating a partial cache based on triggered changes is useful for large data sets with frequent refreshes.
• Native Data Source Load – Using the target repository’s native load functions to load and refresh the cache accelerates loading times by 10x or more.
• Parallel Load – Using multiple threads to load a cache in parallel also accelerates loads.
• Centralized Caching – In centralized mode, all cached data is stored in a single cache repository. Centralized cache refreshes are fully configurable including timed refresh, event-based refresh (CJM or JMS message), incremental refresh and forced refresh.
• Distributed Caching – In distributed mode, users dedicate one or more data virtualization servers as edge servers and configure edge cache policies.  Edge cache policies let you control which cache data is replicated from the central cache to the edge location and the refresh rules. Refresh can be time based, event-based or incremental.
• Clustered Deployment – For clustered deployments, a centralized cache reduces the need for each cluster node to re-fetch the data from the source, which significantly reduces the impact on production data sources.

Enjoy the Flexibility

As you can see, caching’s many options help provide architects and developers with significant flexibility to address nearly any performance or SLA challenge.

Caching is easy to implement, and easy to change as conditions change. Take advantage.

Architecting Tomorrow’s Business Outcomes

Architecting Tomorrow’s Business Outcomes was the conference theme. The conference’s stated objective was to help attendees develop intelligent responses to technology disruptions.

Said another way, how can you achieve better business outcomes by taking advantage of new and changing technology?

Data Virtualization is a Key Sub-Theme

In the session titled Real-Time Business – Lessons from Leaders, Brian Hopkins from Forrester, John Hershberger from USAA, Peter Memon of Barclays, and Phil Shelley of Sears Holdings described how their firms are pushing the big data envelope for better business outcomes including improving customer service, reducing risk, cutting costs and more.

Memon spoke specifically about the role data virtualization plays in Barclay’s big data strategy. He used a credit trading risk management application as an example.  In this case, they use data virtualization to join together current trading data on the trading systems with historical trading data stored in Hadoop.

WIth this approach, for the first time, the credit risk analytics have immediate access to historical data that due to the high volumes and high traditional storage costs, used to be stored on tape. The better business outcome is improved risk management, without extra costs.

Data Consolidator, Investor or Innovator

In the session titled The Data Management Technology Landscape, Forrester Analyst Michele Goetz described a maturity continuum where high performance companies move beyond traditional data consolidation-centric strategies by investing in innovating using new approaches such as data virtualization and big data.

In fact, Goetz went so far as to say that if you are going to do big data, then you have to do data virtualization as well.

That certainly aligns with Barclay’s experience above.

Catching Up with Analysts and Users

The conference also proved a great way to catch up with analysts such as Hopkins and Goetz, as well as Boris Evelson who covers data virtualization as a key enabler in his agile BI research.

It was also useful to discuss data virtualization adoption with users beyond Barclays, including Aflac, Kaplan, Pioneer Natural Resources, and more.

Interestingly, Pioneer’s Martha Gardill will be speaking about data virtualization in her talk Responding to Change with Agility later week at the Petroleum Network Education Conference in Houston.

Will I See You Next Year?

If the past is a good guide, then I expect I will go to this conference again next year.  I hope to see you there.

And in the meantime, if you want to learn more about what analysts and users have to say about data virtualization, check out the Data Virtualization Channel on YouTube.

Data Virtualization: Claudia Imhoff Interviews Rick van der Lans on Getting Started with DV

Tudor purchased the Composite Data Virtualization Platform to create a virtual data layer to streamline access to market and trade data for traders, analysts, and management. “This increased agility will enable us to better serve our user community by reducing the time required to deliver data requests,” says Paul Mark Skittone, Tudor’s Head of Data Services.

Benefits

The data virtualization software will provide Tudor Investment a decoupling layer, providing the necessary agility to integrate new sources of data that come on line throughout the day.

Tudor will also receive many other benefits from the data virtualization platform including:

• Improved governance;
• Simplified enforcement of row level security;
• Abstraction and reuse of existing calculations;
• Upgrade path for integrating existing systems and technologies.

Data Virtualization in Financial Services Organizations

The business agility gained from data virtualization is essential to financial services organizations maintaining their competitive edge. Data virtualization is a more agile, lower cost data integration approach that successfully addresses financial instrument and product line data silos and delivers significant business benefits.

Financial institutions that successfully integrate their disparate data by turning distributed data silos into competitive information assets are the ones who will lead in this ever dynamic industry.

Learn More

To see testimonials of other data virtualization users please visit our YouTube customer playlist at The Data Virtualization Channel.

The analytic data domain includes all kinds of data, including:

• traditional enterprise data from sources like the Enterprise Data Warehouse;
• big data from sources like Hadoop;
• cloud data from SaaS applications and public providers;
• social media data from sites like Facebook and Twitter; and
• personal/desktop data from spreadsheets and flat files.

Data virtualization is an optimal solution to integrate all these sources, thereby improving analysis and business insight.

Enterprise Data

The effective analyst needs access to all enterprise data, not just the data in the warehouse. While most of this data is relational, it still exists in a dispersed collection of silos, each with its own data model. The diversity of connectivity, authentication protocols, SQL dialects, and data models makes enterprise data more difficult to leverage than it should be.

Data Virtualization simplifies access to enterprise data by providing built-in connectivity to most enterprise data management platforms, and providing a standard SQL interface to query the data. It also provides tools to discover relationships among data entities in different silos.

Big Data

Hadoop is fast emerging as a leading repository for big data analytics. However, the map-reduce paradigm used to interact with Hadoop data sources is not well understood in typical enterprise IT organizations. This may not be a problem when performing specialized analytics, but it can be a big barrier when trying to combine Hadoop and enterprise data using enterprise IT standard languages such as SQL.

Data virtualization overcomes the query language challenge by integrating and extending Hive, and thus provides a unified SQL based approach for querying both enterprise and Hadoop data sources. Complex data digestion and reduction will still be done by map-reduce, but leveraging that data to combine with other data can easily be done through data virtualization.

Cloud Data

Many organizations leverage SaaS platforms like Salesforce.com, which results in a nexus of data stored in the cloud, and this data is valuable to analytics. In addition, more and more data from third-party providers is becoming available to companies looking to leverage certain specialized data sets. Both of these types of data require access across the Internet through service protocols.

Data virtualization provides access to most cloud-based data sources through standard SOAP and REST protocols, and leverages other web service standards to complete the picture. Data virtualization also facilitates the querying, transformation, and caching of this data to make it suitable for analytics.

Social Media Data

Facebook, Twitter, and other social media sites hold a tremendous amount of data that can be useful to customer analytics. Unfortunately this data is difficult to access through standard protocols, and it is difficult to acquire the appropriate authorizations.

Data virtualization can access and integrate this social media data through third-party data providers like Gnip. The combination of Gnip, which is officially authorized by the social media sites to distribute their data, and data virtualization, which can access and transform the data, brings social media data to the analyst’s desktop.

Personal/Desktop Data

Although the bulk of data that an analyst works with comes from elsewhere, there are often local spreadsheets and flat files that an analyst would like to use in conjunction with the rest of the data. These files are an important tool in the analyst’s arsenal to create specialized data sets to augment the analysis being done.

With data virtualization analysts easily access and integrate Excel and flat file data. Often this data is either untyped or text-only, and data virtualization allows the analyst to transform and cast the data into an appropriate form.

Learn More

If you want to learn more about how data virtualization can help with analytics, check out these white papers:

• Turbocharge Analytics with Data Virtualization
• Analytics Best Practices: The Analytical Hub
• Analytics Best Practices: The Analytical Sandbox
• A Better Way to Fuel Analytical Needs

In their best-selling book, Big Data: A Revolution That Will Transform How We Live, Work, and Think, Mayer-Schonberger and Cukier describe the synergy that occurs when previously unrelated and disparate data is brought together to uncover hidden insights. But these advanced analytics data requirements are a double-edged sword as these more diverse sources complicate data integration and constrain progress.

The Analytics Pipeline

The analytics pipeline includes six major process stages, often implemented in an iterative manner including:

1. Find the Data
2. Access the Data
3. Build a Sandbox for the Data
4. Build the Analytic Model
5. Analyze the Results
6. Develop and Communicate the Business Insight

Most analysts spend more than half their time and effort assembling the data needed to perform the analytics, and the rise of big data and cloud computing has made this more severe. The typical analyst is faced with numerous data challenges that must be overcome.

Different data shapes

It used to be the case that most data was tabular, and even relational. But that has changed during the last five years with the rise of semi-structured data from web services and other non-relational data streams. Analysts must now work with data in multiple shapes, including tabular, XML, key-value pairs, and semi-structured log data.

Multiple interfaces and protocols

Accessing data has gotten more complicated. An analyst used to simply use ODBC to access a database, or receive a spreadsheet via e-mail from a colleague. But now analysts must access data through a variety of protocols, including web services through SOAP or REST, Hadoop data through Hive, and other types of NOSQL data through proprietary APIs.

Larger data sets

Data sets have grown larger and larger during the last decade, and it is no longer reasonable to assume that all the data can be assembled in one place, especially if that place is your desktop. The rise of Hadoop is fueled by the tremendous amounts of data that can be easily and cheaply stored on this platform. Analysts must be able to work with data by leaving it where it is, and intelligently sub-setting it and combining it with data from multiple sources.

Iterative methodology

The analytic development process is characterized by exploration and experimentation, and this requires data sets to be iteratively assembled and updated as the exploration proceeds. In other words, data agility is an important part of successful analytics.

What is your Analytics Data Challenge?

Do you agree with these challenges? Are there others to consider as well?

Compare notes with Alpine Data Labs’ Steven Hillion, as his video describes the challenges he sees.

In this blog, I will call upon the work of one of the best architects I know, Mike Tinius, to describe a reference architecture Mike developed and that you can use as a guide when abstracting data using Composite’s data virtualization platform or others.

Data Abstraction Reference Architecture

The diagram above captures the various layers in this reference architecture including:

• Data Consumers – Client applications need to retrieve data in various formats and protocols that they understand. Composite delivers the data to consumers using the most popular standards including SOAP, REST, JDBC, etc.
• Application Layer – The “Application Layer” serves to map the Business Layer into the format which each application data consumer wants to see. It might mean formatting into XML for Web services or creating views with different alias names that match the way the consumers are used to seeing their data.
• Business Layer – The “Business Layer” is predicated on the idea that the business has a standard or canonical way to describing key business entities such as customers and products.  In the financial industry, one often accesses information according to financial instruments and issuers amongst many other entities. Typically, a data modeler would work with business experts and data providers to define a set of “logical” or “canonical” views that represent these business entities.   These views are reusable components that can and should be used across business lines by multiple consumers.
• Physical Layer – The “Physical Layer” provides access to underlying data sources and performs a physical to logical mapping by integrating physical metadata and formatting views.
  • The “Physical Metadata” is essentially imported from the physical data sources and used as way to onboard the metadata required by the data abstraction layer to perform its mapping functions.  As an “as-is” layer, entity names and attributes are never changed in this layer.
  • The “Formatting Views” provide a way to map the physical metadata into the Data Virtualization layer by aliasing the physical names to logical names. Additionally the formatting views can facilitate simple tasks such as value formatting, data type casting, derived columns and light data quality mapping. This layer is derived from the physical sources and performs a one-to-one mapping between the physical source attributes and their corresponding “logical/canonical” attribute name. This layer serves as a buffer between the physical source and the logical business layer views.  As such, caching may be introduced at this level if and when it makes sense. Rebinding to different physical views during deployment is another role these views take on. Naming conventions are very important and introduced in this layer.
• Data Sources –The data sources are the physical information assets that exist within and without an organization.  These assets may be databases, packaged applications such as SAP, Web services, Excel spreadsheets and more.

How Have You Implemented Data Abstraction?

I hope you can benefit from Mike’s great work. And Mike and I would be pleased to answer any questions you might have.

And if you have any data abstraction best practices you can share, we would love to learn about them as well.

But they struggle with the variety of enterprise, cloud and big data sources, and all their associated access mechanisms, syntax, security, etc.  Further, few data sources are structured properly for business user or application consumption, let alone reuse.  And often the data is incomplete or duplicated.

Data Abstraction Overcomes These Challenges

Data abstraction overcomes data source to data consumer incompatibility by transforming data from its native structure and syntax into reusable views and data services that are easy for application developers and business analysts to understand and consume.

Data Abstraction Technology Options

Some data abstraction approaches used today work better than others.

For example, some organizations build data abstraction by hand in Java or use business process management (BPM) tools. Unfortunately, these are often constrained by brittleness and inefficiencies. Further, such approaches are not effective for large data sets since they lack the robust federation and query optimization functions required to meet data consumers’ rigorous performance demands.

Data warehouse schemas can also provide data abstraction. Data modeling strategies for dimensions, hierarchies, facts and more are well documented. Also well understood is the high cost and lack of agility in the data warehousing approach. Further, data warehouse based schemas don’t include the so many new classes of data (big data, cloud data, external data services and more) that reside outside the data warehouse.

Data Virtualization is a Superior Solution for Data Abstraction

Data virtualization is an optimal way to implement data abstraction at enterprise scale. From an enterprise architecture point of view, data virtualization provides a semantic abstraction or data services layer supporting multiple consuming applications. This middle layer of reusable services decouples the underlying source data and consuming solution layers. This provides the flexibility required to deal with each layer in the most effective manner, as well as the agility to work quickly across layers as applications, schemas or underlying data sources change.

Key Benefits

Data abstraction using data virtualization provides the following benefits:

• Simplify information access – Bridge business and IT terminology and technology so both can succeed.
• Common business view of the data – Gain agility, efficiency and reuse across applications via an enterprise information model or canonical model.
• More accurate data – Consistently apply data quality and validation rules across all data sources.
• More secure data – Consistently apply data security rules across all data sources and consumers via a unified security framework.
• End-to-end control – Use a data virtualization platform to consistently manage data access and delivery across multiple sources and consumers.
• Business and IT change insulation – Insulate consuming applications and users from changes in the source and vice versa. Business users and applications developers work with a more stable view of the data. IT can make ongoing changes and relocation of physical data sources without impacting information users.

Data Abstraction Reference Architecture

With hundreds of installations in a variety of industries, Composite has developed a Data Abstraction Reference Architecture that architects and analysts can use this as a guide when abstracting data using data virtualization. In my next blog entry I will drill down into its components.

Analytic Sandboxes and Data Hubs

Rick believes these two architectural frameworks, analytical sandboxes and analytical hubs, form the foundation for the kind of agile, self-service data integration required when developing the new analytics required today.

White Paper Format Provides Depth

The new papers focus on the specific business needs and technology solutions—including data virtualization—for implementing analytical sandboxes and hubs,  Analytics Best Practices: The Analytical Sandbox and Analytics Best Practices: The Analytical Hub.

The white papers are follow Rick’s previously published A Better Way to Fuel Analytical Needs, where he explores writes the potential value of business analytics and the amount of data to fuel it are significantly expanding, yet business intelligence and data-integration backlogs are constraining enterprises attempting to tap that value.

Data Is the Key to Analytics Success

Rick recognizes the analytic data challenge that occurs as organizations struggle with the constant influx of data and ever-changing business environment, business analysts need to access increasingly diverse data sets, inside and outside their organization.

Analytical sandboxes and hubs powered by data virtualization address the multiple-query challenges of situational business analytics and provide enterprise-scale processing, storage and networking capabilities while avoiding the pitfalls of makeshift data shadow systems.

Data Virtualization Provides Agility and Flexibility

According to Rick, “Data-integration capability will expand beyond traditional ETL to include data virtualization, which enables organizations to expand the data used in their analysis without requiring that it be physically integrated. Companies do not have to get IT involved (via business requirements, data modeling, ETL and BI design) every time data needs to be added. This iterative and agile approach supports data discovery more productively for both business and IT.”

Learning From an Expert Who Does It Everyday

I like Rick.  He is a sharp guy with more than 25 years of experience in delivering BI solutions and is an author and a visiting lecturer at Northwestern University. I hope you find his white papers an essential resource in your analytics journey.

In addition, if you like webinars, check out the one Rick and I did together few months ago Webcast: A Better Way to Fuel Analytical Needs

Finally, you can stay abreast of all of Rick’s latest insights on his blog The Data Doghouse.

It was an incredible event in terms of content and community including:

 -        32 Gartner analysts presented 3.5 days of content, in over 100 sessions including keynotes, workshops, roundtables and more.  Data Virtualization and Data Federation (Gartner uses both terms) were highlighted as key enablers in a number of these sessions.

-        Over 50 vendors participated in the solution showcase and led an additional 25 or more sessions.

-        Over 1500 attendees, from both business and IT, joined as well with the goal of driving more value from their BI and Analytics investments.

Analytics to the Fore

In the opening keynote, Fast Forward: New Information.  New Challenges.  New Solutions, analysts Ian Bertram, Ted Friedman and Bill Hostmann suggested a tectonic realignment of IT solution thinking that placed Analytics at the center of all things IT.

This revolutionary realignment is right if you believe Professors Andrew McAfee and Erik Brynjolfsson of MIT who said in a recent Harvard Business Review article:

“Companies that inject big data and analytics into their operations show productivity rates and profitability that are 5% to 6% higher than those of their peers.”

CIO’s certainly believe this.  In a 2012 survey of 2300 CIOs by Gartner, Analytics was their number one technology priority.

So I guess it makes sense that Gartner has changed the name of this event, now calling it the BI and Analytics Summit.

Fresher Perspectives

As the guy at Composite charged with staying abreast of the latest analyst perspectives, I truly enjoyed the up-to-minute research and insights that were presented this year.

Unlike prior years where the Gartner analyst’s appeared to be targeting the 50th percentile adopters in the audience, this year they focused on the vanguard, presenting only their latest research and describing the most-advanced, early adopter use cases.

Data Virtualization (and Federation) and Data Integration

In the data integration sessions such as Ted Friedman’s Advancing Your Data Integration Competencies in Support of Analytics, data virtualization was discussed several times.   Ted discussed increasing adoption rates in general and how data virtualization enables the Logical Data Warehouse, data access services in SOA and registry MDM implementation styles.   As a best practices example of an Integration Competency Center, Ted described the shared services data integration approach used by data virtualization user and Composite customer Pfizer.

Data Virtualization (and Federation) and the Logical Data Warehouse

Along the Logical Data Warehouse theme, Mark Beyer in this session entitled Leveraging Big Data In Analytics With The Logical Data Warehouse, described data virtualization as one of the seven key elements in a Logical Data Warehouse conceptual architecture.   Mark also discussed data virtualization’s growing use in federating big data with the enterprise data warehouse.

You can learn more about the intersection of data virtualization and the logical data warehouse in David Besemer’s video below.

In To the Point: Data, Data Everywhere and Yes, It’s Accessible, Jamie Popkin covered data virtualization in the Logical Data Warehouse in a more hands-on design and deployment manner as a complement to Mark’s higher level, conceptual architecture.

These sessions certainly had an impact as a number of attendees stopped by the Composite booth to discuss how we could help them get started on a Logical Data Warehouse initiative.

What Did You Think?

If you attended, I imagine you had a fun and intellectually satisfying time as well. If you did not, Gartner clients can contact their Gartner account managers for reprints and replays.  And for everyone, check out #GartnerBI on Twitter.  The wisdom of this crowd is amazing.

These new technologies improve business competitiveness, increase agility, save money and more.

But migrating from existing systems can be risk prone.  And efforts often drag out.

Use Data Virtualization Migrate Successfully, Faster and with Less Risk?

Most people think of data virtualization as a high productivity, low cost way to integrate data for business intelligence and analytics.  However, it is also very effective at legacy migration.

The key to data virtualization’s migration success is how it decouples source and consuming applications.  This allows legacy migration to be done in a phased approach within the constraints of existing infrastructure.  Said another way, data virtualization inserts flexible middleware that isolates the rest of the system from other “moving parts.”

From “As Is” to “Will Be” and Everything “In Between”

First you identify the future “will be” state and all the “transitional” states in between.

Next, you insert data virtualization between the sources and consumers.  This forms a layer of insulation that allows phased changes to legacy sources or consumers without impacting the related sources and consumers.

Then you make the changes is a controlled fashion, validating success at every step.

As a result, you can modernize quickly, with low risk.

Data virtualization includes seven key capabilities you can use to accelerate and de-risk legacy migration.

1.  Migrate Legacy Data No Matter How Complex, Big or Old

Legacy data with a complex structure or code that is too old and seems difficult to integrate is not an obstacle. Data virtualization excels with complex data, big data and the ability to virtualize a wide range of data, including both structured and unstructured data.

2.  Transition Diverse Sources and/or Consumers

Data virtualization non-invasively integrates data from anywhere across the extended enterprise—in a unified, logically virtualized manner—for consumption by nearly any front-end business solution.

3.  Deploy without Impacting Your Business Processes

Data virtualization’s decoupling of sources and consumers even as you migrate them, assures the continuity of your daily business operations will not be affected.

4.  Ensure Reporting Continuity

Data virtualization also insulates reporting users during data warehouse migrations.

5.  Complete Your Migration Effort Quickly

Think in terms of weeks, not years.  Development and deployment is fast and easy with data virtualization.  Most implementations take just a few weeks.

6.  Get Productive Faster

Your IT staff will enjoy using data virtualization platforms such as the Composite Data Virtualization Platform.   They are easily to learn and use.  And they completely automate many of the tasks required.

7.  Keep Your Data Secure

Data virtualization executes authentication and authorization security functions to protect your data from improper use before, during and after migration.

Share Your Legacy Migration Success

Have you used data virtualization to help migrate a legacy system?  If so, let’s hear about it.