Architecting Analytic Data Hubs
March 3, 2013 2 Comments
I Love Solution Architecture
I have been doing enterprise architecture for a long time. From my early days as a consultant at Accenture to my role at CTO of webMethods and for the past several years working with Composite Software’s largest enterprise customers, I find enterprise architecture a wonderful challenge.
A High Standard
Along this journey, I have developed a high standard for elegance and clarity. And unfortunately far too often I find architectures lacking in both.
The biggest problem is a lack of precision with respect to design principles. Fuzzy design principles lead to even fuzzier architectures.
So I was pleased to read Rick Sherman’s latest white paper on Analytic Data Hub design entitled Analytics Best Practices: The Analytical Hub. He does a fine job providing the elegance and clarity I appreciate.
Read Rick’s guidance for yourself:
Analytic Data Hub Design Principles
“When creating analytical hubs, follow these design principles to provide the right enterprise environment:
- Data from everywhere needs to be accessible and integrated in a timely fashion
Expanding beyond traditional internal BI sources is necessary as data scientists examine such areas as the behavior of a company’s customers and prospects; exchange data with partners, suppliers and governments; gather machine data; acquire attitudinal survey data; and examine econometric data. Unlike internal systems that IT can use to manage data quality, many of these new data sources are incomplete and inconsistent forcing data scientists to leverage the analytical hub to clean the data or synthesize it for analysis.
Advanced analytics has been inhibited by the difficulty in accessing data and by the length of time it takes for traditional IT approaches to physically integrate it. The analytical hub needs to enable data scientists to get the data they need in a timely fashion, either physical integrating it or accessing virtually-integrated data. Data virtualization speeds time-to-analysis and avoids the productivity and error-prone trap of physically integrating data.
- Building solutions must be fast, iterative and repeatable
Today’s competitive business environment and fluctuating economy are putting the pressure on businesses to make fast, smart decisions.Predictive modeling and advanced analytics enable those decisions to be informed. Data scientists need to get data and create tentative models fast, change variables and data to refine the models, and do it all over again as behavior, attitudes, products, competition and the economy change.The analytical hub needs to be architected to ensure that solutions can be built to be fast, iterative and repeatable.
- The advanced analytics elite needs “run the show”
IT has traditionally managed the data and application environments.In this custodial role, IT has controlled access and has gone through a rigorous process to ensure that data is managed and integrated as an enterprise asset.The enterprise, and IT, needs to entrust data scientists with the responsibility to understand and appropriately use data of varying quality in creating their analytical solutions. Data is often imperfect, but data scientists are the business’s trusted advisors who have the knowledge required to be the decision-makers.
- Solutions’ models must be integrated back into business processes
When predictive models are built, they often need to be integrated into business processes to enable more informed decision-making. After the data scientists build the models, there is a hand-off to IT to perform the necessary integration and support their ongoing operation.
- Sufficient infrastructure must be available for conducting advanced analytics
This infrastructure must be scalable and expandable as the data volumes, integration needs and analytical complexities naturally increase. Insufficient infrastructure has historically limited the depth, breadth and timeliness of advanced analytics as data scientists often used makeshift environments.”
What Do You Think?
Are you as impressed with Rick’s thoughts as I?