March 6, 2007

10:15 AM - 11:15 AM

Level: Introductory/All Levels

Data warehouse architectures are characterized by complex models, lots of latency and difficult maintenance. For example, a conventional data warehouse has a staging area, one or more Operational Data Stores, an "Enterprise" data warehouse, metadata schema and a slew of dimensional models, relational or otherwise. Loading new data is a cascading batch process and time-consuming, also frustrating the need for more current data. Modifying a schema requires examination of all of the dependencies in the Augean Stable, with all of the attendant latency in putting changes into effect.

• EII proved that is possible to retrieve information in a federated way from online systems without destroying performance
• Semantic technology makes it possible to design metamodels that can reason, allowing for dynamic transformation of data, much richer rules and the ability to build abstraction layers using open standards
• Moore's Law allows us not to have to manage from scarcity anymore. A primary reason data warehouse's exist is because there was a lack of resources to perform more on-the-fly processing
• Operational BI is forcing data warehouses to step up to not only real-time data, but real-time query performance
• Open standards, SOA, web 2.0, open source
• all of these factors are hastening the convergence of operational and analytical processing. Data warehouses won't disappear anytime soon, but it's time to start think about where they can add value and where they should step out of the way.
  

Neil Raden is the founder of Hired Brains, a research and advisory firm in the BI industry. In the past he's been a mathematician, an actuary, a software developer, a DSS consultant, and a founder of a SI doing DSS/Data warehousing/BI. He has almost 30 years experience, is a widely published writer, well-known speaker and consultant.

He has personally designed over 100 data warehouses, implemented dozens of large analytical applications in finance, marketing, distribution, logistics, actuarial, scientific, statistical, consumer products and more.