Leveraging Meta Data to Automate the ETL Process
Tom Harrocks
Senior Principal
Knightsbridge Solutions
Monday, April 24, 2023
5:00 pm - 6:00 pm
Level: Intermediate

A common goal of enterprise data warehouse (EDW) projects is to use meta data to facilitate the ETL. By defining business rules within a rigorous technical framework, and using coding templates for data manipulation, significant proportions of the ETL code in a data warehousing project can be automated. Automation reduces maintenance costs, improves code quality and provides higher precision testing.

A major health insurer was concerned with providing a basis for automated code generation while retaining business value. The insurer developed a framework for storing enriched technical meta data to support business rules, semantic translations and data lineage. The accuracy and completeness of the technical meta data was ensured by providing a migration utility to import source and target database schemas. An interface was developed that associated business rules with a specific source to target transformations.

This innovated approach created dramatic efficiencies in creating and maintaining accurate meta data. The organization reduced development time by 50% and ongoing maintenance by 25%.

In this case study-driven session, learn how to:
  • Integrate technical and business meta data
  • Identify and utilize templates with meta data to automate process
  • Automate semantic translations
  • Reduce development costs and response to changes in the data model
  • Maximize the robustness of DML operations
  • Leverage optimization efforts across the project
  • Validate the business and technical
This presentation is intended for intermediate and advance-level data warehouse architects.
Mr. Harrocks is a senior principal with Knightsbridge Solutions. He has over 20 years experience delivering enterprise information integration solutions. His specialty is the development of highly configurable solutions that focus on configuration efforts to develop reusable patterns. Mr. Harrocks has 15 years experience providing integration solutions in the development of analytical systems in the pharmaceutical and healthcare industries. He has also authored published articles in scientific academic journals.