Solution

A "classic" solution was quickly ruled out by the project team due to the enormous workload. Instead, an early decision was made in collaboration with the client to adopt a metadata-driven approach, which also influenced the selection of components.

The new (primary) cloud platform chosen was Microsoft Azure, which could be ideally extended with b.telligent’s metaFactory. This not only allows for the automated ingestion of data sources into the Azure Data Lake (ADLS Gen2) but also analyzes these sources and adds extracted information to the metadata. The parameterized Synapse Pipelines enable processing up to the Historized Layer (Delta Parquet), where the source data is versioned and made available for further use.

Automated Processing and Streaming Architecture

Next, the b.telligent team focused on automated processing in the Snowflake Data Warehouse using Data Vault methodology for the Core Layer. To ensure seamless integration, they extended the b.telligent metaFactory with a generator for Snowflake External Tables. With this consistent ELT approach, automation with Datavault Builder was conducted exclusively within Snowflake. While metadata usage allowed for the automation of many processing steps, some development effort was still required.

Certain reporting requirements necessitated the establishment of a streaming architecture to provide users with timely and up-to-date data in Snowflake. Data originating from a PostgreSQL database was prepared for streaming using Debezium and written into a local Kafka cluster. Mirror Maker was then used to replicate the data from one Kafka cluster to another, enabling replication to the cloud via an Azure Event Hub, which subsequently stored the data in the Azure Data Lake via Event Capture. The interplay between EventGrid and Storage Queue allowed for the efficient transfer of data to Snowflake for further processing.

It quickly became evident that this area also needed to be automated as efficiently as possible. The approach included Continuous Integration/Continuous Deployment (CI/CD) via Azure DevOps, a carefully planned release management process, and the provision of independent development environments (sandboxes) for each use case. At peak times, up to 90 developers and 200 business testers worldwide were working on the project.

Secure Authorization Management for Regulatory Requirements

To address regulatory requirements from the outset, a range of functional roles for users (developers and consumers) were defined. These roles could be automatically mapped to technical roles and assigned to corresponding user accounts in applications such as Snowflake, Azure, and Datavault Builder via the central user provisioning system within Siemens. Additionally, in Snowflake, Row- and Column-Level Policies were automatically implemented to enforce the "Need-to-Know" principle, which is crucial in the financial industry. Individual permissions could then be requested via self-service and workflow within the user provisioning system.

b.metaFactory

The b.telligent metaFactory is a powerful framework developed by b.telligent for data integration with Azure Data Factory. With over 100 connectors, including those for Oracle and SAP, it offers maximum flexibility in connecting various source systems. Parameterized pipelines and a central control table allow for the easy and quick integration of new data sources. Data is stored in a high-performance Delta format in the Azure Data Lake. Thanks to comprehensive logging and monitoring functions, the metaFactory ensures the highest levels of maintainability and security—all within the Azure Subscription. Fast, secure, and future-proof: your data, optimally integrated.