The Varigence Blog

Dev Diary: Integration with the BimlCatalog

Written by Roelant Vos on 9.15.2021

Every data solution benefits from a robust control framework for data logistics. One that manages if, how and when individual data logistics processes should be executed. A control framework also provides essential information to complete the audit trail of how data is processed through the system and is ultimately made available to users. BimlFlex provides the BimlCatalog for this.

Dev Diary – Orchestrating Mapping Data Flows

Written by Roelant Vos on 9.7.2021

In Azure, a Mapping Data Flow itself is not an object that can be executed directly. Instead, it needs to be called from an Execute Pipeline. This pipeline can be run, and in turn it will start the data flow. BimlFlex has advanced features to manage this.

Dev Diary – Defining Mapping Data Flow Parameters with Biml

Written by Roelant Vos on 8.31.2021

BimlFlex output uses Parameters at Mapping Data Flow level to integrate with the BimlCatalog and store metadata for use inside the data flow. This post explains how to set this up in Biml.

Using built-in logging for Biml

Written by Roelant Vos on 8.26.2021

When working with Biml in any situation, be it using BimlExpress, BimlStudio or BimlFlex, it can be helpful to peek into what is happening in the background. Biml provides logging features to do so.

Dev Diary - Using Mapping Data Flow Parameters to dynamically use BimlFlex metadata

Written by Roelant Vos on 8.22.2021

Data Flow Mappings for Azure Data Factory allow a large degree of parametrization, which is important to strike a good balance between generating dynamic content and managing schema change.

Dev Diary - Deploying Biml-Generated ADF Data Flow Mappings

Written by Roelant Vos on 8.18.2021

BimlStudio can translate Biml into Data Flow Mappings, and this post looks into the deployment and results in Azure Data Factory.