The Varigence Blog

The Top Five Challenges of Data Lakes and how to overcome them with a Data Lakehouse

Written by Peter Avenant on 2.23.2023

BimlFlex's next version will include Databricks automation support, extending to all data pipeline layers, and has received positive feedback from customers. This development can save time and costs, improve data quality, and provide robust data insights for businesses. An upcoming blog post will provide instructions on configuring and implementing a fully automated Databricks solution.

Automating Change Data Capture on Azure Data Factory: Streamline Your Data Integration with BimlFlex

Written by Peter Avenant on 2.15.2023

Discover how BimlFlex, a metadata-driven framework, can help you automate Change Data Capture (CDC) on Azure Data Factory (ADF), creating efficient, scalable, and reusable solutions with significantly less manual effort. Learn how to overcome common CDC implementation challenges, including full load or initial load and restarting CDC, and improve the overall efficiency of your data integration process.

Combining DBT with BimlFlex

Written by Roelant Vos on 8.21.2022

During our travels, we are regularly asked how BimlFlex compares against DBT. Our BimlFlex data solution automation framework would be compared against all our competitors, of course, but DBT is the odd one out in this case. It is worth covering this in detail. This is because BimlFlex and DBT are not really competing. In fact, they can complement each other.

Dev Diary - Initial Extension Points added to Mapping Data Flows

Written by Roelant Vos on 5.26.2022

As part of ongoing improvements for our Mapping Data Flows feature, we are adding the first (of many) extension points.

Dev Diary - Embracing the new Azure Data Factory Script Activity

Written by Roelant Vos on 3.16.2022

Varigence has moved fast to adopt the new Script Activity feature in Azure Data Factory.

Dev Diary - Pushing down data extraction logic to the operational system environment

Written by Roelant Vos on 3.6.2022

For the upcoming 2022 R2 release, we have added a pushdown extraction feature which in some scenarios optimizes the integration of data into the data solution.