How to create a metadata-driven Azure Data Factory Pipeline

When we want to copy huge amounts of objects (for example, thousands of tables) or load data from variety of sources, the appropriate approach is to input the name list of the objects with required copy behaviors in a control table, and then use parameterized pipelines to read the same from the control table and… Continue reading How to create a metadata-driven Azure Data Factory Pipeline

Merge Data into a Single Table using Azure Data Factory

I recently came across a scenario where the source files in Azure blob storage container had slightly different schemas, i.e., one set of files had an extra column compared to another set of files. The challenge was to identify this diverse set of files and load it into a single target table on the fly.… Continue reading Merge Data into a Single Table using Azure Data Factory