ETL Framework In Production

Uma 466

Hello,

I am leveraging Python within SQL Server Integration Services (SSIS) packages, primarily through the Execute Process Task.

I am extracting Data from E2 SQL to Azure SQL Server.

The connection has already been established.

I am building complete ETL using Function

Extraction of Data (Full load/Incremental Load)
Transformation
Logging
Audit Table
Loading

The entire execution will be drive using Driver table or Configuration table.

May someone please help me how can we achieve this parameter based ETL framework.

do I need to create any config table or any metadata table how to pass value like .

input parameter
return value on output parameter
What action function is performing

Please suggest if any production grade ETL Framework, which helps to achieve this.

Appreciate your help.

Thanks a lot

Akhil Gajavelly 645 Reputation points Microsoft External Staff Moderator

2025-10-31T11:36:07.2166667+00:00
Hi @Uma,

Good question, since you’re using Python within SSIS, this setup belongs to the SQL Server → Integration Services (SSIS) category.

To build a parameter-driven ETL framework:

Create a Config/Driver table to store task name, source, target, and load type (Full/Incremental).

Use SSIS variables to read those parameters and pass them to your Python script.

Capture audit info (start time, end time, row count, status) in a log table for monitoring.

You can then control the full ETL flow dynamically from that driver table.

Thanks,
Akhil.

Your answer