How to find and remove unused Azure Data Factory Pipelines

Why You Should Clean Up Unused Azure Data Factory Pipelines

Unused Azure Data Factory (ADF) resources may not incur direct compute costs, but they still contribute to overhead. Keeping them around can:

Introduce security risks
Increase log and monitoring costs
Consume Integration Runtime (IR) resources
Cause noise from test or legacy pipelines
Lead to confusion in environments with many deployments

More broadly, retaining unused infrastructure leads to higher maintenance effort and environmental waste. Cleaning up unused ADF pipelines is a good FinOps practice and aligns with sustainable cloud management.

What Counts as an Unused Azure Data Factory Pipeline?
PowerShell Script to Detect Inactive Azure Data Factory Pipelines
Conclusion

What Counts as an Unused Azure Data Factory Pipeline?

A pipeline is considered unused when it meets either of the following criteria:

It hasn’t run in the past 30 days (or a set period)
It was deployed but never triggered or executed

This check helps surface pipelines that are inactive or potentially forgotten.

PowerShell Script to Detect Inactive Azure Data Factory Pipelines

Below is a PowerShell script that checks all your Azure subscriptions and lists Data Factory instances with no pipeline runs in the last 30 days. It’s a solid starting point to identify ADFs that may be ready for clean-up.

Script Overview: Check for Unused Pipelines in Azure Data Factory

This script:

Logs into Azure if needed
Loops through all subscriptions
Retrieves all ADF instances
Checks for pipeline activity over the past 30 days
Outputs the ADFs with no recent runs

Sample Output

Her’s what the final result may look like:

JSON

Subscription       ResourceGroup       DataFactoryName      Last30DaysRuns     Tags
-----------        --------------      ------------------   ----------------   -----------------------
ContosoProd        rg-contoso-data     dft-contoso-prod     0                  env=prod;owner=data-team
MarketingDev       rg-marketing-test   dft-marketing-dev    0                  env=dev;owner=marketer

Subscription       ResourceGroup       DataFactoryName      Last30DaysRuns     Tags
-----------        --------------      ------------------   ----------------   -----------------------
ContosoProd        rg-contoso-data     dft-contoso-prod     0                  env=prod;owner=data-team
MarketingDev       rg-marketing-test   dft-marketing-dev    0                  env=dev;owner=marketer

PowerShell Script: Find Unused Azure Data Factory Pipelines

Bash

# Log in to Azure if needed
Connect-AzAccount
# Define time window (past 30 days)
$startTime = (Get-Date).AddDays(-30).ToString("o")
$endTime = (Get-Date).ToString("o")
# Store unused pipelines
$results = @()
# Get all subscriptions
$subscriptions = Get-AzSubscription
foreach ($sub in $subscriptions) {
    Write-Warning "Scanning subscription: $($sub.Name)"
    Set-AzContext -SubscriptionId $sub.Id
    # Get all Data Factory instances
    $factories = Get-AzDataFactoryV2
    foreach ($factory in $factories) {
        try {
            # Get recent pipeline runs
            $runs = Get-AzDataFactoryV2PipelineRun `
                -ResourceGroupName $factory.ResourceGroupName `
                -DataFactoryName $factory.DataFactoryName `
                -LastUpdatedAfter $startTime `
                -LastUpdatedBefore $endTime `
                -ErrorAction Stop
            # Add to results if no runs found
            if ($runs.Count -eq 0) {
                $results += [PSCustomObject]@{
                    Subscription    = $sub.Name
                    ResourceGroup   = $factory.ResourceGroupName
                    DataFactoryName = $factory.DataFactoryName
                    Last30DaysRuns  = 0
                    Tags            = ($factory.Tags.GetEnumerator() | ForEach-Object { "$($_.Key)=$($_.Value)" }) -join "; "
                }
            }
        } catch {
            Write-Warning "Failed to query $($factory.DataFactoryName): $_"
        }
    }
}
# Output unused Data Factories as a table
$results | Sort-Object Subscription, ResourceGroup | Format-Table -AutoSize

# Log in to Azure if needed
Connect-AzAccount

# Define time window (past 30 days)
$startTime = (Get-Date).AddDays(-30).ToString("o")
$endTime = (Get-Date).ToString("o")

# Store unused pipelines
$results = @()

# Get all subscriptions
$subscriptions = Get-AzSubscription

foreach ($sub in $subscriptions) {
    Write-Warning "Scanning subscription: $($sub.Name)"
    Set-AzContext -SubscriptionId $sub.Id

    # Get all Data Factory instances
    $factories = Get-AzDataFactoryV2

    foreach ($factory in $factories) {
        try {
            # Get recent pipeline runs
            $runs = Get-AzDataFactoryV2PipelineRun `
                -ResourceGroupName $factory.ResourceGroupName `
                -DataFactoryName $factory.DataFactoryName `
                -LastUpdatedAfter $startTime `
                -LastUpdatedBefore $endTime `
                -ErrorAction Stop

            # Add to results if no runs found
            if ($runs.Count -eq 0) {
                $results += [PSCustomObject]@{
                    Subscription    = $sub.Name
                    ResourceGroup   = $factory.ResourceGroupName
                    DataFactoryName = $factory.DataFactoryName
                    Last30DaysRuns  = 0
                    Tags            = ($factory.Tags.GetEnumerator() | ForEach-Object { "$($_.Key)=$($_.Value)" }) -join "; "
                }
            }
        } catch {
            Write-Warning "Failed to query $($factory.DataFactoryName): $_"
        }
    }
}

# Output unused Data Factories as a table
$results | Sort-Object Subscription, ResourceGroup | Format-Table -AutoSize

Conclusion

Using PowerShell to find and remove unused Azure Data Factory pipelines helps reduce clutter, lower operational risk, and support cloud cost optimisation. It’s a small but valuable step in a larger FinOps strategy.