Read those comments and every single reported symptom is a hard failure. Something stopped working, somebody saw an error, a ticket was raised.

Not one of them mentions performance.

That is the version of this problem I ended up with, and it took two weeks to find, because nothing failed. Refreshes completed. No errors appeared anywhere. A semantic model that had run in under thirty minutes for months was simply taking over three hours, and the only reason anyone noticed at all was that this particular extract was big enough for a four times slower read to cross the line from irritating to unusable.

If your Oracle connections survived the upgrade, this post is about the thing you might not have checked.

In one paragraph: the June 2026 gateway switched Oracle connections to a built-in managed ODP.NET driver. That driver ignores the FetchSize value in the registry and falls back to a 128 KB default. If you had tuned FetchSize, your tuning silently stopped applying on upgrade day. There is no error and nothing in the release notes. Microsoft documents both the behaviour and the fix. What nobody seems to be discussing is that this failure mode is invisible unless you go looking for it.

Power Query Oracle database connector - Power Query | Microsoft Learn

That chart is the whole argument, and the reason it's convincing is the second line. One connector changed. The others, on the same machines over the same days, did not.

Does this affect you?

All five true means your tuning is being ignored right now. Two commands to confirm:

# Reference implementation. Read-only.
$cfg = 'C:\Program Files\On-premises data gateway\Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config'

$x = [xml](Get-Content $cfg)
"Bundled provider disabled: " + (($x.SelectNodes("//setting") |
    Where-Object { $_.name -eq 'MashupFlight_DisableOracleBundledOdacProviderV2' }).value)

Get-ChildItem 'HKLM:\SOFTWARE\Oracle\ODP.NET' -ErrorAction SilentlyContinue | ForEach-Object {
    $p = Get-ItemProperty $_.PSPath
    [PSCustomObject]@{ ProviderVersion = $_.PSChildName; FetchSize = $p.FetchSize }
} | Format-Table -AutoSize

False plus a FetchSize value means yes.

One detail that makes this easy to miss: that registry path is version-keyed. A setting registered under one provider version is invisible to a different provider version. The built-in driver is a different version from anything you installed, so even before you learn it ignores the registry entirely, the version mismatch alone would have orphaned your setting. It's right there, it looks correct, and it applies to a provider that is no longer being used.

Microsoft documents this in the Oracle database connector article, under "FetchSize in the registry isn't used by the built-in driver". The documentation is accurate and the fix is the same one-line config change that fixes the connection failures.

The gap isn't the fix. The gap is knowing you need it. Getting from "a refresh is slow" to that documentation page took two weeks, and here are the five things that cost the most time.

1. The string that changed is not the thing that changed

The first theory came from the customer's DBA and it was a good one. Their audit log showed the USERHOST value for the connection had changed on the exact date the trouble started, from one Active Directory domain prefix to another. Conclusion: the gateway server had been moved to a different datacenter.

It hadn't. USERHOST is a client-supplied string. The Oracle client on Windows composes it from the logon domain of the process account, not from the machine's domain and certainly not from its location. The gateway service restarted as part of the upgrade, the new process got a fresh logon token, and the string changed accordingly. Same rack, same subnet, same AD site.

Anything that restarts changes a surprising number of reported values. If your investigation starts from "this value changed on the same day", check whether it's derived from process state before building a theory on it.

2. On a shared gateway, the errors in the log are not your errors

The logs contained plenty of Oracle errors. Duplicate transaction identifiers over a database link. Cancelled operations. Listener errors for service names that no longer existed. Every one looked like a lead.

None were related. All predated the upgrade by weeks. On a busy gateway environment, the log is mostly other people's problems.

The structural difficulty is that QueryExecutionReport keys on the gateway object, not on individual datasets. There's no clean join from "this error" to "that semantic model". What you do have is the DataSource column, which holds the connection path, so you can at least filter to the source you care about.

Before you investigate an error, establish that it's yours. Filter to your data source, then check whether its error rate actually changed on the date in question. Here, the most promising-looking errors had been running at a steady rate for months.

3. The metric that shows it, and the control group that proves it

Every gateway writes report files into the service account's profile:

C:\Users\<gateway service account>\AppData\Local\Microsoft\On-premises data gateway\report\

QueryExecutionReport_*.log is the one that matters. CSV, roughly 100 MB per file, retention measured in months. The schema is documented in Monitor and optimize gateway performance.

Two columns do the work:

Divide one by the other for an effective read rate per query. Daily median, split by source kind, plotted over time. That's the chart at the top of this post.

One warning on field selection. There's a DataReadingDuration (ms) column in the files that isn't in the published schema. It tracks the documented field almost exactly, because serialization is under two seconds on typical queries. Use the documented one anyway. Hand a vendor numbers computed from undocumented fields and the first reply you get will be about your methodology rather than your problem.

Then the part that matters more than the metric itself.

A number that changed on a date is not evidence. A number that changed while a comparable number didn't is.

Same gateway, same days, same calculation, different source kind. Oracle dropped by roughly a factor of four. SQL Server didn't move, and neither did MySQL, Snowflake or SharePoint.

That single comparison eliminates, in one step: the network, the servers, storage, memory pressure, time of day, seasonal load, and anything else affecting the machine as a whole. Whatever changed, it changed for one connector.

Before reporting a regression to anyone, find something that should not have regressed and confirm it didn't.

4. A fleet median can be flat while your largest workloads are dying

The obvious next move was to confirm the throughput drop in refresh durations. Power BI records every refresh, so this should have been free corroboration.

It wasn't. Median refresh duration for Oracle-backed datasets before and after gave a ratio of about 1.04. Essentially unchanged. The non-Oracle control gave 1.00. For an uncomfortable hour that looked like the throughput measurement was wrong.

It wasn't wrong. It was measuring something real that mostly didn't matter. If a model reads 20 MB, a four times slower read adds a few seconds to a refresh dominated by other work. If a model reads several hundred megabytes, the same regression turns a quarter of an hour of reading into most of an afternoon.

Banding datasets by baseline duration made it visible immediately: the longest-running Oracle models had roughly doubled, everything under five minutes was flat.

If your monitoring reports averages across all datasets, this entire class of problem is invisible to you. You hear about it from the one team whose extract is big enough to notice.

This is also, I suspect, why nobody reported it publicly. The people who upgraded and lost their connections knew within hours. The people who upgraded and lost three quarters of their Oracle read throughput had nothing to see.

One sampling trap worth naming, because it produces a confident and meaningless number. The refresh history API returns the last N refreshes per dataset, not a time window. An older comparison window is therefore biased toward datasets that refresh infrequently, because frequent refreshers have already aged out of it. Use a symmetric window either side of your change date, and require a minimum number of runs on both sides.

5. The config edit does nothing until the service restarts

This one is embarrassing.

Having found the documented fix, I applied it to one node and watched. Nothing moved. Not that day, not the next, not the day after. For three days it looked like the documented remedy simply didn't work, and I nearly wrote that back to Microsoft.

The setting was in the file. The running process was still using the value it read at startup. The service hadn't been restarted since the edit. When it eventually was, for an unrelated reason, the read rate returned to its previous level within a day and stayed there.

Verify from the running process, not from the file. If the process is older than your change, your change is not in effect, whatever the file says.

Get-CimInstance Win32_Process -Filter "Name='Microsoft.PowerBI.EnterpriseGateway.exe'" |
    Select-Object ProcessId, CreationDate     # must be later than your edit

Putting the pieces together

The whole measurement in one place. Reference implementation, not a drop-in solution. It reads the gateway's own report files, works out your upgrade date from the registry, and produces a daily median read rate per source kind. It changes nothing.

# =============================================================================
# Gateway read throughput, daily median by source kind, before vs after upgrade
# Read-only. No configuration is touched.
# =============================================================================

param(
    [string]  $ReportDir,
    [datetime]$ChangeDate,       # leave empty to derive from the gateway install date
    [int]     $MinSpoolMB = 5    # ignore small queries: metadata calls swamp the stats
)

# --- when was this gateway last upgraded -------------------------------------
if (-not $ChangeDate) {
    $gw = Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*',
                           'HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*' -ErrorAction SilentlyContinue |
          Where-Object { $_.DisplayName -eq 'GatewayComponents' } | Select-Object -First 1
    if (-not $gw.InstallDate) { throw 'Could not determine upgrade date. Pass -ChangeDate explicitly.' }
    $ChangeDate = [datetime]::ParseExact($gw.InstallDate, 'yyyyMMdd', $null)
    Write-Host "Gateway components $($gw.DisplayVersion), installed $($ChangeDate.ToString('yyyy-MM-dd'))"
}
$cut = $ChangeDate.ToString('yyyy-MM-dd')

# --- find the report directory -----------------------------------------------
# It lives under the profile of the account the gateway service runs as,
# which is usually not the account you are logged in with.
if (-not $ReportDir) {
    $svc  = Get-CimInstance Win32_Service -Filter "Name='PBIEgwService'"
    $acct = ($svc.StartName -split '[@\\]')[0]
    $prof = Get-ChildItem 'C:\Users' -Directory |
            Where-Object { $_.Name -like "*$acct*" } | Select-Object -First 1
    $ReportDir = Join-Path $prof.FullName 'AppData\Local\Microsoft\On-premises data gateway\report'
}

$files = Get-ChildItem "$ReportDir\QueryExecutionReport_*.log" | Sort-Object Name
if (-not $files) { throw "No query execution reports found in $ReportDir" }

$hdr      = (Get-Content $files[0].FullName -TotalCount 1) -split ','
$minBytes = $MinSpoolMB * 1MB
$rows     = New-Object System.Collections.Generic.List[object]

foreach ($f in $files) {
    Write-Host "parsing $($f.Name)" -ForegroundColor DarkGray

    # Stream it. These files are ~100 MB each and you do not want them in memory.
    $reader = [System.IO.StreamReader]::new($f.FullName)
    $null = $reader.ReadLine()                        # header

    while ($null -ne ($line = $reader.ReadLine())) {

        # Cheap prefilter: a line with no large number cannot pass the size test.
        if ($line -notmatch '\d{7,}') { continue }

        try {
            $r     = $line | ConvertFrom-Csv -Header $hdr
            $spool = [double]($r.'SpoolingTotalDataSize(byte)' -replace '^$','0')
            if ($spool -lt $minBytes) { continue }

            $dur = [double]($r.'DataReadingAndSerializationDuration(ms)' -replace '^$','0')
            if ($dur -le 0) { continue }
            if ($r.QueryExecutionEndTimeUTC -notmatch '^\d{4}') { continue }

            # DataSource holds escaped JSON. After ConvertFrom-Csv has stripped one
            # layer of quoting it reads:
            #   ["{\"kind\":\"Oracle\",\"path\":\"host:1521/service\"}"]
            # so SINGLE backslashes here. See the note below.
            $kind = ''
            $m = [regex]::Match($r.DataSource, 'kind\\":\\"([^\\]+)')
            if ($m.Success) { $kind = $m.Groups[1].Value }

            $rows.Add([PSCustomObject]@{
                Day      = $r.QueryExecutionEndTimeUTC.Substring(0,10)
                Kind     = $kind
                SpoolMB  = [math]::Round($spool / 1MB, 2)
                ReadSec  = [math]::Round($dur / 1000, 1)
                ExecSec  = [math]::Round([double]($r.'QueryExecutionDuration(ms)' -replace '^$','0') / 1000, 1)
                MBperSec = [math]::Round(($spool / 1MB) / ($dur / 1000), 3)
            })
        } catch { }
    }
    $reader.Close()
}

function Get-Median([object[]]$v) {
    if (-not $v -or $v.Count -eq 0) { return 0 }
    $s = $v | Sort-Object
    if ($s.Count % 2) { $s[[int](($s.Count - 1) / 2)] }
    else { ($s[$s.Count/2 - 1] + $s[$s.Count/2]) / 2 }
}

# --- daily trend: your suspect connector plus its control group ---------------
$rows | Group-Object Day, Kind | ForEach-Object {
    $p = $_.Name -split ', '
    [PSCustomObject]@{
        Day        = $p[0]
        Kind       = $p[1]
        Queries    = $_.Count
        MedMBps    = [math]::Round((Get-Median ($_.Group.MBperSec)), 3)
        AvgSpoolMB = [math]::Round(($_.Group | Measure-Object SpoolMB -Average).Average, 1)
        AvgExecSec = [math]::Round(($_.Group | Measure-Object ExecSec -Average).Average, 1)
    }
} | Sort-Object Day, Kind | Format-Table -AutoSize

# --- one-line verdict per source kind ----------------------------------------
$rows | Group-Object Kind | ForEach-Object {
    $b = Get-Median (($_.Group | Where-Object { $_.Day -lt  $cut }).MBperSec)
    $a = Get-Median (($_.Group | Where-Object { $_.Day -ge $cut }).MBperSec)
    [PSCustomObject]@{
        Kind   = $_.Name
        Before = [math]::Round($b, 3)
        After  = [math]::Round($a, 3)
        Ratio  = if ($b -gt 0) { [math]::Round($a / $b, 2) } else { $null }
    }
} | Sort-Object Ratio | Format-Table -AutoSize

Notes on the shape of it.

Median, not mean. One query with a read time of a few milliseconds produces an absurd rate that drags the average around. The median held to within a few percent day to day for three months, which is what made a step change obvious rather than arguable.

The 5 MB floor is doing real work. Without it, thousands of sub-second metadata and preview queries dominate the counts and the signal disappears.

AvgSpoolMB is your sanity check. It should be roughly flat across your change date. If average payload per query grew, you have a data volume story rather than a driver story.

AvgExecSec is what lets you release the DBA. That's QueryExecutionDuration, the database executing the query. If execution time is flat and read time has grown, the database is doing its job at the same speed and the problem is in getting results back. Demonstrating that saved several days of database-side investigation that would have found nothing.

The escaped JSON will cost you an hour if you let it. How many backslashes you need depends on whether you've parsed the CSV yet:

# Raw file text, before parsing: backslashes are DOUBLED
Select-String -Path $file -Pattern 'kind\\"":\\""Oracle'

# After ConvertFrom-Csv: one layer stripped, SINGLE backslashes
$parsed.DataSource -match 'kind\\":\\"Oracle'

Wrong in either direction gives zero rows and no error, which looks exactly like "there is no Oracle traffic on this gateway". I lost time to this twice, in both directions. If you'd rather not think about it, match the plain word and filter after parsing.

The fix

Opting out of the built-in driver returns you to the ODP.NET provider from your installed Oracle client, which does honour the registry FetchSize. Microsoft documents the procedure: back up the config, set the flag to True, restart the gateway.

# Reference implementation. This CHANGES CONFIGURATION.
# Read the documentation first. Requires administrative rights.

$cfg = 'C:\Program Files\On-premises data gateway\Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config'

Copy-Item $cfg "$cfg.bak-$(Get-Date -f yyyyMMdd-HHmmss)" -Force   # back up first, always

Stop-Service PBIEgwService -Force        # the running process holds the old value
Start-Sleep -Seconds 10

$x = [xml](Get-Content $cfg)             # edit as XML, not text, so you cannot mangle it
$node = $x.SelectNodes("//setting") |
        Where-Object { $_.name -eq 'MashupFlight_DisableOracleBundledOdacProviderV2' }
if (-not $node) { Start-Service PBIEgwService; throw 'Setting not found. Aborted, nothing changed.' }
$node.value = 'True'
$x.Save($cfg)

Start-Service PBIEgwService
Start-Sleep -Seconds 30

$v = [xml](Get-Content $cfg)
"Flag on disk: " + (($v.SelectNodes("//setting") |
    Where-Object { $_.name -eq 'MashupFlight_DisableOracleBundledOdacProviderV2' }).value)
Get-CimInstance Win32_Process -Filter "Name='Microsoft.PowerBI.EnterpriseGateway.exe'" |
    Select-Object ProcessId, CreationDate     # must be later than your edit

Two things worth knowing. This setting survives gateway upgrades, which Microsoft states explicitly and which is unusual for anything in that directory. And give it a full day before judging the result, because a single refresh tells you very little.

The loud failures, for completeness

If your Oracle connections broke outright rather than slowing down, the same flag fixes most of it, but two other causes surfaced in the community thread and neither is obvious from the error text.

TNS_ADMIN must be set at machine scope. The gateway service account cannot see user-scoped environment variables, so a TNS_ADMIN set for your own login resolves nothing for the service. Set it at system scope, point it at the directory containing tnsnames.ora, restart the service. Several people hit ORA-12154 for this reason, and the error is misleading: the driver was not failing to find the alias, it was failing to find the file.

Some previously ignored TNS parameters are now rejected. UR=A in a tnsnames.ora connect descriptor is the documented example. Older clients ignored it, the built-in managed driver rejects it with ORA-50122: The following configuration parameters are not allowed to be set. If a connection descriptor has accumulated optional parameters over the years, that error names which one is the problem. Removing it is usually safe, but check why it was there first.

Two other documented differences are worth knowing even if nothing is failing, because neither necessarily errors:

• NLS_LANG is ignored. The built-in provider takes language, territory, and number and date formats from the machine's .NET culture instead. Depending on regional settings that can change how values parse. Sometimes it errors. Sometimes it returns quietly different numbers, which is worse.

• Oracle proxy-user authentication is not supported. Connections in the User Id=proxy_user[target_schema] form fail.

The Oracle specifics here will be fixed eventually and this post will become a historical curiosity. The shape underneath it won't be.

A managed platform component changed a dependency underneath me. My tuning lived outside that component and applied to the old dependency. When the component swapped, the tuning was still present, still valid-looking, and doing nothing at all. There is no error state for "this setting is being ignored", and no screen anywhere would have told me.

What strikes me most, having read every comment on that release announcement, is the selection effect. The failures got reported within a day because failure is loud. The silent version of the same change, on the same date, from the same root cause, appears nowhere. That does not mean nobody has it. It means the only people who find it are the ones who were already measuring, or the ones with a workload big enough to make four times slower into a business problem.

The only thing that found it here was a ratio, computed the same way before and after, with a control group next to it.

If you're doing gateway dependency work more generally, the dependency map from my previous post is what turns "one team complained" into "here is every artifact on that connection and who owns it".

If you upgraded in June and your Oracle connections kept working, I'd be curious whether anyone has actually checked their read throughput since. And if anyone has found a supported way to configure fetch size for the built-in managed provider, rather than opting out of it entirely, I'd like to hear it.

You open the Fabric admin portal. You can see your gateway clusters. You can click into one and see the data sources configured on it. You can see which users have access to each one.

What you cannot see, anywhere, in any screen, is the answer to the question you were actually asked: which reports, datasets, dataflows and pipelines depend on this connection?

That inversion is the gap. Power BI shows you gateway to data source. It does not show you data source to everything that uses it. And the second direction is the one that every operational question needs.

I recently built that map for a large enterprise tenant: many thousands of workspaces, and a great many more links between artifacts and the gateway data sources they consume. This post covers why it's worth having, and the five things that cost me the most time building it. One of them meant my original numbers were substantially overstated, in a direction that flattered the estate.

Why the map is worth building

If you manage a small estate, you hold the dependencies in your head. Past a few hundred data sources you don't, and the cost of not knowing shows up in specific, expensive ways.

You're changing things blind

Every credential rotation, server migration and data source cleanup is currently a guess. You make the change, then you find out what depended on it when the complaints arrive. Your feedback loop runs through your users, which is the most expensive possible way to discover a dependency.

With the map, "what breaks if I touch this?" becomes a query that answers in a second, before you touch anything.

Incident response runs backwards

A data source goes offline at 3am. Right now the sequence is: refreshes fail silently, users notice broken reports over the following hours, tickets arrive, and eventually someone correlates them into a cause.

With a dependency map the sequence inverts. The moment a data source fails you can list every affected artifact and every owner, and send one message before the first ticket. Same outage, completely different experience for the business, and a much shorter time to communicate, which is often what people actually judge you on.

Cleanup is too risky to attempt

Most large tenants accumulate hundreds of data sources nobody can account for. Created for a proof of concept, left behind by someone who changed teams, duplicated because that was faster than finding the original.

Everyone knows they should go. Nobody removes them, because the downside of deleting something still in use vastly outweighs the benefit of tidiness. So they sit there for years, cluttering the portal and quietly holding credentials to production systems.

The map turns that from a judgement call into a fact. A data source with no dependent artifacts, no assigned users, and no successful connection in ninety days is not a risk to delete. It's a liability to keep.

That last criterion needs its own note, because the platform will not give it to you. The API can tell you whether a data source is reachable right now, but it keeps no history. And "unreachable right now" is close to useless for cleanup decisions, because a source that failed this morning during a database restart looks identical to one that has been dead since last year.

To tell those apart you have to build the history yourself: snapshot the status of every data source on a schedule, write it to a table with a timestamp, and let it accumulate. After a few months you can ask the question that actually matters, which is whether this connection has succeeded even once in the last ninety days. That's a very different and much safer question than whether it happens to be up at this moment.

Decommissioning and migration are unplannable

Server retirements, gateway consolidation, moves to VNet gateways, Fabric migrations. All of them start with "what's affected, and who owns it?" Without the map, the honest answer is an email to all report owners asking them to check, which produces a partial response and a lingering suspicion that something was missed. Usually because it was.

Nobody can audit access to on-prem systems through BI

Your gateway data sources hold credentials to production databases and file shares. The chain from "this person can view a report" back through which dataset, on which gateway, using which service account, against which server, is not visible in any single place. For anyone asking compliance questions, the map is the only way to answer honestly.

What the map actually is

One flat table, one row per artifact-to-datasource relationship:

That's the whole thing. Every question above becomes a WHERE clause against this table. The value isn't in the schema, it's in the fact that no UI can produce this table, and the API that can has a few behaviours that will cost you time.

Here are the five that cost me the most.

A note on the code below: the Scanner API calls are real, but the flags, helper functions and table names are my own. There is no built-in resume switch or safety threshold. Those are patterns you implement yourself, and I've named them here the way I named them in my own notebook.

1. The parameter combination that returns nothing, silently

If you only care about gateway relationships, the obvious optimisation is to narrow the scan. Skip dataset schemas, skip expressions, ask only for what you need. Less data over the wire, faster runs, smaller payloads.

It works. It returns quickly. And datasourceInstances and datasourceUsages come back empty.

Datasource details are only populated when lineage is enabled. Turn lineage off and you get a fast, clean, entirely useless result, with no error and no warning that you've asked for something the API won't give you in that configuration.

# The combination that actually returns datasource details
scan_workspaces(
    workspaces=batch,
    lineage=True,               # REQUIRED. Without this, no datasourceInstances
    datasource_details=True,
    dataset_schema=False,       # safe to skip, large payload reduction
    dataset_expressions=False,  # safe to skip
)

The failure mode is what makes this expensive. An empty array looks exactly like "this tenant has no gateway-bound artifacts", so I spent a long time questioning my transformation logic before I thought to question the request.

If you want datasource information, lineage=True is not optional, no matter how narrow you think your scan is.

2. Stage the raw response before you transform it

Scanner responses for a large tenant get big enough to cause problems if you try to hold everything in memory and write it in one pass.

The fix is to write the raw scan output to storage first, and transform from there as a separate step. write_jsonl below is just a stand-in for whatever you use to append text files to your lakehouse:

# Phase 1: scan, and immediately persist the raw result
for batch in workspace_batches:
    result = scan_workspaces(batch, lineage=True, datasource_details=True)
    write_jsonl(f"Files/scan_staging/scan_{batch_id}.jsonl", result)

# Phase 2: transform from staging, never from the API
df = spark.read.json("Files/scan_staging/*.jsonl")

The obvious benefit is memory. The real benefit is bigger.

A full tenant scan takes a long time. Your transformation logic will fail on the first several attempts, because of an unexpected null, a type that varies between artifact kinds, or a nested field you didn't anticipate. If your transform reads from the API, every one of those failures costs you another complete scan. If it reads from staging, it costs you seconds.

That single decision turned my iteration loop from an hour into a moment, and it's the habit I'd carry into any similar project on any platform. It also means you can add your own resume switch, so you can rebuild the output tables from staged files without touching the API again:

# my own flag, not an API feature
RESUME_FROM_STAGING = True   # rebuild output tables from staged JSON, zero API calls

Once that exists you can restructure your schema as often as you like, for free.

3. Flatten your structs or Delta will refuse the merge

Scanner output is deeply nested. Workspaces contain datasets, which contain datasource usages, which reference datasource instances whose connection details vary by type.

Write that structure directly to Delta across multiple batches and you'll eventually hit a schema merge failure. Different batches infer subtly different schemas, because a field that's absent in one batch appears in another, and nested struct types don't reconcile cleanly.

Flatten aggressively to scalar columns, and keep the original JSON alongside:

df_flat = df.select(
    F.col("id").alias("workspace_id"),
    F.col("name").alias("workspace_name"),
    F.col("datasource.datasourceId").alias("datasource_id"),
    F.col("datasource.gatewayId").alias("gateway_id"),
    F.col("datasource.datasourceType").alias("datasource_type"),
    # keep the full structure for anything you didn't anticipate
    F.to_json(F.col("datasource.connectionDetails")).alias("connection_details"),
)

A flat table of scalars plus a JSON column merges cleanly every time, stays queryable from SQL, and loses nothing. When you eventually need a field you didn't flatten, parse it out of the JSON rather than rebuilding your schema.

4. Put a guard between the scan and the load

This is the failure mode that should worry you most, because it produces no error at all.

Your scan runs on a schedule and feeds a table that reporting and cleanup processes depend on. One night it half fails, from throttling or a token expiry or a transient error, and returns a small fraction of your workspaces. Your load step truncates the target table and writes the partial result.

Nothing errored. Every downstream process now believes most of your estate doesn't exist. If anything automated acts on that data, for example a cleanup job that deletes data sources with no dependencies, the consequences escalate quickly.

There is no built-in protection against this. You have to add it yourself, and it costs about three lines:

# my own guard. Set these well below your true numbers,
# but high enough to catch a badly broken run.
MIN_EXPECTED_WORKSPACES = <your_baseline>
MIN_EXPECTED_LINKS      = <your_baseline>

if workspace_count < MIN_EXPECTED_WORKSPACES:
    raise Exception(
        f"Only {workspace_count} workspaces scanned (expected >= {MIN_EXPECTED_WORKSPACES}). "
        f"Aborting before load, refusing to overwrite good data with a partial scan."
    )

Fail loudly and leave yesterday's good data in place. A missed refresh is an inconvenience. A silently truncated inventory that automated processes then act on is an incident.

The principle generalises well beyond this: any pipeline that replaces rather than appends needs a sanity check on what it's about to destroy.

5. "Bound to a gateway" does not mean "on-premises"

This is the finding that changed how I read every number the scan produced.

The intuitive reading of gatewayId IS NOT NULL is "this connection goes through an on-premises data gateway". It doesn't. Power BI assigns a gatewayId to cloud connections as well, including SharePoint, web sources, cloud-hosted databases and Databricks.

So a raw count of gateway-bound artifacts substantially overstates your on-premises footprint, in my case by a wide margin.

To get a truthful number you have to join back to your own gateway inventory and separate real clusters from cloud connections. The table names below are mine, so substitute your own:

SELECT COUNT(*) AS on_prem_links
FROM   fabric_gateway_usage f
JOIN   gateway_inventory     g ON g.GatewayID = f.gateway_id
WHERE  g.GatewayName IS NOT NULL
  AND  LTRIM(RTRIM(g.GatewayName)) <> '';   -- blank name means a cloud connection

This matters more than it first appears. If you're presenting dependency figures to anyone making decisions about gateway consolidation, licensing, infrastructure investment or migration scope, an inflated count leads to a materially different conversation than the real one. Halving a headline number after someone has already planned around it is a bad look, and an easy one to avoid.

It's worth checking whether any dashboard you already publish is quietly making this mistake.

Putting it to work

With a complete, trustworthy map in a queryable table, the questions that were previously unanswerable become one-liners.

What breaks if I remove this data source?

SELECT artifact_type, workspace_name, item_name
FROM   fabric_gateway_usage
WHERE  datasource_id = @datasource_id;

Which data sources can I safely delete? The ones with no dependent artifacts at all, and if you've been snapshotting status over time as described earlier, the ones that haven't connected successfully in months either.

Which gateways actually carry production load? Count distinct artifacts per gateway, rather than guessing from the name.

Who do I notify about this outage? Join the affected artifacts to their workspace owners, and send one message before the tickets start.

Putting the pieces together

The sections above are fragments, so here is the whole flow in one place. This is a reference implementation, not a drop-in solution: paths, batch sizes and thresholds depend on your environment, and you'll want to adapt the flattening to the artifact types you care about.

It runs as a single Fabric notebook with a lakehouse attached. One prerequisite that isn't obvious: %pip install is disabled when a notebook runs inside a pipeline, so attach a Fabric Environment with semantic-link-labs as a library instead. Installing only semantic-link-sempy is not enough, because the admin scan functions live in the labs package.

# =============================================================================
# Tenant-wide gateway dependency map
# Scans all workspaces, stages raw JSON, flattens to a queryable table.
# =============================================================================

from pyspark.sql import functions as F
from pyspark.sql.types import StructType
import json, uuid, datetime

# ----- configuration ---------------------------------------------------------
BATCH_SIZE          = 90        # workspaces per scan call
STAGING_PATH        = "Files/scan_staging"
TARGET_TABLE        = "fabric_gateway_usage"

RESUME_FROM_STAGING = False     # True = rebuild tables from staged JSON, no API calls
KEEP_HISTORY        = True      # False = keep only the latest run

# safety guards. Set well below your true numbers, high enough to catch a bad run.
MIN_EXPECTED_WORKSPACES = 0     # e.g. 80% of your normal workspace count
MIN_EXPECTED_LINKS      = 0     # e.g. 80% of your normal link count

SCAN_RUN_ID = str(uuid.uuid4())
SCAN_DATE   = datetime.datetime.utcnow()


# ----- phase 1: scan and stage ----------------------------------------------
def run_scan():
    """Scan every workspace and write raw responses to the lakehouse."""
    import sempy_labs.admin as admin          # lazy import: resume runs skip this

    workspaces = admin.list_workspaces()
    ws_ids     = [w for w in workspaces["Id"]]
    print(f"{len(ws_ids)} workspaces to scan")

    for i in range(0, len(ws_ids), BATCH_SIZE):
        batch    = ws_ids[i:i + BATCH_SIZE]
        batch_no = i // BATCH_SIZE

        result = admin.scan_workspaces(
            workspace=batch,
            lineage=True,               # REQUIRED for datasource details
            datasource_details=True,
            dataset_schema=False,
            dataset_expressions=False,
        )

        # stage immediately: a later failure must not cost another scan
        path = f"/lakehouse/default/{STAGING_PATH}/scan_{batch_no:05d}.json"
        with open(path, "w", encoding="utf-8") as f:
            json.dump(result, f)

        print(f"  batch {batch_no}: {len(batch)} workspaces staged")


# ----- phase 2: flatten ------------------------------------------------------
def build_links():
    """Read staged JSON and flatten to one row per artifact-to-datasource link."""
    raw = spark.read.option("multiline", "true").json(f"{STAGING_PATH}/*.json")

    # datasource instances: id -> type + connection details
    instances = (
        raw.select(F.explode("datasourceInstances").alias("d"))
           .select(
               F.col("d.datasourceId").alias("datasource_id"),
               F.col("d.datasourceType").alias("datasource_type"),
               F.col("d.gatewayId").alias("gateway_id"),
               # keep the raw structure: connection shape varies by type
               F.to_json(F.col("d.connectionDetails")).alias("connection_details"),
           )
           .dropDuplicates(["datasource_id"])
    )

    frames = []

    # each artifact type sits in its own array on the workspace object
    for artifact_type, array_col in [
        ("dataset",          "datasets"),
        ("dataflow",         "dataflows"),
        ("datapipeline",     "datapipelines"),
        ("lakehouse",        "lakehouses"),
        ("notebook",         "notebooks"),
    ]:
        if array_col not in raw.select("workspaces.*").columns:
            continue

        frames.append(
            raw.select(F.explode("workspaces").alias("w"))
               .select("w.id", "w.name", F.explode(f"w.{array_col}").alias("item"))
               .select(
                   F.lit(artifact_type).alias("artifact_type"),
                   F.col("id").alias("workspace_id"),
                   F.col("name").alias("workspace_name"),
                   F.col("item.id").alias("item_id"),
                   F.col("item.name").alias("item_name"),
                   F.explode("item.datasourceUsages").alias("usage"),
               )
               .select("*", F.col("usage.datasourceInstanceId").alias("datasource_id"))
               .drop("usage")
        )

    if not frames:
        raise Exception("No artifacts found in staged data. Check lineage=True.")

    links = frames[0]
    for f_ in frames[1:]:
        links = links.unionByName(f_, allowMissingColumns=True)

    return (
        links.join(instances, on="datasource_id", how="left")
             .withColumn("scan_date",   F.lit(SCAN_DATE))
             .withColumn("scan_run_id", F.lit(SCAN_RUN_ID))
    )


# ----- phase 3: guard, then load --------------------------------------------
def load(links):
    """Refuse to publish a partial scan over good data."""
    ws_count   = links.select("workspace_id").distinct().count()
    link_count = links.count()
    print(f"{ws_count} workspaces, {link_count} links")

    if ws_count < MIN_EXPECTED_WORKSPACES:
        raise Exception(
            f"Only {ws_count} workspaces (expected >= {MIN_EXPECTED_WORKSPACES}). "
            f"Aborting: refusing to overwrite good data with a partial scan."
        )
    if link_count < MIN_EXPECTED_LINKS:
        raise Exception(
            f"Only {link_count} links (expected >= {MIN_EXPECTED_LINKS}). Aborting."
        )

    mode = "append" if KEEP_HISTORY else "overwrite"
    (links.write
          .format("delta")
          .mode(mode)
          .option("mergeSchema", "true")
          .saveAsTable(TARGET_TABLE))

    # a view exposing only the current run, for downstream consumers
    spark.sql(f"""
        CREATE OR REPLACE VIEW {TARGET_TABLE}_latest AS
        SELECT * FROM {TARGET_TABLE}
        WHERE  scan_run_id = (
            SELECT scan_run_id FROM {TARGET_TABLE}
            ORDER BY scan_date DESC LIMIT 1
        )
    """)
    print(f"Loaded into {TARGET_TABLE} (mode={mode})")


# ----- run -------------------------------------------------------------------
if not RESUME_FROM_STAGING:
    run_scan()

load(build_links())

A few notes on the shape of it.

The scan and the transform are separate functions on purpose. That's what makes RESUME_FROM_STAGING possible, and it's the difference between a one minute iteration loop and an hour one.

The import of sempy_labs.admin sits inside run_scan. On a resume run you never touch the API, so there's no reason to pay the import cost.

The guards run before the write, not after. That ordering is the entire point: yesterday's complete data survives a bad scan today.

The _latest view exists because the table accumulates. With KEEP_HISTORY = True you get every run, which is useful for tracking how dependencies change over time. Downstream consumers that only want the current picture read the view instead of the table.

Adapt the artifact type list to what you care about. Scanner output includes more types than the five above, and the exact set shifts as Fabric adds item types, so treat that loop as the place you'll be editing most often.

None of this is exotic once the map exists. Building it correctly is the hard part, and most of that difficulty turned out not to be volume or performance. It was five behaviours that no documentation warned me about, one of which meant my numbers were wrong in a direction that made the estate look better than it was.

If you're running Scanner API scans at scale I'd be interested in what you've hit, particularly throttling behaviour on very large tenants, and whether anyone has found a cleaner way to distinguish cloud from on-prem connections than joining back to the gateway inventory.