Labsco
microsoft logo

pipeline-migration

✓ Official729

by microsoft · part of microsoft/skills-for-fabric

Migrate Synapse Data Factory pipeline artifacts to Microsoft Fabric Data Factory. Handles: linked services → Fabric connections, dataset definitions inlined into pipeline activities, global parameters → Variable Libraries, SynapseNotebook activities → TridentNotebook. SSIS, SHIR-only, and Databricks activities are parked. Use when: (1) migrating Synapse pipelines to Fabric Data Factory, (2) converting SynapseNotebook activities to TridentNotebook, (3) translating linked services to Fabric connec

🧩 One of 7 skills in the microsoft/skills-for-fabric package — works on its own, and pairs well with its siblings.

This is the playbook your agent receives when the skill activates — you don't need to read it to use the skill, but it's here to audit before installing.

Update Check — ONCE PER SESSION (mandatory for live migration runs) The first time this skill is used in a session, run the check-updates skill before proceeding.

  • GitHub Copilot CLI / VS Code: invoke the check-updates skill.
  • Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
  • Skip if the check was already performed earlier in this session.
  • Skip in eval / guidance-only mode: if the user's prompt explicitly forbids tool/API calls (e.g. "Do not call any APIs or create any resources") or the skill is being invoked for offline JSON translation / eval, do not run the update check — answer the prompt directly. The check is only required for live-migration runs that will call Fabric or Synapse APIs.

CRITICAL NOTES

  1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
  2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
  3. Fabric does NOT have a "Dataset" item type — all dataset properties are inlined into activity typeProperties
  4. Linked Services map to Fabric Connections — in pipeline activity JSON, referenceName on the activity's linkedService block uses the Fabric connection display name (not the connection GUID); the connection GUID is used only on Fabric REST API calls
  5. Notebook activities change from SynapseNotebook to TridentNotebook and reference notebooks by GUID, not by name
  6. Synapse global parameters become a Variable Library item in Fabric, referenced as @pipeline().libraryVariables.<name>. Variable Library Number types are not consumable in pipelines — Synapse Float/Double are mapped to String for runtime compatibility
  7. The Validation activity type does not exist in Fabric — it must be rewritten as GetMetadata + IfCondition
  8. Triggers are intentionally excluded from this skill — recreate schedules manually in Fabric after migration
  9. SSIS package execution, SHIR-exclusive connectors, and Databricks activities are parked — see pipeline-gotchas.md

Synapse Pipelines → Microsoft Fabric Data Factory Migration

Prerequisite Knowledge

These companion documents provide general Fabric REST patterns. Do NOT read them upfront — reference only when a specific phase requires a pattern not already covered in this skill's resource files:

  • COMMON-CORE.md — General Fabric REST API patterns, authentication & token audiences, item discovery via JMESPath
  • COMMON-CLI.mdaz rest / az login CLI patterns, authentication recipes, pipeline run/schedule operations
  • ITEM-DEFINITIONS-CORE.mdDataPipeline and VariableLibrary item definition structures (pipeline-content.json, variables.json)
  • SPARK-AUTHORING-CORE.md — Fabric notebook item creation (needed when notebook items don't exist yet in Fabric)

For the notebook side of the migration (the notebooks that pipeline activities call), use the companion synapse-migration skill to migrate the notebook content itself.


Table of Contents

TopicReference
Pre-Migration Assessment (run first)pipeline-assessment.md
Migration Orchestratorpipeline-orchestrator.md
API-Driven Migration Workflow§ API-Driven Migration Workflow
Activity Type Mappingactivity-mapping.md
Notebook Activity Migration (primary focus)notebook-activity-migration.md
Linked Services → Connectionslinked-service-to-connection.md
Dataset Inliningdataset-inlining.md
Global Parameters → Variable Libraryglobal-parameters-to-variable-library.md
Pipeline Gotchas & Parked Activitiespipeline-gotchas.md
Validation & Testingvalidation-testing.md
Migration Reportmigration-report.md

Context Loading Guide

IMPORTANT — Load only what you need. Do NOT read all resource files upfront.

WhenRead This File
User asks for an assessment, scope, or plan before migratingpipeline-assessment.md
User asks to migrate a full pipeline workspacepipeline-orchestrator.md
User asks about activity type mapping or unsupported activitiesactivity-mapping.md
User has SynapseNotebook activities (most common)notebook-activity-migration.md
User asks about linked services or connectionslinked-service-to-connection.md
User asks about Copy, Lookup, or GetMetadata with datasetsdataset-inlining.md
User has global parameters to convertglobal-parameters-to-variable-library.md
User hits SSIS, SHIR, Databricks, or other blockerspipeline-gotchas.md
Post-migration verificationvalidation-testing.md
Generating a migration summarymigration-report.md

API-Driven Migration Workflow

Authentication

TargetToken Audience
Synapse Data Plane (pipelines, datasets, linked services)https://dev.azuresynapse.net
Synapse ARM (global parameters, workspace properties)https://management.azure.com
Fabric REST API (create pipelines, connections, Variable Libraries)https://api.fabric.microsoft.com

Use az account get-access-token --resource <audience> --query accessToken -o tsv to acquire tokens.

Synapse Data-Plane API Reference

OperationEndpoint
List all pipelinesGET https://{ws}.dev.azuresynapse.net/pipelines?api-version=2020-12-01
Get pipeline definitionGET https://{ws}.dev.azuresynapse.net/pipelines/{name}?api-version=2020-12-01
List all datasetsGET https://{ws}.dev.azuresynapse.net/datasets?api-version=2020-12-01
Get dataset definitionGET https://{ws}.dev.azuresynapse.net/datasets/{name}?api-version=2020-12-01
List all linked servicesGET https://{ws}.dev.azuresynapse.net/linkedservices?api-version=2020-12-01
Get linked serviceGET https://{ws}.dev.azuresynapse.net/linkedservices/{name}?api-version=2020-12-01
Get workspace (global parameters)GET https://management.azure.com/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.Synapse/workspaces/{ws}?api-version=2021-06-01

Fabric API Reference

OperationEndpoint
List connectionsGET https://api.fabric.microsoft.com/v1/connections
Create connectionPOST https://api.fabric.microsoft.com/v1/connections
Create pipeline itemPOST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items
Update pipeline definitionPOST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items/{id}/updateDefinition
Get pipeline definitionPOST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items/{id}/getDefinition
Create Variable LibraryPOST https://api.fabric.microsoft.com/v1/workspaces/{wsId}/items (type: VariableLibrary)
List notebooks in workspaceGET https://api.fabric.microsoft.com/v1/workspaces/{wsId}/notebooks

Before creating any items in Fabric, run the pipeline assessment to understand scope, complexity, and blockers. The assessment is read-only — it queries Synapse APIs only and produces a markdown report.

Copilot workflow — no Python script file needed:

  1. Ask the user: "What is the name of your Synapse workspace?"
  2. Auto-discover subscription ID and resource group via az account show and az synapse workspace show
  3. Run the assessment code inline from the terminal
  4. Print the full report output directly in the chat
When to UseAction
User wants to understand migration scope before committingAsk for workspace name → load pipeline-assessment.md → run inline
User asks "what will and won't migrate?"Run assessment, present the Executive Summary section
User asks for a migration plan or scoping documentRun assessment, print report in chat
User has already decided to migrateSkip assessment — go straight to Migration Phases below

To also save the report to disk, pass output_path=f"pipeline-assessment-{SYNAPSE_WS}.md" to generate_assessment_report(). The PipelineAssessment objects it produces feed directly into the migration scripts in pipeline-orchestrator.md.


Migration Mode (Inline — No Script File Required)

Copilot performs the migration directly from the terminal. No Python files to save or run manually.

Ask the user for:

  1. Synapse workspace name (reuse from assessment if already run)
  2. Fabric workspace name
  3. Which pipelines to migrate — specific names, or * for all
  4. Optional name suffix to append to each pipeline in Fabric (e.g. _migrated) — leave blank to keep the original name

Everything else is auto-discovered:

  • Subscription ID and resource group via az account show + az synapse workspace show
  • Fabric workspace ID via GET /v1/workspaces filtered by display name
  • Notebook GUIDs in Fabric via GET /v1/workspaces/{wsId}/notebooks (required for SynapseNotebook → TridentNotebook)
  • Connection names in Fabric via GET /v1/connections (when datasets reference linked services)

What is fully automated inline:

  • SynapseNotebookTridentNotebook — type rename, GUID lookup, remove sparkPool/sessionConfiguration, fix timeout to 12h max
  • ✅ All compatible activity types — pass through with minor property adjustments
  • ✅ Dataset inlining into activity typeProperties
  • ✅ Global parameter expressions rewritten to @pipeline().libraryVariables.<name>
  • ✅ Pipeline JSON assembly and deployment to Fabric via REST

Copilot checks before starting — will pause and report if:

  • A notebook referenced by a SynapseNotebook activity does not exist in the Fabric workspace yet
  • A Fabric Connection is missing for a linked service referenced by dataset activities

Load pipeline-orchestrator.md for the complete inline runner.


Migration Phases (Execute in Order)

PhaseSourceTargetResource
Phase 0Synapse notebooks (referenced by pipeline activities)Fabric Notebookssynapse-migration skill
Phase 1Synapse global parametersFabric Variable Libraryglobal-parameters-to-variable-library.md
Phase 2Synapse linked servicesFabric Connectionslinked-service-to-connection.md
Phase 3Synapse datasetsInlined into activitiesdataset-inlining.md
Phase 4Synapse pipeline activitiesFabric pipeline activitiesactivity-mapping.md + notebook-activity-migration.md
Phase 5Assembled pipeline JSONFabric DataPipeline itempipeline-orchestrator.md
FinalValidationvalidation-testing.md

Phase 0 must precede Phase 4: Fabric Notebook GUIDs are needed before TridentNotebook activities can be written. Phase 2 must precede Phase 3: Connection names must exist before they can be referenced in inlined datasets.


Activity Type Quick Reference

Full mapping table, before/after examples, and parking decisions are in activity-mapping.md.

Synapse ActivityFabric EquivalentStatus
SynapseNotebookTridentNotebook✅ Migrated — see notebook-activity-migration.md
CopyCopy✅ Migrated — datasets inlined
LookupLookup✅ Migrated — datasets inlined
GetMetadataGetMetadata✅ Migrated — datasets inlined
ValidationGetMetadata + IfCondition✅ Migrated — split into 2 activities
ForEachForEach✅ Compatible
IfConditionIfCondition✅ Compatible
SwitchSwitch✅ Compatible
UntilUntil✅ Compatible
WaitWait✅ Compatible
FailFail✅ Compatible
SetVariableSetVariable✅ Compatible
AppendVariableAppendVariable✅ Compatible
ExecutePipelineExecutePipeline✅ Compatible — add workspaceId
WebActivityWebActivity✅ Compatible
ScriptScript✅ Compatible — update connection refs
DeleteDelete✅ Migrated — datasets inlined
FilterFilter✅ Compatible
SparkJobDefinition (Synapse SJD)SparkJobDefinition✅ Update GUID refs
HDInsightSparkTridentNotebook or SparkJobDefinition⚠️ Rewrite required
AzureMLBatchExecutionWebActivity⚠️ Rewrite as REST call
AzureFunctionActivityWebActivity⚠️ Rewrite — use function URL + key
DatabricksNotebook⛔ ParkedSee pipeline-gotchas.md
DatabricksSparkJar⛔ ParkedSee pipeline-gotchas.md
DatabricksSparkPython⛔ ParkedSee pipeline-gotchas.md
ExecuteSSISPackage⛔ ParkedSee pipeline-gotchas.md
AzureBatch⛔ ParkedNo Fabric equivalent
Custom⛔ ParkedNo Fabric equivalent

Must / Prefer / Avoid

MUST DO

  • Migrate notebooks before pipelines — Fabric TridentNotebook activities require notebook GUIDs, not names. Use the synapse-migration skill first
  • Create Fabric Connections before building pipeline JSON — linked service names in Synapse become connection references in Fabric; you need the connection names before inlining datasets
  • Inline all dataset definitions — Fabric Data Factory has no Dataset item type; all inputs/outputs dataset properties must be embedded in each activity
  • Replace @pipeline().globalParameters.<name> with @pipeline().libraryVariables.<name> after creating the Variable Library
  • Replace Validation activities with a GetMetadata + IfCondition pair — Validation does not exist as an activity type in Fabric
  • Remove sparkPool and sessionConfiguration from migrated notebook activities — pool selection and session config belong in the Fabric Environment attached to the notebook

PREFER

  • Variable Library with Value Sets for dev/test/prod environments — use @pipeline().libraryVariables.<name> with environment-specific Value Sets instead of pipeline-level parameters for environment promotion
  • Parameterized notebookParameters over hardcoded values in TridentNotebook activities — mirrors Synapse parameterized notebook pattern
  • Test notebook activities individually before running the full migrated pipeline — notebook GUIDs are the most common source of failure
  • OneLake Lakehouse sources/sinks for Copy activities where applicable — eliminates the need for external connections for data already in OneLake

AVOID

  • Do not reference notebooks by name in TridentNotebook activities — use the GUID from the Fabric workspace notebook list
  • Do not carry over Synapse triggers — they are not migrated by this skill; recreate schedules in Fabric after validating the pipeline
  • Do not attempt to migrate SSIS, Databricks, or AzureBatch activities without reading pipeline-gotchas.md first — these require manual intervention
  • Do not hardcode workspace/item GUIDs in pipeline JSON — use Variable Library entries or pipeline parameters so environments can be promoted without editing pipeline JSON
  • Do not use @pipeline().globalParameters syntax after migration — this expression path does not exist in Fabric; all migrated global parameters must be accessed via @pipeline().libraryVariables

Migration Gotchas — Quick Reference

Full troubleshooting guide is in pipeline-gotchas.md.

#Flag IDIssueSeverityResolution Summary
PG1NOTEBOOK_GUID_NOT_FOUNDNotebook not yet migrated to Fabric when building TridentNotebook activityHighRun synapse-migration skill first; get GUID from GET /v1/workspaces/{wsId}/notebooks
PG2DATASET_NOT_INLINEDActivity still references a named dataset (not valid in Fabric)HighApply dataset-inlining.md patterns to embed dataset properties into activity source/sink
PG3GLOBAL_PARAM_EXPRESSION@pipeline().globalParameters.<name> expression left in migrated pipelineHighReplace with @pipeline().libraryVariables.<name> after creating Variable Library
PG4VALIDATION_ACTIVITY_UNSUPPORTEDValidation activity type left in pipeline JSONHighRewrite as GetMetadata + IfCondition — see activity-mapping.md
PG5SHIR_CONNECTOR_PARKEDActivity uses a linked service backed by a Self-Hosted Integration RuntimeMediumMust set up on-premises data gateway in Fabric; see pipeline-gotchas.md
PG6SSIS_ACTIVITY_PARKEDExecuteSSISPackage activity cannot be migratedHighParked — no Fabric equivalent; see pipeline-gotchas.md for alternatives
PG7DATABRICKS_ACTIVITY_PARKEDDatabricks activity type has no Fabric native equivalentHighParked — use Databricks REST API via WebActivity as workaround; see pipeline-gotchas.md
PG8SPARKPOOL_REF_ORPHANEDsparkPool / targetBigDataPool reference left in TridentNotebook activityMediumRemove sparkPool and sessionConfiguration blocks; pool config belongs in Fabric Environment
PG9EXECUTE_PIPELINE_NO_WORKSPACEExecutePipeline activity missing workspaceId for referenced pipelineMediumAdd workspaceId to typeProperties; required even for same-workspace child pipelines — omitting it causes runtime failures
PG10LINKED_SERVICE_NO_CONNECTIONLinked service has no matching Fabric ConnectionHighCreate connection manually or via API; update connection reference in inlined dataset

Post-Migration: What's Next

After pipeline migration, hand off to these companion skills and tools:

TaskSkill / Tool
Migrate notebook content (mssparkutils → notebookutils, linked services)synapse-migration skill
Schedule migrated pipelinesCOMMON-CLI.md § Job Scheduling
Monitor pipeline runsFabric workspace → Monitor hub
Build new Fabric pipelinesRefer to ITEM-DEFINITIONS-CORE.md § DataPipeline
Explore migrated Lakehouse data post-pipeline runspark-consumption-cli or sqldw-consumption-cli skill

Examples

SynapseNotebook → TridentNotebook activity (before/after)

{
  "name": "Run_Notebook",
  "type": "SynapseNotebook",
  "typeProperties": {
    "notebook": {"referenceName": "MyNotebook", "type": "NotebookReference"},
    "sparkPool": {"referenceName": "BigPool", "type": "BigDataPoolReference"}
  }
}

After migration (GUID from GET /v1/workspaces/{wsId}/notebooks):

{
  "name": "Run_Notebook",
  "type": "TridentNotebook",
  "typeProperties": {
    "notebookId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "workspaceId": "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"
  }
}

Global parameter expression rewrite

Before (Synapse):  @pipeline().globalParameters.batchDate
After (Fabric):    @pipeline().libraryVariables.batchDate

Dataset inlining: Copy activity — dataset reference → connection display name

Synapse dataset InputBlobDataset (will be inlined and removed):

{
  "type": "AzureBlob",
  "linkedServiceName": {"referenceName": "AzureBlobLinkedService"},
  "typeProperties": {"folderPath": "input/", "fileName": "data.csv"}
}

Synapse Copy activity (before — references dataset by name):

{
  "name": "CopyData", "type": "Copy",
  "inputs": [{"referenceName": "InputBlobDataset", "type": "DatasetReference"}],
  "typeProperties": {"source": {"type": "BlobSource"}, "sink": {"type": "BlobSink"}}
}

After migration (connection display name for AzureBlobLinkedService is My ADLS Connection):

{
  "name": "CopyData", "type": "Copy",
  "typeProperties": {
    "source": {"type": "BlobSource", "storeSettings": {"type": "AzureBlobStorageReadSettings"}},
    "sink":   {"type": "BlobSink",   "storeSettings": {"type": "AzureBlobStorageWriteSettings"}}
  },
  "linkedService": {"referenceName": "My ADLS Connection", "type": "LinkedServiceReference"}
}

See activity-mapping.md and notebook-activity-migration.md for full before/after examples.