How to store fabric data pipeline logs in Log Analytics workspace?

Hello Pradeep Agrawal,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Your question is how to store fabric data pipeline logs in Log Analytics workspace.

Since you found a right tag on Microsoft Q&A, I think I can answer you: The below are the step-by-step guide to store Fabric Pipeline Logs in Log Analytics:

Prerequisites:

• You must be a Fabric Admin.
• You need an Azure subscription with a Log Analytics workspace.
• Your Fabric environment must be on a paid capacity (not trial)-https://community.fabric.microsoft.com/t5/Data-Engineering/how-to-get-logs-from-dataflow-gen2-and-data-pipeline-to-log/m-p/4707133

Choose between Workspace monitoring and Log Analytics: You cannot enable both in the same workspace.

• If Workspace Monitoring is enabled, disable it first before configuring Log Analytics - https://free.blessedness.top/en-us/fabric/fundamentals/workspace-monitoring-overview

Configure diagnostic settings:

• Go to Azure Portal > Microsoft Fabric Capacity.
• Navigate to Monitoring > Diagnostic Settings.
• Click Add Diagnostic Setting.
• Select Log Analytics Workspace as the destination.
• Choose categories like:
  • PipelineRun
  • PipelineActivityRun
  • DataflowActivity

https://community.fabric.microsoft.com/t5/Data-Engineering/how-to-get-logs-from-dataflow-gen2-and-data-pipeline-to-log/m-p/4707133

Save and verify:

• Save the configuration.
• Go to Azure Monitor > Log Analytics Workspace.
• Run a KQL query to verify logs:

    PipelineRun
    | where TimeGenerated > ago(1h)
    | summarize count() by Status
  

If categories like PipelineRun are missing, then check if you're using preview features. Make sure you are using the Fabric Capacity Metrics App as an alternative - https://free.blessedness.top/en-us/answers/questions/2281337/set-up-fabric-capacity-specific-workload-logging-i and to ask more technical questions and troubleshooting, you can post in the Fabric Community - https://community.fabric.microsoft.com for tenant-specific issues.

I hope this is helpful! Do not hesitate to let me know if you have any other questions or clarifications.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.