Create and manage Jupyter notebook jobs

You can create scheduled jobs to run at specific times or intervals using the Microsoft Sentinel extension for Visual Studio Code. Jobs allow you to automate data processing tasks to summarize, transform, or analyze data in the Microsoft Sentinel data lake. Jobs are also used to process data and write results to custom tables in the lake tier or analytics tier.

Permissions

Microsoft Entra ID roles provide broad access across all workspaces in the data lake. To create and schedule jobs, read tables across all workspaces, write to the analytics and lake tiers, you must have one of the supported Microsoft Entra ID roles. For more information on roles and permissions, see Roles and permissions in Microsoft Sentinel.

To create new custom tables in the analytics tier, the data lake managed identity must be assigned the Log Analytics Contributor role in the Log Analytics workspace.

To assign the role, follow the steps below:

1. In the Azure portal, navigate to the Log Analytics workspace that you want to assign the role to.
2. Select Access control (IAM) in the left navigation pane.
3. Select Add role assignment.
4. In the Role table, select Log Analytics Contributor, then select Next
5. Select Managed identity, then select Select members.
6. Your data lake managed identity is a system assigned managed identity named msg-resources-<guid>. Select the managed identity, then select Select.
7. Select Review and assign.

For more information on assigning roles to managed identities, see Assign Azure roles using the Azure portal.

Create and schedule a job

You can create a job in one of three ways:

1. In the notebook editor, select Create schedule Job from the toolbar.

2. In the Explorer pane, right-click the notebook file and select Microsoft Sentinel, then select Create schedule Job.

   

3. From the list of jobs, select the + icon to create a new job.

   

4. Select Use existing notebook to select an existing notebook file, or select Create new notebook to create a new notebook file for the job.

   

5. On the Job configuration page, in the Job details section enter a name and description for the job.

6. Select the spark pool size to run the job according to your jobs compute needs.

7. To run a job manually without a schedule, select On demand in the Schedule section, then select Submit to save the job configuration and publish the job.

8. To specify a schedule for the job, select Scheduled in the Schedule section.

   1. Select a Repeat frequency for the job. You can choose from By the minute, Hourly, Weekly, Daily, or Monthly.

   2. Additional options are displayed to configure the schedule, depending on the frequency you select. For example day of the week, time of day, or day of the month.

   3. Select a Start on time for the schedule to start running.

   4. Select an End on time for the schedule to stop running. If you don't want to set an end time for the schedule, select Set job to run indefinitely. Dates and times are in the user's timezone.

   5. Select Submit to save the job configuration and publish the job.

   

9. To view your jobs, select the Microsoft Sentinel shield icon in the left toolbar. Jobs are displayed on the Jobs panel.

10. Select a job to see the job details.

11. You can run the job immediately by selecting Run now, disable and enable the job schedule, or delete the job.

    

12. View the job history in the Run history tab.

    

13. Select an activity to see more details.

Edit a submitted job

Submitting a job creates a job definition that includes the notebook file, the job configuration, and the schedule. The job definition is uploaded from your VS Code editor and stored in the Microsoft Sentinel data lake. Once submitted, the job is no longer connected to the notebook file on your local file system. If you want to edit the code in the notebook job, you must download the job definition, edit the notebook file, and then resubmit the job.

To edit a submitted job follow the steps below:

1. In the Jobs section, select the job you want to edit.

2. Select the Download cloud icon to download the job definition to your local file system. In the jobs details editor, you can see the job configuration. You can also select Download latest notebook.

   

3. Edit the downloaded ipynb workbook file to make your changes.

4. Return to the Job details tab and select Edit job.

5. Edit the job name, description, cluster configuration, and schedule. Changing the job name creates a new job definition when you submit the job.

6. Select Submit to upload the updated notebook file and job configuration.

7. A confirmation is displayed when the job is successfully submitted.

   

View jobs in the Microsoft Defender portal

In addition to viewing jobs in VS Code, you can also view your notebook jobs in the Defender portal. To view your jobs in the Defender portal, Select Microsoft Sentinel > Data lake exploration > Jobs .

The page shows a list of jobs and their types. Select a notebook job to view its details. You can enable and disable the job's schedule but you can't edit a notebook job in the Defender portal.

1. Select a job to view the job details.

1. Select View history to see the history of job runs.

Service parameters and limits and troubleshooting

For a list of service limits for the Microsoft Sentinel data lake, see Microsoft Sentinel data lake service limits.

For information on troubleshooting, see Run notebooks on the Microsoft Sentinel data lake.

Related content

• Sample notebooks for Microsoft Sentinel data lake
• Microsoft Sentinel Provider class reference
• Microsoft Sentinel data lake overview
• Roles and permissions in Microsoft Sentinel.