Share via

Spark Batch - Create Spark Batch Job

Service:
Synapse
API Version:
2020-12-01

Create new spark batch job.

POST {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches
With optional parameters: 
POST {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches?detailed={detailed}

URI Parameters

Name In Required Type Description
endpoint
 path True

string

The workspace development endpoint, for example https://myworkspace.dev.azuresynapse.net.
livyApiVersion
 path True

string

Valid api-version for the request.
sparkPoolName
 path True

string

Name of the spark pool.
detailed
 query

boolean

Optional query param specifying whether detailed response is returned beyond plain livy.

Request Body

Name Required Type Description
file True

string

name True

string

archives

string[]

args

string[]

artifactId

string

className

string

conf

object

driverCores

integer (int32)

driverMemory

string

executorCores

integer (int32)

executorMemory

string

files

string[]

jars

string[]

numExecutors

integer (int32)

pyFiles

string[]

tags

object

Responses

Name Type Description
200 OK

SparkBatchJob

Success

Examples

Create new spark batch job.

Sample request

POST myWorkspace.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/mySparkPool/batches?detailed=True

{
  "tags": null,
  "artifactId": "fill in here",
  "name": "jobname",
  "file": "abfss://",
  "className": "classname",
  "args": [],
  "jars": [],
  "files": [],
  "archives": [],
  "conf": null,
  "driverMemory": "4g",
  "driverCores": 4,
  "executorMemory": "2g",
  "executorCores": 4,
  "numExecutors": 2
}

Sample response

Status code:
200 
{
  "livyInfo": null,
  "name": "jobname",
  "workspaceName": "myWorkspace",
  "sparkPoolName": "mySparkPool",
  "submitterName": "thetime",
  "submitterId": "thesubmitterid",
  "artifactId": "fill in here",
  "jobType": "SparkBatch",
  "result": "Failed",
  "schedulerInfo": null,
  "pluginInfo": null,
  "errorInfo": [],
  "tags": null,
  "id": 1,
  "appId": "fill in here",
  "appInfo": null,
  "state": "the state",
  "log": []
}

Definitions

Name Description
LivyStates

The batch state
PluginCurrentState
SchedulerCurrentState
SparkBatchJob
SparkBatchJobOptions
SparkBatchJobResultType

The Spark batch job result.
SparkBatchJobState
SparkErrorSource
SparkJobType

The job type.
SparkRequest
SparkScheduler
SparkServiceError
SparkServicePlugin

LivyStates

Enumeration

The batch state

Value Description
not_started
starting
idle
busy
shutting_down
error
dead
killed
success
running
recovering

PluginCurrentState

Enumeration
Value Description
Preparation
ResourceAcquisition
Queued
Submission
Monitoring
Cleanup
Ended

SchedulerCurrentState

Enumeration
Value Description
Queued
Scheduled
Ended

SparkBatchJob

Object
Name Type Description
appId

string

The application id of this session
appInfo

object

The detailed application info.
artifactId

string

The artifact identifier.
errorInfo

SparkServiceError[]

The error information.
id

integer (int32)

The session Id.
jobType

SparkJobType

The job type.
livyInfo

SparkBatchJobState

log

string[]

The log lines.
name

string

The batch name.
pluginInfo

SparkServicePlugin

The plugin information.
result

SparkBatchJobResultType

The Spark batch job result.
schedulerInfo

SparkScheduler

The scheduler information.
sparkPoolName

string

The Spark pool name.
state

LivyStates

The batch state
submitterId

string

The submitter identifier.
submitterName

string

The submitter name.
tags

object

The tags.
workspaceName

string

The workspace name.

SparkBatchJobOptions

Object
Name Type Description
archives

string[]

args

string[]

artifactId

string

className

string

conf

object

driverCores

integer (int32)

driverMemory

string

executorCores

integer (int32)

executorMemory

string

file

string

files

string[]

jars

string[]

name

string

numExecutors

integer (int32)

pyFiles

string[]

tags

object

SparkBatchJobResultType

Enumeration

The Spark batch job result.

Value Description
Uncertain
Succeeded
Failed
Cancelled

SparkBatchJobState

Object
Name Type Description
currentState

string

the Spark job state.
deadAt

string (date-time)

time that at which "dead" livy state was first seen.
jobCreationRequest

SparkRequest

killedAt

string (date-time)

the time that at which "killed" livy state was first seen.
notStartedAt

string (date-time)

the time that at which "not_started" livy state was first seen.
recoveringAt

string (date-time)

the time that at which "recovering" livy state was first seen.
runningAt

string (date-time)

the time that at which "running" livy state was first seen.
startingAt

string (date-time)

the time that at which "starting" livy state was first seen.
successAt

string (date-time)

the time that at which "success" livy state was first seen.

SparkErrorSource

Enumeration
Value Description
System
User
Unknown
Dependency

SparkJobType

Enumeration

The job type.

Value Description
SparkBatch
SparkSession

SparkRequest

Object
Name Type Description
archives

string[]

args

string[]

className

string

conf

object

driverCores

integer (int32)

driverMemory

string

executorCores

integer (int32)

executorMemory

string

file

string

files

string[]

jars

string[]

name

string

numExecutors

integer (int32)

pyFiles

string[]

SparkScheduler

Object
Name Type Description
cancellationRequestedAt

string (date-time)

currentState

SchedulerCurrentState

endedAt

string (date-time)

scheduledAt

string (date-time)

submittedAt

string (date-time)

SparkServiceError

Object
Name Type Description
errorCode

string

message

string

source

SparkErrorSource

SparkServicePlugin

Object
Name Type Description
cleanupStartedAt

string (date-time)

currentState

PluginCurrentState

monitoringStartedAt

string (date-time)

preparationStartedAt

string (date-time)

resourceAcquisitionStartedAt

string (date-time)

submissionStartedAt

string (date-time)