Share via

Serverless environment version 4

This article outlines the system environment information for serverless environment version 4.

To ensure compatibility for the application, serverless workloads use a versioned API, known as the environment version, which remains compatible with newer server versions.

You can select the environment version using the Environment side panel in your serverless notebooks. See Select an environment version.

New features and improvements

The following new features and improvements are available in serverless environment 4.

Spark ML support on serverless compute

Serverless compute now supports Spark ML in PySpark (pyspark.ml) and MLflow for Spark (mlflow.spark) in environment version 4. For hyperparameter tuning on serverless compute, Databricks recommends using Optuna and Joblib Spark.

The following limitations apply when running SparkML on serverless compute:

  • The maximum model size is 100MB.
  • The maximum total size of in-memory models per session is 1GB.
  • Tree model training stops early if the model size is about to exceed 100MB.
  • The following SparkML models are not supported:
    • DistributedLDAModel
    • FPGrowthModel

Databricks Connect upgraded to 17.0

Use the features and improvements available on Databricks Connect for Databricks Runtime 17.0.

Scalar Python UDFs now support service credentials

Scalar Python UDFs can use Unity Catalog service credentials to securely access external cloud services. For more information, see Service credentials in Scalar Python UDFs.

PySpark and Spark Connect now support the DataFrames df.mergeInto API

PySpark and Spark Connect now support the df.mergeInto API, which was previously only available for Scala.

API updates

Serverless environment 4 includes the following API updates:

  • SPARK-50915 Add getCondition and deprecate getErrorClass in PySparkException
  • SPARK-50719 Support interruptOperation for PySpark
  • SPARK-50718 Support addArtifact(s) for PySpark
  • SPARK-49530 Support pie subplots in PySpark plotting
  • SPARK-50357 Support Interrupt(Tag|All) APIs for PySpark
  • SPARK-51178 Raise proper PySpark error instead of SparkConnectGrpcException
  • SPARK-51227 Fix PySpark Connect _minimum_grpc_version to 1.67.0
  • SPARK-50778 Add metadataColumn to PySpark DataFrame
  • SPARK-50311 Support (add|remove|get|clear)Tag(s) APIs for PySpark
  • SPARK-50310 Add a flag to disable DataFrameQueryContext for PySpark
  • SPARK-50238 Add Variant Support in PySpark UDFs/UDTFs/UDAFs and Python UC UDFs
  • SPARK-50183 Unify the internal functions for Pandas API and PySpark Plotting
  • SPARK-50170 Move _invoke_internal_function_over_columns to pyspark.sql.utils
  • SPARK-50167 Improve PySpark plotting error messages and imports
  • SPARK-48961 Make the parameter naming of PySparkException consistent with JVM
  • SPARK-49567 Use classic instead of vanilla from PySpark code base
  • SPARK-48755 transformWithState PySpark base implementation and ValueState support
  • SPARK-48714 Implement DataFrame.mergeInto in PySpark
  • SPARK-47365 Add toArrow() DataFrame method to PySpark
  • SPARK-48075 Type checking for PySpark avro functions
  • SPARK-46984 Remove pyspark.copy_func
  • SPARK-46213 Introduce PySparkImportError for error framework
  • SPARK-46226 Migrate all remaining RuntimeError into PySpark error framework
  • SPARK-45450 Fix imports according to PEP8: pyspark.pandas and pyspark (core)

System environment

  • Operating System: Ubuntu 24.04.2 LTS
  • Python: 3.12.3
  • Databricks Connect: 17.0.1

Installed Python libraries

To reproduce serverless environment 4 in your local Python virtual environment, download the requirements-env-4.txt file and run pip install -r requirements-env-4.txt. This command installs all of the open source libraries from serverless environment 4.

Library Version Library Version Library Version
annotated-types 0.7.0 anyio 4.6.2 argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0 arrow 1.3.0 asttokens 2.0.5
astunparse 1.6.3 async-lru 2.0.4 attrs 24.3.0
autocommand 2.2.2 azure-common 1.1.28 azure-core 1.34.0
azure-identity 1.20.0 azure-mgmt-core 1.5.0 azure-mgmt-web 8.0.0
azure-storage-blob 12.23.0 azure-storage-file-datalake 12.17.0 babel 2.16.0
backports.tarfile 1.2.0 beautifulsoup4 4.12.3 black 24.10.0
bleach 6.2.0 blinker 1.7.0 boto3 1.36.2
botocore 1.36.3 cachetools 5.5.1 certifi 2025.1.31
cffi 1.17.1 chardet 4.0.0 charset-normalizer 3.3.2
click 8.1.7 cloudpickle 3.0.0 comm 0.2.1
contourpy 1.3.1 cryptography 43.0.3 cycler 0.11.0
Cython 3.0.12 databricks-connect 17.0.1 databricks-sdk 0.49.0
dbus-python 1.3.2 debugpy 1.8.11 decorator 5.1.1
defusedxml 0.7.1 Deprecated 1.2.13 distlib 0.3.9
docstring-to-markdown 0.11 executing 0.8.3 facets-overview 1.1.1
fastapi 0.115.12 fastjsonschema 2.21.1 filelock 3.18.0
fonttools 4.55.3 fqdn 1.5.1 fsspec 2023.5.0
gitdb 4.0.11 GitPython 3.1.43 google-api-core 2.20.0
google-auth 2.40.0 google-cloud-core 2.4.3 google-cloud-storage 3.1.0
google-crc32c 1.7.1 google-resumable-media 2.7.2 googleapis-common-protos 1.65.0
grpcio 1.67.0 grpcio-status 1.67.0 h11 0.14.0
httpcore 1.0.2 httplib2 0.20.4 httpx 0.27.0
idna 3.7 importlib-metadata 6.6.0 importlib_resources 6.4.0
inflect 7.3.1 iniconfig 1.1.1 ipyflow-core 0.0.209
ipykernel 6.29.5 ipython 8.30.0 ipython-genutils 0.2.0
ipywidgets 7.8.1 isodate 0.6.1 isoduration 20.11.0
jaraco.context 5.3.0 jaraco.functools 4.0.1 jaraco.text 3.12.1
jedi 0.19.2 Jinja2 3.1.5 jmespath 1.0.1
joblib 1.4.2 json5 0.9.25 jsonpointer 3.0.0
jsonschema 4.23.0 jsonschema-specifications 2023.7.1 jupyter-events 0.10.0
jupyter-lsp 2.2.0 jupyter_client 8.6.3 jupyter_core 5.7.2
jupyter_server 2.14.1 jupyter_server_terminals 0.4.4 jupyterlab 4.3.4
jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0 jupyterlab_server 2.27.3
kiwisolver 1.4.8 launchpadlib 1.11.0 lazr.restfulclient 0.14.6
lazr.uri 1.0.6 markdown-it-py 2.2.0 MarkupSafe 3.0.2
matplotlib 3.10.0 matplotlib-inline 0.1.7 mccabe 0.7.0
mdurl 0.1.0 mistune 2.0.4 mlflow-skinny 2.22.0
mmh3 5.1.0 more-itertools 10.3.0 msal 1.32.3
msal-extensions 1.3.1 mypy-extensions 1.0.0 nbclient 0.8.0
nbconvert 7.16.4 nbformat 5.10.4 nest-asyncio 1.6.0
nodeenv 1.9.1 notebook 7.3.2 notebook_shim 0.2.3
numpy 2.1.3 oauthlib 3.2.2 opentelemetry-api 1.32.1
opentelemetry-sdk 1.32.1 opentelemetry-semantic-conventions 0.53b1 overrides 7.4.0
packaging 24.1 pandas 2.2.3 pandocfilters 1.5.0
parso 0.8.4 pathspec 0.10.3 patsy 1.0.1
pexpect 4.8.0 pillow 11.1.0 pip 25.0.1
platformdirs 3.10.0 plotly 5.24.1 pluggy 1.5.0
prometheus_client 0.21.0 prompt-toolkit 3.0.43 proto-plus 1.26.1
protobuf 5.29.4 psutil 5.9.0 psycopg2 2.9.3
ptyprocess 0.7.0 pure-eval 0.2.2 py4j 0.10.9.9
pyarrow 19.0.1 pyasn1 0.4.8 pyasn1-modules 0.2.8
pyccolo 0.0.71 pycparser 2.21 pydantic 2.10.6
pydantic_core 2.27.2 pyflakes 3.2.0 Pygments 2.15.1
PyGObject 3.48.2 pyiceberg 0.9.0 PyJWT 2.10.1
pyodbc 5.2.0 pyparsing 3.2.0 pyright 1.1.394
pyspark 4.0.0+databricks.connect.17.0.1 pytest 8.3.5 python-dateutil 2.9.0.post0
python-json-logger 3.2.1 python-lsp-jsonrpc 1.1.2 python-lsp-server 1.12.0
pytoolconfig 1.2.6 pytz 2024.1 PyYAML 6.0.2
pyzmq 26.2.0 referencing 0.30.2 requests 2.32.3
rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rich 13.9.4
rope 1.12.0 rpds-py 0.22.3 rsa 4.9.1
s3transfer 0.11.3 scikit-learn 1.6.1 scipy 1.15.1
seaborn 0.13.2 Send2Trash 1.8.2 setuptools 74.0.0
six 1.16.0 smmap 5.0.0 sniffio 1.3.0
sortedcontainers 2.4.0 soupsieve 2.5 sqlparse 0.5.3
ssh-import-id 5.11 stack-data 0.2.0 starlette 0.46.2
statsmodels 0.14.4 strictyaml 1.7.3 tenacity 9.0.0
terminado 0.17.1 threadpoolctl 3.5.0 tinycss2 1.4.0
tokenize_rt 6.1.0 tomli 2.0.1 tornado 6.4.2
traitlets 5.14.3 typeguard 4.3.0 types-python-dateutil 2.9.0.20241206
typing_extensions 4.12.2 tzdata 2024.1 ujson 5.10.0
unattended-upgrades 0.1 uri-template 1.3.0 urllib3 2.3.0
uvicorn 0.34.2 virtualenv 20.29.3 wadllib 1.3.6
wcwidth 0.2.5 webcolors 24.11.1 webencodings 0.5.1
websocket-client 1.8.0 whatthepatch 1.0.2 wheel 0.45.1
widgetsnbextension 3.6.6 wrapt 1.17.0 yapf 0.40.2
zipp 3.21.0 zstandard 0.23.0