One-time Data migration from Cosmos DB to Storage Account (data lake) using Azure Synapse Error converting value "MultiHash" to type 'Microsoft.Azure.Documents.PartitionKind'

Hi Galin Jelev,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Hello,

Thank you for the answer. If I understand correctly, you're suggesting using a different tool to do the export. It would be nice if you could share a minimal Spark config or a sample statement.

Thanks.

Sure Galin Jelev.
here’s a small, On initial setup to export from a Cosmos DB container that uses hierarchical partition keys to ADLS in Parquet. Works the same in Synapse Spark or Databricks (just match the Cosmos Spark v4 connector to your Spark/Scala version).

Cosmos DB (HPK) → ADLS (Parquet): Using PySpark

# 1) Cluster / pool dependency:
#    Add the Maven package for the Cosmos Spark v4 connector, e.g.:
#    com.azure.cosmos.spark:azure-cosmos-spark_3-4_2.12:<latest>
#    (Pick the artifact that matches your Spark & Scala versions.)

# 2) Cosmos read options (HPK is supported out of the box in v4)
cosmos_cfg = {
  "spark.cosmos.accountEndpoint": "https://<cosmos-account>.documents.azure.com:443/",
  "spark.cosmos.accountKey":      "<cosmos-key>",     # or use AAD (see note below)
  "spark.cosmos.database":        "<db-name>",
  "spark.cosmos.container":       "<container-name>",
  "spark.cosmos.read.inferSchema.enabled": "true"
  # Optional performance knobs (uncomment if needed):
  # "spark.cosmos.read.maxItemCount": "1000",
  # "spark.cosmos.throughputControl.enabled": "true",
  # "spark.cosmos.throughputControl.targetThroughput": "40000"
}

df = (spark.read
           .format("cosmos.oltp")
           .options(**cosmos_cfg)
           .load())

# If you only need a subset of columns, project early to reduce I/O:
# df = df.select("id", "someField", "anotherField", "_ts")

# 3) Give the Spark runtime access to ADLS (one of these):
#    • Databricks: use a service principal/MI and a storage credential or mount
#    • Synapse Spark: grant the workspace managed identity “Storage Blob Data Contributor”
#    (If you must set ABFS OAuth in code, an example is shown below.)
#
# 4) Write to ADLS in Parquet
target_path = "abfss://<filesystem>@<storage-account>.dfs.core.windows.net/cosmos_export/runDate=<yyyyMMdd>"
(df
  # .repartition(64)  # choose a file count if you like
  .write
  .mode("overwrite")
  .format("parquet")
  .save(target_path))

ABFS OAuth config in code (service principal)

Use this only if you’re not mounting or not relying on workspace/managed identity.

acct = "<storage-account>"
spark.conf.set(f"fs.azure.account.auth.type.{acct}.dfs.core.windows.net", "OAuth")
spark.conf.set(f"fs.azure.account.oauth.provider.type.{acct}.dfs.core.windows.net",
               "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set(f"fs.azure.account.oauth2.client.id.{acct}.dfs.core.windows.net", "<appId>")
spark.conf.set(f"fs.azure.account.oauth2.client.secret.{acct}.dfs.core.windows.net", "<secret>")
spark.conf.set(f"fs.azure.account.oauth2.client.endpoint.{acct}.dfs.core.windows.net",
               "https://login.microsoftonline.com/<tenant-id>/oauth2/token")

Notes

• No special switch is required for hierarchical partition keys—the v4 connector understands them. If you filter on the full HPK (all components), reads will be cheaper; otherwise expect cross-partition scans.
• Prefer AAD over keys in production:

    # cosmos_cfg update if using AAD:
    # cosmos_cfg["spark.cosmos.auth.type"] = "AzureAD"
    # cosmos_cfg["spark.cosmos.auth.aad.clientId"] = "<appId>"         # or use MSI on the cluster
    # cosmos_cfg["spark.cosmos.auth.aad.clientSecret"] = "<secret>"
    # cosmos_cfg["spark.cosmos.auth.aad.tenantId"] = "<tenant-id>"
  

• To write CSV instead of Parquet:

    (df.write.mode("overwrite").option("header","true").csv(target_path))
  

Alternative (if Synapse Link is enabled): CETAS from serverless SQL to ADLS

This avoids the OLTP connector entirely and works fine with hierarchical PK.

-- Create (once) external objects pointing to your lake
CREATE EXTERNAL DATA SOURCE [Lake]
WITH (LOCATION = 'abfss://<filesystem>@<storage-account>.dfs.core.windows.net/');

CREATE EXTERNAL FILE FORMAT [ParquetFmt] WITH (FORMAT_TYPE = PARQUET);

-- Export: point SELECT at your Synapse Link view/table for the container
CREATE EXTERNAL TABLE [export].[container_snapshot]
WITH (
  DATA_SOURCE = [Lake],
  LOCATION = 'cosmos_export/cetas_snapshot/',
  FILE_FORMAT = [ParquetFmt]
)
AS
SELECT * FROM [<link_db>].[dbo].[<container_view_or_table>];