Firebase data ingestion to Delta Lake (Databricks)

Hi Subhash Nagaraju,

Thanks for posting your question in the Microsoft Q&A forum

There’s no direct connector from Firebase to Delta Lake in Azure Databricks, but you can still integrate them easily. For batch jobs, you can use tools like Airbyte or export Firebase data with the Firebase Admin SDK, store it in Blob/ADLS, and load it into Delta Lake with Databricks Auto Loader.

So, even though Azure doesn’t provide a native connector, Firebase data can still be moved to Delta Lake using batch pipelines, streaming pipelines, or third-party tools depending on your needs.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful

Hi Hossein,

Can you please provide any documentation which can provide information around the steps to integrate focussing on SDK's which can be used in Azure env to access and download the data from Firebase.

Thanks.

Subhash

Hi @Subhash Nagaraju  

Microsoft’s connector list does not include Firebase, so there is no native “Firebase → Delta Lake” connector. However, you can integrate Firebase with Azure Databricks using SDKs and supported ingestion patterns. Below is a consolidated guide with official documentation links and steps focusing on SDK-based access in an Azure environment. Firebase Admin SDKs (Python, Java, Node.js, etc.) allow you to:

• Authenticate using service account credentials.
• Read/write data from Firestore or Realtime Database.
• Access Cloud Storage buckets for file exports.

You can run these SDKs inside Azure Databricks notebooks or Azure Functions to pull data and then write to Delta Lake tables.

Official Documentation for SDK Setup

Firebase Admin SDK

• Setup Guide (Service Account + Initialization) Add the Firebase Admin SDK to your server Key steps: [firebase.google.com]
  1. Generate a service account JSON key from Firebase Console.
     1. Install SDK:
        • Python: pip install firebase-admin (Firebase Admin Python SDK) [firebase.google.com]

          - Java: Add Maven dependency ([Firebase Admin Java SDK](https://github.com/firebase/firebase-admin-java)) [[github.com]](https://github.com/firebase/firebase-admin-java)
          
             1. Initialize:
          

Python

import firebase_admin

from firebase_admin import credentials

cred = credentials.Certificate("/dbfs/FileStore/serviceAccountKey.json")

firebase_admin.initialize_app(cred)

(Store the key securely in Azure Key Vault or DBFS.)

• Firestore & Realtime Database Access https://firebase.google.com/docs/reference/admin/python/firebase_admin.firestore Example:

Python

from firebase_admin import firestore

db = firestore.client()

docs = db.collection('users').stream()

for doc in docs:

print(doc.id, doc.to_dict())

Azure Databricks SDKs

• Databricks SDK for Python [free.blessedness.top]
• Databricks SDK for Java These SDKs help automate cluster jobs and integrate ingestion scripts into Databricks workflows. [free.blessedness.top]

Integration Patterns Using SDKs

Batch Ingestion

1. Use Firebase Admin SDK to export data:
   • Firestore → JSON or stream documents.
     • Realtime DB → JSON export.
     1. Write files to Azure Blob Storage or ADLS Gen2.
2. Use Databricks Auto Loader to ingest into Delta Lake: https://free.blessedness.top/en-us/azure/databricks/ingestion/auto-loader/ Example: [free.blessedness.top]

Python

df = (spark.readStream

.format("cloudFiles")

.option("cloudFiles.format", "json")

.load("abfss://******@storageaccount.dfs.core.windows.net/firebase-data"))

df.writeStream.format("delta").option("checkpointLocation", "/delta/checkpoints").start("/delta/firebase")

Real-Time Ingestion

• Use Firebase triggers (Cloud Functions) to publish changes to Google Pub/Sub.
• Consume Pub/Sub in Databricks using Structured Streaming: https://free.blessedness.top/en-us/azure/databricks/connect/streaming/pub-sub Example:

Python

df = (spark.readStream

.format("pubsub")

.option("projectId", "<gcp-project>")

.option("subscriptionId", "<subscription>")

.load())

df.writeStream.format("delta").option("checkpointLocation", "/delta/checkpoints").start("/delta/firebase-stream")

``

Authentication in Azure Environment

• Store Firebase service account JSON in Azure Key Vault or DBFS.
• Use Databricks secrets to manage credentials securely.
• For Pub/Sub, configure GCP service account and provide credentials via Databricks secret scope.

Troubleshooting & Best Practices

• Ensure Blaze plan for Firebase to enable export APIs.
• Handle schema evolution in Delta Lake with .option("mergeSchema", "true").
• Use checkpointing for streaming jobs to maintain exactly-once semantics.

Additional References

• Export Firestore data to Cloud Storage

Hope the above steps were helpful. If you have any other questions, please feel free to contact us.

Thanks,
Vrishabh

Hi Subhash Nagaraju,

We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.

Thanks,
Vrishabh