Edit

Share via

IoT analytics with Azure Data Explorer and Azure IoT Hub

Azure Cosmos DB
Azure Data Explorer
Azure Digital Twins
Azure IoT Hub

Solution ideas

This article describes a solution idea. Your cloud architect can use this guidance to help visualize the major components for a typical implementation of this architecture. Use this article as a starting point to design a well-architected solution that aligns with your workload's specific requirements.

This solution idea describes how Azure Data Explorer provides near real-time analytics for fast flowing, high volume streaming data from internet of things (IoT) devices and sensors. This analytics workflow is part of an overall IoT solution that integrates operational and analytical workloads with Azure Cosmos DB and Azure Data Explorer.

Jupyter is a trademark of its respective company. No endorsement is implied by the use of this mark. Apache® and Apache Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Architecture

Diagram showing IoT telemetry analytics with Azure Data Explorer.

Download a Visio file of this architecture.

Dataflow

  1. Azure Event Hubs, Azure IoT Hub, or Kafka ingest a wide variety of fast-flowing streaming data such as logs, business events, and user activities.

  2. Azure Functions or Azure Stream Analytics process the data in near real time.

  3. Azure Cosmos DB stores streamed messages in JSON format to serve a real-time operational application.

  4. Azure Data Explorer ingests data for analytics, using its connectors for Azure Event Hubs, Azure IoT Hub, or Kafka for low latency and high throughput.

    Alternatively, you can ingest blobs from your Azure Blob Storage or Azure Data Lake Storage account into Azure Data Explorer by using an Event Grid data connection.

    You can also continuously export data to Azure Storage in compressed, partitioned Apache Parquet format, and seamlessly query the data with Azure Data Explorer. For more information, see Continuous data export overview.

  5. To serve both the operational and analytical use cases, data can either route to Azure Data Explorer and Azure Cosmos DB in parallel, or from Azure Cosmos DB to Azure Data Explorer.

    • Azure Cosmos DB transactions can trigger Azure Functions via change feed. Functions will stream data to Event Hubs for ingestion into Azure Data Explorer.

      -or-

    • Azure Functions can invoke Azure Digital Twins through its API, which then streams data to Event Hubs for ingestion into Azure Data Explorer.

  6. The following interfaces get insights from data stored in Azure Data Explorer:

  7. Azure Data Explorer integrates with Azure Databricks and Azure Machine Learning to provide machine learning (ML) services. You can also build ML models using other tools and services, and export them to Azure Data Explorer for scoring data.

Components

This solution idea uses the following Azure components.

Azure Data Explorer

  • Anomaly detection and forecasting is a built-in analytics feature in Azure Data Explorer. It detects outliers and predicts future values to support proactive monitoring and decision-making. In this architecture, it identifies unusual patterns in IoT telemetry and forecasts expected behavior over time.

  • Anomaly diagnosis for root analysis is a Kusto Query Language (KQL) capability that helps identify the root causes of anomalies. It analyzes contributing dimensions and metrics to streamline troubleshooting. In this architecture, it isolates the source of anomalies detected in device data.

  • Azure Data Explorer is a fully managed, high-performance analytics service. It processes large volumes of streaming data from applications, websites, and IoT devices in near real-time. In this architecture, it serves as the central analytics engine for ingesting, querying, and visualizing IoT data.

  • Azure Data Explorer dashboards are a visualization feature within the Web UI. They allow users to export Kusto queries into interactive dashboards for real-time data exploration. In this architecture, they display insights from IoT data streams and anomaly detection results.

  • Azure Data Explorer web UI is a browser-based interface for working with Azure Data Explorer clusters. It supports writing, running, and sharing KQL commands and queries. In this architecture, it provides a workspace for analysts to query and explore IoT telemetry.

  • Time series analysis is a built-in capability in Azure Data Explorer. It enables users to explore temporal patterns, trends, and seasonality in time-based data. In this architecture, it reveals long-term trends and cyclical behavior in IoT sensor readings.

Other Azure components

  • Azure Cosmos DB is a fully managed, fast NoSQL database service for modern app development with open APIs for any scale. In this architecture, it stores operational data from IoT devices for scalable, low-latency access.

  • Azure Digital Twins is a platform for modeling physical environments as digital representations. In this architecture, it maintains digital models of IoT-connected assets to support spatial analysis and contextual insights.

  • Azure IoT Hub enables bi-directional communication between IoT devices and the Azure cloud. In this architecture, it serves as the central messaging hub for device telemetry and command-and-control operations.

  • Event Hubs is a fully managed, real-time data ingestion service. In this architecture, it ingests telemetry from IoT devices and streams it into the analytics pipeline.

  • Kafka on HDInsight is an enterprise-grade, cost-effective service for running Apache Kafka on Azure. In this architecture, it provides an alternative streaming backbone for ingesting and distributing IoT data.

Scenario details

This solution uses Azure Data Explorer to get near real-time IoT telemetry analytics on fast-flowing, high-volume streaming data from a wide variety of IoT devices.

Potential use cases

  • Fleet management, for predictive maintenance of vehicle parts. This solution is ideal for the automotive and transportation industry.
  • Facilities management, for energy and environment optimization.
  • Combining real-time road conditions with weather data for safer autonomous driving.

Contributors

This article is maintained by Microsoft. It was originally written by the following contributors.

Principal author:

Next steps