Overview of data quality in Microsoft Purview Unified Catalog

Data quality in Microsoft Purview Unified Catalog empowers governance domain and data owners to assess and oversee the quality of their data ecosystem, facilitating targeted actions for improvement. In today's AI-driven landscape, the reliability of data directly impacts the accuracy of AI-driven insights and recommendations. Without trustworthy data, there's a risk of eroding trust in AI systems and hindering their adoption.

Poor data quality or incompatible data structures can hamper business processes and decision-making capabilities. Data quality in Unified Catalog addresses these challenges by offering users the ability to evaluate data quality using no-code or low-code rules, including out-of-the-box (OOB) rules and AI-generated rules. These rules are applied at the column level and aggregated to provide scores at the levels of data assets, data products, and governance domains, ensuring end-to-end visibility of data quality within each domain.

Data quality in Microsoft Purview also incorporates AI-powered data profiling capabilities, recommending columns for profiling while allowing human intervention to refine these recommendations. This iterative process not only enhances the accuracy of data profiling but also contributes to the continuous improvement of the underlying AI models.

By applying data quality, organizations can effectively measure, monitor, and enhance the quality of their data assets, bolstering the reliability of AI-driven insights and fostering confidence in AI-based decision-making processes.

Data quality life cycle

1. Assign users(s) data quality steward permissions in Unified Catalog to use all data quality features.
2. Register and scan a data source in Microsoft Purview Data Map.
3. Add your data asset to a data product
4. Set up a data source connection to prepare your source for data quality assessment.
5. Configure and run data profiling for an asset in your data source.
   1. When profiling is complete, browse the results for each column in the data asset to understand your data's current structure and state.
6. Set up data quality rules based on the profiling results, and apply them to your data asset.
7. Configure and run a data quality scan on a data product to assess the quality of all supported assets in the data product.
8. Review your scan results to evaluate your data product's current data quality.
9. Repeat steps 5-8 periodically over your data asset's life cycle to ensure it's maintaining quality.
10. Continually monitor your data quality
    1. Review data quality actions to identify and resolve problems.
    2. Set data quality notifications to alert you to quality issues.

Supported data quality regions

Data quality is currently supported in the following regions.

Supported multicloud data sources

View the list of supported data sources.

Currently, Microsoft Purview can only run data quality scans by using Managed Identity as an authentication option. Data quality services run on Apache Spark 3.4 and Delta Lake 2.4.

Data quality features

• Data source connection configuration
  • Configure connection to allow Microsoft Purview data quality SaaS application to have read access to data for quality scanning and profiling.
  • Microsoft Purview uses Managed Identity as an authentication option.
• Data profiling
  • AI-enabled data profiling experience.
  • Industry standard statistical snapshot (distribution, min, max, standard deviation, uniqueness, completeness, duplicate, and more).
  • Drill down column level profiling measures.
• Data quality rules
  • Out of box rules to measure six industry standards data quality dimensions (completeness, consistency, conformity, accuracy, freshness, and uniqueness).
  • Custom rules creation features include number of out of the box functions and expression values.
  • Auto generated rules with AI integrated experience.
• Data quality scanning
  • Select and assign rules to columns for data quality scan.
  • Apply data freshness rule in the entity or table level to measure the data freshness SLA.
  • Scheduling data quality scanning job for time period (hourly, daily, weekly, monthly, and more).
• Data quality job monitoring
  • Enable monitoring data quality job status (active, completed, failed, and more).
  • Enable browsing the data quality scanning history.
• Data quality scoring
  • Data quality score in rule level (what is the quality score for a rule that applied to a column).
  • Data quality score for data assets, data products, and governance domains (one governance domain can have many data products, one data product can have many data assets, one data asset can have many data columns).
• Data quality alerts
  • Configure alerts to notify data owners and data stewards if data quality threshold missed the expectation.
  • Configure email alias or distribution group to send the notification about data quality issues.
• Data quality actions
  • Actions center for data quality with actions to address data quality anomaly states, including diagnostic queries for data quality steward to zero in on the specific data to fix for each anomaly state.
• Data quality managed virtual network
  • A virtual network managed by data quality that connects with private endpoints to your Microsoft Azure data sources.

Data residency and encryption

Microsoft Managed Storage account stores data quality metadata and profiling summary. It stores them in the same region as the data source, so data residency remains intact. All data is encrypted. The Purview Resource Provider regional user data store is used for metadata. It handles all the encryption and is common across all Purview services. If you want more control over your data encryption with a customer-managed encryption key (CMK), use a separate process. Learn more about Microsoft Purview Customer Key.

Data quality compute pricing

Data quality usage is billed based on the Data Governance Processing Unit (DGPU) pay-as-you-go meters. Find details on how pricing is computed for data quality.

Limitation

• Virtual network isn't supported for Google Big Query yet.

Related content

• Data quality for Fabric data estate
• Data quality for Fabric Mirrored data sources
• Data quality for Fabric shortcut data sources
• Data quality for Azure Synapse serverless and data warehouses
• Data quality for Azure Databricks Unity Catalog
• Data quality for Snowflake data sources
• Data quality for Google Big Query
• Data quality native support for iceberg data

Next steps

1. Assign users data quality steward permissions in Unified Catalog so they can use all data quality features.
2. Set up a data source connection to prepare your source for a data quality assessment.
3. Configure and run data profiling for an asset in your data source.