ResponseCompletenessEvaluator Class
Note
This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Evaluates the extent to which a given response contains all necessary and relevant information with respect to the provided ground truth.
The completeness measure assesses how thoroughly an AI model's generated response aligns with the key information, claims, and statements established in the ground truth. This evaluation considers the presence, accuracy, and relevance of the content provided. The assessment spans multiple levels, ranging from fully incomplete to fully complete, ensuring a comprehensive evaluation of the response's content quality. Use this metric when you need to evaluate an AI model's ability to deliver comprehensive and accurate information, particularly in text generation tasks where conveying all essential details is crucial for clarity, context, and correctness. Completeness scores range from 1 to 5: 1: Fully incomplete — Contains none of the necessary information. 2: Barely complete — Contains only a small portion of the required information. 3: Moderately complete — Covers about half of the required content. 4: Mostly complete — Includes most of the necessary details with minimal omissions. 5: Fully complete — Contains all key information without any omissions. :param model_config: Configuration for the Azure OpenAI model. :type model_config: Union[~azure.ai.evaluation.AzureOpenAIModelConfiguration,
~azure.ai.evaluation.OpenAIModelConfiguration]
Constructor
ResponseCompletenessEvaluator(model_config, *, threshold: float | None = 3, credential=None, **kwargs)Parameters
| Name | Description |
|---|---|
|
model_config
Required
|
|
Keyword-Only Parameters
| Name | Description |
|---|---|
|
threshold
|
Default value: 3
|
|
credential
|
Default value: None
|
Examples
Initialize and call CompletenessEvaluator using Azure AI Project URL in the following format https://{resource_name}.services.ai.azure.com/api/projects/{project_name}
import os
from azure.ai.evaluation import CompletenessEvaluator
model_config = {
"azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"), # https://<account_name>.services.ai.azure.com
"api_key": os.environ.get("AZURE_OPENAI_KEY"),
"azure_deployment": os.environ.get("AZURE_OPENAI_DEPLOYMENT"),
}
completeness_eval = CompletenessEvaluator(model_config=model_config)
completeness_eval(
response="The capital of Japan is Tokyo.",
ground_truth="Tokyo is Japan's capital.",
)
Attributes
id
id = 'azureai://built-in/evaluators/response_completeness'