Share via

TaskAdherenceEvaluator Class

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

The Task Adherence evaluator assesses how well an AI-generated response follows the assigned task based on:

  • Alignment with instructions and definitions

  • Accuracy and clarity of the response

  • Proper use of provided tool definitions

Scoring is based on five levels:

  1. Fully Inadherent - Response completely ignores instructions.
  2. Barely Adherent - Partial alignment with critical gaps.
  3. Moderately Adherent - Meets core requirements but lacks precision.
  4. Mostly Adherent - Clear and accurate with minor issues.
  5. Fully Adherent - Flawless adherence to instructions.

The evaluation includes a step-by-step reasoning process, a brief explanation, and a final integer score.

Constructor

TaskAdherenceEvaluator(model_config, *, threshold=3, credential=None, **kwargs)

Parameters

Name Description
model_config
Required
Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration]

Configuration for the Azure OpenAI model.

Keyword-Only Parameters

Name Description
threshold
Default value: 3
credential
Default value: None

Examples

Initialize and call TaskAdherenceEvaluator using Azure AI Project URL in the following format https://{resource_name}.services.ai.azure.com/api/projects/{project_name}


   import os
   from azure.ai.evaluation import TaskAdherenceEvaluator

   model_config = {
       "azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),  # https://<account_name>.services.ai.azure.com
       "api_key": os.environ.get("AZURE_OPENAI_KEY"),
       "azure_deployment": os.environ.get("AZURE_OPENAI_DEPLOYMENT"),
   }

   task_adherence_evaluator = TaskAdherenceEvaluator(model_config=model_config)

   query = [
       {"role": "system", "content": "You are a helpful customer service agent."},
       {"role": "user", "content": [{"type": "text", "text": "What is the status of my order #123?"}]},
   ]

   response = [
       {
           "role": "assistant",
           "content": [
               {
                   "type": "tool_call",
                   "tool_call": {
                       "id": "tool_001",
                       "type": "function",
                       "function": {"name": "get_order", "arguments": {"order_id": "123"}},
                   },
               }
           ],
       },
       {
           "role": "tool",
           "tool_call_id": "tool_001",
           "content": [
               {"type": "tool_result", "tool_result": '{ "order": { "id": "123", "status": "shipped" } }'}
           ],
       },
       {"role": "assistant", "content": [{"type": "text", "text": "Your order #123 has been shipped."}]},
   ]

   tool_definitions = [
       {
           "name": "get_order",
           "description": "Get order details.",
           "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}},
       }
   ]

   task_adherence_evaluator(query=query, response=response, tool_definitions=tool_definitions)

Attributes

id

Evaluator identifier, experimental and to be used only with evaluation in cloud.

id = 'azureai://built-in/evaluators/task_adherence'