Share via

RelevanceEvaluator Class

Evaluates relevance score for a given query and response or a multi-turn conversation, including reasoning.

The relevance measure assesses the ability of answers to capture the key points of the context. High relevance scores signify the AI system's understanding of the input and its capability to produce coherent and contextually appropriate outputs. Conversely, low relevance scores indicate that generated responses might be off-topic, lacking in context, or insufficient in addressing the user's intended queries. Use the relevance metric when evaluating the AI system's performance in understanding the input and generating contextually appropriate responses.

Relevance scores range from 1 to 5, with 1 being the worst and 5 being the best.

Note

To align with our support of a diverse set of models, an output key without the gpt_ prefix has been added.

To maintain backwards compatibility, the old key with the gpt_ prefix is still be present in the output;

however, it is recommended to use the new key moving forward as the old key will be deprecated in the future.

Constructor

RelevanceEvaluator(model_config, *, credential=None, threshold=3)

Parameters

Name Description
model_config
Required
Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration]

Configuration for the Azure OpenAI model.
threshold
Required
int

The threshold for the relevance evaluator. Default is 3.

Keyword-Only Parameters

Name Description
credential
Default value: None
threshold
Default value: 3

Examples

Initialize with threshold and call a RelevanceEvaluator with a query, response, and context.


   import os
   from azure.ai.evaluation import RelevanceEvaluator

   model_config = {
       "azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),
       "api_key": os.environ.get("AZURE_OPENAI_KEY"),
       "azure_deployment": os.environ.get("AZURE_OPENAI_DEPLOYMENT"),
   }

   relevance_eval = RelevanceEvaluator(model_config=model_config, threshold=2)
   relevance_eval(
       query="What is the capital of Japan?",
       response="The capital of Japan is Tokyo.",
   )

Attributes

id

Evaluator identifier, experimental and to be used only with evaluation in cloud.

id = 'azureai://built-in/evaluators/relevance'