Share via

SimilarityEvaluator Class

Evaluates similarity score for a given query, response, and ground truth.

The similarity measure evaluates the likeness between a ground truth sentence (or document) and the AI model's generated prediction. This calculation involves creating sentence-level embeddings for both the ground truth and the model's prediction, which are high-dimensional vector representations capturing the semantic meaning and context of the sentences.

Use it when you want an objective evaluation of an AI model's performance, particularly in text generation tasks where you have access to ground truth responses. Similarity enables you to assess the generated text's semantic alignment with the desired content, helping to gauge the model's quality and accuracy.

Similarity scores range from 1 to 5, with 1 being the least similar and 5 being the most similar.

Note

To align with our support of a diverse set of models, an output key without the gpt_ prefix has been added.

To maintain backwards compatibility, the old key with the gpt_ prefix is still be present in the output;

however, it is recommended to use the new key moving forward as the old key will be deprecated in the future.

Constructor

SimilarityEvaluator(model_config, *, threshold=3, credential=None)

Parameters

Name Description
model_config
Required
Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration]

Configuration for the Azure OpenAI model.
threshold
Required
int

The threshold for the similarity evaluator. Default is 3.

Keyword-Only Parameters

Name Description
threshold
Default value: 3
credential
Default value: None

Examples

Initialize with a threshold and call a SimilarityEvaluator.


   import os
   from azure.ai.evaluation import SimilarityEvaluator

   model_config = {
       "azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),
       "api_key": os.environ.get("AZURE_OPENAI_KEY"),
       "azure_deployment": os.environ.get("AZURE_OPENAI_DEPLOYMENT"),
   }

   similarity_eval = SimilarityEvaluator(model_config=model_config, threshold=3)
   similarity_eval(
       query="What is the capital of Japan?",
       response="The capital of Japan is Tokyo.",
       ground_truth="Tokyo is Japan's capital.",
   )

Attributes

id

Evaluator identifier, experimental and to be used only with evaluation in cloud.

id = 'azureai://built-in/evaluators/similarity'