MeteorScoreEvaluator Class
Calculates the METEOR score for a given response and ground truth.
The METEOR (Metric for Evaluation of Translation with Explicit Ordering) score grader evaluates generated text by comparing it to reference texts, focusing on precision, recall, and content alignment. It addresses limitations of other metrics like BLEU by considering synonyms, stemming, and paraphrasing. METEOR score considers synonyms and word stems to more accurately capture meaning and language variations. In addition to machine translation and text summarization, paraphrase detection is an optimal use case for the METEOR score.
Use the METEOR score when you want a more linguistically informed evaluation metric that captures not only n-gram overlap but also accounts for synonyms, stemming, and word order. This is particularly useful for evaluating tasks like machine translation, text summarization, and text generation.
The METEOR score ranges from 0 to 1, with 1 indicating a perfect match.
Constructor
MeteorScoreEvaluator(alpha: float = 0.9, beta: float = 3.0, gamma: float = 0.5, *, threshold: float = 0.5)Parameters
| Name | Description |
|---|---|
|
alpha
|
The METEOR score alpha parameter. Default is 0.9. Default value: 0.9
|
|
beta
|
The METEOR score beta parameter. Default is 3.0. Default value: 3.0
|
|
gamma
|
The METEOR score gamma parameter. Default is 0.5. Default value: 0.5
|
|
threshold
Required
|
The threshold for the METEOR score evaluator. Default is 0.5. |
Keyword-Only Parameters
| Name | Description |
|---|---|
|
threshold
|
Default value: 0.5
|
Examples
Initialize with threshold and call a MeteorScoreEvaluator.
from azure.ai.evaluation import MeteorScoreEvaluator
meteor_evaluator = MeteorScoreEvaluator(alpha=0.8, threshold=0.3)
meteor_evaluator(response="Paris is the capital of France.", ground_truth="France's capital is Paris.")
Attributes
id
Evaluator identifier, experimental and to be used only with evaluation in cloud.
id = 'azureai://built-in/evaluators/meteor_score'