Share via

Request example for multigrader with custom grader in Reinforcement fine tuning in AI foundry

Oliver Su (Artech Consulting LLC) 20 Reputation points Microsoft Employee
2025-10-24T18:52:49.33+00:00

Hi there, my custom grader is working fine when using it separately but when i combine with it with other in-built grader, it always failed. In the website tutorial, there is no template for multiple grader which includes custom grader, could u give an example please?

This is what i have. 

{
"name":"sample_multi_grader",
"type":"multi",
"graders":{"ext_text_similarity":{"name":"ext_text_similarity",
"type":"text_similarity",
"input":"{{sample.output_json.ext_text}}",
"reference":"{{item.ext_text}}",
"evaluation_metric":"fuzzy_match"},

"custom_check":{
"type":"python",
"source":"{import re ....}",
}
},
"calculate_output":"0.5 * ext_text_similarity + 0.5 * custom_check"
}
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Azar 30,735 Reputation points MVP Volunteer Moderator
    2025-10-24T19:19:45.3366667+00:00

    Hi there Oliver Su (Artech Consulting LLC)

    Thanks for using QandA platform

    the multi-grader setup in Azure AI Foundry is a bit picky when combining built-in and custom graders. The main thing to check is that each grader inside your graders block explicitly defines both input and reference, even for the custom Python grader. Also, make sure the names you use in calculate_output exactly match the grader keys. For example, you can structure it like this: one grader for text similarity and another for your custom check, then combine them with something like "calculate_output": "0.5 * ext_text_similarity + 0.5 * custom_check". The custom grader’s source should return a numeric value (like 0 or 1). Once you align those details, it should work fin

    If this helps kindly accept the answer

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.