Azure AI Evaluation: Unable to run custom evaluator in Foundry Portal

[ManniArora]

Describe the bug

I am unable to run my custom evaluator through the Foundry portal. When triggering an evaluation run, the following error appears:

"Please provide a valid evaluator. Invalid or missing numeric result."

I was able to:

• Successfully create the evaluator in the portal
• Configure a new eval run using this evaluator
• Start the eval run, but it fails with the above error

No detailed logs are available to indicate what went wrong.

Region - westus

To Reproduce

1. Create a custom evaluator on the Foundry portal
2. Configure an eval run and select this evaluator
3. Start the eval run
4. Observe failure with the error above

Expected behavior

The evaluation should run successfully using the custom evaluator, or at minimum provide detailed logs indicating what part of the evaluator output is invalid.

Screenshots

Additional context

It appears the evaluator is being detected and invoked, but the system rejects its output due to an “invalid or missing numeric result.” More verbose logging or an example of the required return format would be helpful for debugging.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.

[github-actions]

Hi @ManniArora, I'm an AI Support assistant here to help with your issue. While the Azure SDK team reviews your request, I wanted to provide some possible tips and documentation that might help you in the meantime.

• Ensure the evaluator's output includes a numeric value in the expected location. The error indicates that the evaluator's result is either missing or of an incorrect type.
• Verify that your evaluator returns the result data with the proper numeric type. If a string is returned instead, this error may be triggered.
• Check that the evaluator's implementation adheres to the expected result schema outlined in our model inference API documentation: Azure AI Model Inference API.
• Consider adding diagnostic logging (in your evaluator implementation) to capture the output prior to sending it back to the Foundry portal to assist in isolating the mismatch.

The Azure SDK team will respond to your issue shortly. I hope these suggestions are helpful in the meantime. If this comment helped you, please give it a 👍. If the suggestion was not helpful or incorrect, please give it a 👎. Your feedback helps us improve!

[w-javed]

We need more information to reproduce the issue. Can you provide sample prompt which is failing for you.

Meanwhile, please refer to the sample code for prompt based custom evaluator and its evaluations.
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_prompt_based_evaluators.py

Above sample provide python script to create prompt based custom evaluator and use it in the evaluation APIs.

[ManniArora]

Evaluator Prompt:

You are a Test Plan Accuracy Evaluator.
Your task is to evaluate how accurately the generated test plan matches the expected test plan by comparing test IDs/names.
Focus ONLY on test names/IDs — ignore other test metadata like descriptions, steps, expected results, priorities, etc.
---
     ### Input:
     Query (Test Plan Request):
     {{query}}

     Response (Generated Test Plan):
     {{response}}

    Ground Truth (Expected Test Plan):
    {{ground_truth}}
---
    ### Evaluation Instructions:
1. Extract all test IDs/names from the Response
2. Extract all test IDs/names from the Ground Truth
3. Compare the two sets and calculate match percentage

### Scoring Scale (1–5):
5 → Perfect match. All test names in ground truth are present in response, and no extra tests.
4 → High accuracy. 80-99% of test names match (minor missing or extra tests).
3 → Moderate accuracy. 50-79% of test names match.
2 → Low accuracy. 20-49% of test names match.
1 → Poor accuracy. Less than 20% of test names match or completely different tests.

---
     ### Output Format (JSON):
 {
 "result": <integer from 1 to 5>,
"reason": "<brief explanation for the score>"
  }