[azure-ai-projects] Evaluation sample sample_agent_evaluation.py has bug

[kuojianlu]

• Package Name: azure-ai-projects
• Package Version:
• Operating System:
• Python Version:

Describe the bug
When run this sample, error: INVALID VALUE: (UserError) Missing inputs for line 1: 'data.response'. This is because the data mapping is wrong: "response": "{{item.response}}"}, thus actually there is no response in item. The correct mapping should be "response": "{{sample.output_text}}".

BTW, I have one question for custom code-based evaluator. Do you have a full documentation on the custom evaluators? I wonder how the parameters should be and how to get responses of agent. I ran an evaluation with custom code based evaluator targeted on an agent, but I cannot get the responses of the agent. I tried sample.get("output_text"), sample.output_text. Code of my evaluator:

def grade(sample, item) -> float:
    try:
        response = sample.get("output_text")
        return 1.0 if len(response) < 100 else 0.0
    except:
        return 0.1

To Reproduce
Steps to reproduce the behavior:
1.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dargilco @glharper @howieleung @kingernupur @nick863 @trangevi @trrwilson.

[dargilco]

Thank you @kuojianlu for creating this GitHub issue. Which version of azure-ai-projects are you using?

I contacted the Evaluation team to have someone address this issue.

[kuojianlu]

Thanks, the version is 2.0.0b2

[kuojianlu]

@w-javed Any update?