尚拙
Search K

Appearance

智能体评测 (Agent Evals) ​

使用 AgentEvals 和 LangSmith，通过确定性匹配或大模型裁判（LLM-as-judge）评测器评估智能体执行轨迹。

评测（Evaluations，简称 "evals"）通过评估智能体的执行轨迹——即其产生的消息和工具调用序列——来衡量智能体的表现。与验证基本正确性的集成测试不同，评测会根据参考标准（reference）或评分细则（rubric）对智能体行为进行打分，这使得它们在更改提示词、工具或模型时非常有助于捕获回归问题。

评测器是一个接收智能体输出（以及可选的参考输出）并返回分数的函数：

typescript
function evaluator({ outputs, referenceOutputs }: {
  outputs: Record<string, any>;
  referenceOutputs: Record<string, any>;
}) {
  const outputMessages = outputs.messages;
  const referenceMessages = referenceOutputs.messages;
  const score = compareMessages(outputMessages, referenceMessages);
  return { key: "evaluator_score", score: score };
}

agentevals 软件包为智能体轨迹提供了预构建的评测器。你可以通过执行轨迹匹配（确定性比对）或使用 LLM 裁判（定性评估）来进行评测：

评估方法适用场景
轨迹匹配已知预期的工具调用，并希望进行快速、确定性且零成本的检查
大模型裁判 (LLM-as-judge)希望评估整体质量和推理能力，而无需设定严格的预期

安装 AgentEvals ​

bash
npm install agentevals @langchain/core

或者，直接克隆 AgentEvals 仓库。

轨迹匹配评测器 ​

AgentEvals 提供了 createTrajectoryMatchEvaluator 函数，用于将智能体的轨迹与参考轨迹进行匹配。它包含四种模式：

模式描述使用场景
strict严格匹配相同顺序的消息结构和工具调用（消息内容可以不同）测试特定序列（例如在授权操作前查询策略）
unordered与参考具有相同的消息结构和工具调用，但工具调用可以按任意顺序发生验证信息检索，且顺序无关紧要的场景
subset智能体仅调用参考轨迹中的工具（不允许额外调用）确保智能体不会超出预期范围
superset智能体至少调用参考轨迹中的工具（允许额外调用）验证已执行最少所需的必要操作

下面的示例共享相同的初始设置，即一个带有 get_weather 工具的智能体：

ts
import { createAgent } from "langchain";
import { tool } from "@langchain/core/tools";
import { HumanMessage, AIMessage, ToolMessage } from "@langchain/core/messages";
import { createTrajectoryMatchEvaluator } from "agentevals";
import * as z from "zod";

const getWeather = tool(
  async ({ city }) => {
    return `It's 75 degrees and sunny in ${city}.`;
  },
  {
    name: "get_weather",
    description: "Get weather information for a city.",
    schema: z.object({ city: z.string() }),
  }
);

const agent = createAgent({
  model: "claude-sonnet-4-6",
  tools: [getWeather],
});

严格匹配 (Strict match)

strict 模式确保轨迹以相同的顺序包含完全相同的消息和相同的工具调用，尽管它允许消息内容存在差异。当你需要强制执行特定的操作顺序时（例如要求在授权某项操作之前必须先查询策略），这非常有用。

ts
const evaluator = createTrajectoryMatchEvaluator({  
  trajectoryMatchMode: "strict",  
});  

async function testWeatherToolCalledStrict() {
  const result = await agent.invoke({
    messages: [new HumanMessage("What's the weather in San Francisco?")]
  });

  const referenceTrajectory = [
    new HumanMessage("What's the weather in San Francisco?"),
    new AIMessage({
      content: "",
      tool_calls: [
        { id: "call_1", name: "get_weather", args: { city: "San Francisco" } }
      ]
    }),
    new ToolMessage({
      content: "It's 75 degrees and sunny in San Francisco.",
      tool_call_id: "call_1"
    }),
    new AIMessage("The weather in San Francisco is 75 degrees and sunny."),
  ];

  const evaluation = await evaluator({
    outputs: result.messages,
    referenceOutputs: referenceTrajectory
  });
  expect(evaluation.score).toBe(true);
}

无序匹配 (Unordered match)

unordered 模式允许以任意顺序进行相同的工具调用。当你想要验证是否检索了特定信息，但并不关心其调用顺序时，这很有帮助。例如，智能体通过不同的工具调用分别查询某城市的天气和活动。

ts
const getEvents = tool(
  async ({ city }: { city: string }) => {
    return `Concert at the park in ${city} tonight.`;
  },
  {
    name: "get_events",
    description: "Get events happening in a city.",
    schema: z.object({ city: z.string() }),
  }
);

const agent = createAgent({
  model: "claude-sonnet-4-6",
  tools: [getWeather, getEvents],
});

const evaluator = createTrajectoryMatchEvaluator({  
  trajectoryMatchMode: "unordered",  
});  

async function testMultipleToolsAnyOrder() {
  const result = await agent.invoke({
    messages: [new HumanMessage("What's happening in SF today?")]
  });

  const referenceTrajectory = [
    new HumanMessage("What's happening in SF today?"),
    new AIMessage({
      content: "",
      tool_calls: [
        { id: "call_1", name: "get_events", args: { city: "SF" } },
        { id: "call_2", name: "get_weather", args: { city: "SF" } },
      ]
    }),
    new ToolMessage({
      content: "Concert at the park in SF tonight.",
      tool_call_id: "call_1"
    }),
    new ToolMessage({
      content: "It's 75 degrees and sunny in SF.",
      tool_call_id: "call_2"
    }),
    new AIMessage("Today in SF: 75 degrees and sunny with a concert at the park tonight."),
  ];

  const evaluation = await evaluator({
    outputs: result.messages,
    referenceOutputs: referenceTrajectory,
  });
  expect(evaluation.score).toBe(true);
}

子集与超集匹配 (Subset and superset match)

superset 和 subset 模式用于匹配部分轨迹。superset 模式验证智能体至少调用了参考轨迹中的工具，并允许进行额外的工具调用。subset 模式则确保智能体没有调用参考轨迹之外的任何工具。

ts
const getDetailedForecast = tool(
  async ({ city }: { city: string }) => {
    return `Detailed forecast for ${city}: sunny all week.`;
  },
  {
    name: "get_detailed_forecast",
    description: "Get detailed weather forecast for a city.",
    schema: z.object({ city: z.string() }),
  }
);

const agent = createAgent({
  model: "claude-sonnet-4-6",
  tools: [getWeather, getDetailedForecast],
});

const evaluator = createTrajectoryMatchEvaluator({  
  trajectoryMatchMode: "superset",  
});  

async function testAgentCallsRequiredToolsPlusExtra() {
  const result = await agent.invoke({
    messages: [new HumanMessage("What's the weather in Boston?")]
  });

  const referenceTrajectory = [
    new HumanMessage("What's the weather in Boston?"),
    new AIMessage({
      content: "",
      tool_calls: [
        { id: "call_1", name: "get_weather", args: { city: "Boston" } },
      ]
    }),
    new ToolMessage({
      content: "It's 75 degrees and sunny in Boston.",
      tool_call_id: "call_1"
    }),
    new AIMessage("The weather in Boston is 75 degrees and sunny."),
  ];

  const evaluation = await evaluator({
    outputs: result.messages,
    referenceOutputs: referenceTrajectory,
  });
  expect(evaluation.score).toBe(true);
}

NOTE

你还可以设置 toolArgsMatchMode 属性和/或 toolArgsMatchOverrides，以自定义评测器如何判定实际轨迹与参考轨迹中工具调用之间的相等性。默认情况下，只有调用相同工具且传入相同参数的工具调用才会被视为相等。请访问代码仓库了解更多详情。

大模型裁判评测器 ​

你可以使用 createTrajectoryLLMAsJudge 函数，利用 LLM 来评估智能体的执行路径。与轨迹匹配评测器不同，它不强制要求参考轨迹，但如果已有参考轨迹也可以传入。

无参考轨迹

ts
import { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT } from "agentevals";

const evaluator = createTrajectoryLLMAsJudge({  
  model: "openai:o3-mini",  
  prompt: TRAJECTORY_ACCURACY_PROMPT,  
});  

async function testTrajectoryQuality() {
  const result = await agent.invoke({
    messages: [new HumanMessage("What's the weather in Seattle?")]
  });

  const evaluation = await evaluator({
    outputs: result.messages,
  });
  expect(evaluation.score).toBe(true);
}

包含参考轨迹

如果你拥有参考轨迹，请使用预构建的 TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE 提示词：

ts
import { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE } from "agentevals";

const evaluator = createTrajectoryLLMAsJudge({
  model: "openai:o3-mini",
  prompt: TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,
});

const evaluation = await evaluator({
  outputs: result.messages,
  referenceOutputs: referenceTrajectory,
});

NOTE

如需了解关于如何让 LLM 评测轨迹的更多可配置项，请访问代码仓库。

在 LangSmith 中运行评测 ​

为了长期跟踪实验，可以将评测器结果记录到 LangSmith。首先，设置所需的环境变量：

bash
export LANGSMITH_API_KEY="your_langsmith_api_key"
export LANGSMITH_TRACING="true"

LangSmith 提供了两种运行评测的主要方式：Vitest/Jest 集成和 evaluate 函数。

使用 Vitest/Jest 集成

ts
import * as ls from "langsmith/vitest";
// import * as ls from "langsmith/jest";

import { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT } from "agentevals";

const trajectoryEvaluator = createTrajectoryLLMAsJudge({
  model: "openai:o3-mini",
  prompt: TRAJECTORY_ACCURACY_PROMPT,
});

ls.describe("trajectory accuracy", () => {
  ls.test("accurate trajectory", {
    inputs: {
      messages: [
        { role: "user", content: "What is the weather in SF?" }
      ]
    },
    referenceOutputs: {
      messages: [
        new HumanMessage("What is the weather in SF?"),
        new AIMessage({
          content: "",
          tool_calls: [
            { id: "call_1", name: "get_weather", args: { city: "SF" } }
          ]
        }),
        new ToolMessage({
          content: "It's 75 degrees and sunny in SF.",
          tool_call_id: "call_1"
        }),
        new AIMessage("The weather in SF is 75 degrees and sunny."),
      ],
    },
  }, async ({ inputs, referenceOutputs }) => {
    const result = await agent.invoke({
      messages: [new HumanMessage("What is the weather in SF?")]
    });

    ls.logOutputs({ messages: result.messages });

    await trajectoryEvaluator({
      inputs,
      outputs: result.messages,
      referenceOutputs,
    });
  });
});

使用你的测试运行器运行评测：

bash
vitest run test_trajectory.eval.ts
# 或者
jest test_trajectory.eval.ts

使用 evaluate 函数

创建一个 LangSmith 数据集并使用 evaluate 函数。该数据集必须符合以下模式（schema）：

  • input：{"messages": [...]} 用于调用智能体的输入消息。
  • output：{"messages": [...]} 智能体输出中预期的历史消息。对于轨迹评测，你可以选择仅保留助手（assistant）消息。
ts
import { evaluate } from "langsmith/evaluation";
import { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT } from "agentevals";

const trajectoryEvaluator = createTrajectoryLLMAsJudge({
  model: "openai:o3-mini",
  prompt: TRAJECTORY_ACCURACY_PROMPT,
});

async function runAgent(inputs: any) {
  const result = await agent.invoke(inputs);
  return result.messages;
}

await evaluate(
  runAgent,
  {
    data: "your_dataset_name",
    evaluators: [trajectoryEvaluator],
  }
);

TIP

欲了解更多关于评测智能体的信息，请参阅 LangSmith 文档。