TRAE-AI编程
OpenAI 使用 Evals
2026-11-09 13:55 更新
评测（evaluation，常简称 eval）用于检验模型输出是否符合指定的风格与内容标准。通过评测了解 LLM 应用与预期的差距，尤其是升级或试用新模型时，是构建可靠应用的重要步骤。
本指南重点介绍如何用 Evals API以编程方式配置评测，也可以在 OpenAI 控制台中操作。
OpenAI 正弃用 Evals 平台。过渡期内既有内容仍可访问；官方时间表列明既有用户自2026年10月31日起进入只读状态，平台计划于2026年11月30日关闭。详见弃用时间表。
刚接触评测或需要便于边构建边试验的交互环境时，可先使用数据集。为 LLM 应用构建并运行评测大致分为三步：
- 用评测定义要完成的任务。
- 使用测试输入（提示词和数据）运行评测。
- 分析结果，迭代改进提示词。
这与行为驱动开发（BDD）相似：先规定系统应如何表现，再实现并测试。下面使用 Evals API 完成这一流程。
为任务创建评测
先描述模型要做的任务。例如，将 IT 支持工单分成Hardware（硬件）、Software（软件）或Other（其他）。可用 Chat Completions API或 Responses API实现。下面的示例都结合开发者消息与包含工单内容的用户消息。
来源示例 1（javascript）：
// [中文] 将IT支持工单分类为Hardware（硬件）、Software（软件）或Other（其他），要求只返回一个英文标签；工单是显示器无法开机。
import OpenAI from "openai";
const client = new OpenAI();
const instructions = `
You are an expert in categorizing IT support tickets. Given the support
ticket below, categorize the request into one of "Hardware", "Software",
or "Other". Respond with only one of those words.
`;
const ticket = "My monitor won't turn on - help!";
const response = await client.responses.create({
model: "gpt-6-astra",
input: [
{ role: "developer", content: instructions },
{ role: "user", content: ticket },
],
});
console.log(response.output_text);来源示例 2（python）：
## [中文] 将IT支持工单分类为Hardware（硬件）、Software（软件）或Other（其他），要求只返回一个英文标签；工单是显示器无法开机。
from openai import OpenAI
client = OpenAI()
instructions = """
You are an expert in categorizing IT support tickets. Given the support
ticket below, categorize the request into one of "Hardware", "Software",
or "Other". Respond with only one of those words.
"""
ticket = "My monitor won't turn on - help!"
response = client.responses.create(
model="gpt-6-astra",
input=[
{"role": "developer", "content": instructions},
{"role": "user", "content": ticket},
],
)
print(response.output_text)来源示例 3（go）：
// [中文] 将IT支持工单分类为Hardware（硬件）、Software（软件）或Other（其他），要求只返回一个英文标签；工单是显示器无法开机。
package main
import (
"context"
"fmt"
"github.com/openai/openai-go/v3"
"github.com/openai/openai-go/v3/responses"
)
func main() {
client := openai.NewClient()
instructions := "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of \"Hardware\", \"Software\", or \"Other\". Respond with only one of those words."
response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{
Model: "gpt-6-astra",
Input: responses.ResponseNewParamsInputUnion{OfInputItemList: responses.ResponseInputParam{
responses.ResponseInputItemParamOfMessage(instructions, responses.EasyInputMessageRoleDeveloper),
responses.ResponseInputItemParamOfMessage("My monitor won't turn on - help!", responses.EasyInputMessageRoleUser),
}},
})
if err != nil {
panic(err)
}
fmt.Println(response.OutputText())
}来源示例 4（java）：
// [中文] 将IT支持工单分类为Hardware（硬件）、Software（软件）或Other（其他），要求只返回一个英文标签；工单是显示器无法开机。
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.responses.EasyInputMessage;
import com.openai.models.responses.ResponseCreateParams;
import com.openai.models.responses.ResponseInputItem;
import java.util.List;
ResponseCreateParams params =
ResponseCreateParams.builder()
.model("gpt-6-astra")
.inputOfResponse(
List.of(
ResponseInputItem.ofEasyInputMessage(
EasyInputMessage.builder()
.role(EasyInputMessage.Role.DEVELOPER)
.content(
"You are an expert in categorizing IT support tickets. Categorize each request as Hardware, Software, or Other. Respond with only one of those words.")
.build()),
ResponseInputItem.ofEasyInputMessage(
EasyInputMessage.builder()
.role(EasyInputMessage.Role.USER)
.content("My monitor won't turn on - help!")
.build())))
.build();
client.responses().create(params).output().stream()
.flatMap(item -> item.message().stream())
.flatMap(message -> message.content().stream())
.flatMap(content -> content.outputText().stream())
.forEach(text -> System.out.println(text.text()));来源示例 5（csharp）：
// [中文] 将IT支持工单分类为Hardware（硬件）、Software（软件）或Other（其他），要求只返回一个英文标签；工单是显示器无法开机。
using OpenAI.Responses;
#pragma warning disable OPENAI001
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
ResponsesClient client = new(key);
ResponseResult response = await client.CreateResponseAsync(
"gpt-6-astra",
[
ResponseItem.CreateDeveloperMessageItem(
"Categorize the IT support ticket as Hardware, Software, or Other. Respond with only one of those words."
),
ResponseItem.CreateUserMessageItem("My monitor will not turn on. Help!"),
]
);
Console.WriteLine(response.GetOutputText());来源示例 6（ruby）：
## [中文] 将IT支持工单分类为Hardware（硬件）、Software（软件）或Other（其他），要求只返回一个英文标签；工单是显示器无法开机。
require "openai"
client = OpenAI::Client.new
instructions = <<~INSTRUCTIONS
You are an expert in categorizing IT support tickets. Given the support
ticket below, categorize the request as Hardware, Software, or Other.
Respond with only one of those words.
INSTRUCTIONS
response = client.responses.create(
model: "gpt-6-astra",
input: [
{
role: :developer,
content: instructions
},
{
role: :user,
content: "My monitor won't turn on - help!"
}
]
)
puts(response.output_text)来源示例 7（bash）：
## [中文] 将IT支持工单分类为Hardware（硬件）、Software（软件）或Other（其他），要求只返回一个英文标签；工单是显示器无法开机。
curl https://api.openai.com/v1/responses \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-6-astra",
"input": [
{
"role": "developer",
"content": "Categorize the following support ticket into one of Hardware, Software, or Other."
},
{
"role": "user",
"content": "My monitor wont turn on - help!"
}
]
}'同页HTML还提供Chat Completions版本，完整保留如下：
来源示例 8（javascript）：
// [中文] 同页HTML恢复的Chat Completions工单分类版本；只返回Hardware、Software或Other三个英文标签之一。
import OpenAI from "openai";
const client = new OpenAI();
const instructions = `
You are an expert in categorizing IT support tickets. Given the support
ticket below, categorize the request into one of "Hardware", "Software",
or "Other". Respond with only one of those words.
`;
const ticket = "My monitor won't turn on - help!";
const completion = await client.chat.completions.create({
model: "gpt-6-astra",
messages: [
{ role: "developer", content: instructions },
{ role: "user", content: ticket },
],
});
console.log(completion.choices[0].message.content);来源示例 9（python）：
## [中文] 同页HTML恢复的Chat Completions工单分类版本；只返回Hardware、Software或Other三个英文标签之一。
from openai import OpenAI
client = OpenAI()
instructions = """
You are an expert in categorizing IT support tickets. Given the support
ticket below, categorize the request into one of "Hardware", "Software",
or "Other". Respond with only one of those words.
"""
ticket = "My monitor won't turn on - help!"
completion = client.chat.completions.create(
model="gpt-6-astra",
messages=[
{"role": "developer", "content": instructions},
{"role": "user", "content": ticket},
],
)
print(completion.choices[0].message.content)来源示例 10（go）：
// [中文] 同页HTML恢复的Chat Completions工单分类版本；只返回Hardware、Software或Other三个英文标签之一。
package main
import (
"context"
"fmt"
"github.com/openai/openai-go/v3"
)
func main() {
client := openai.NewClient()
instructions := "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of \"Hardware\", \"Software\", or \"Other\". Respond with only one of those words."
completion, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
Model: "gpt-6-astra",
Messages: []openai.ChatCompletionMessageParamUnion{
openai.DeveloperMessage(instructions),
openai.UserMessage("My monitor won't turn on - help!"),
},
})
if err != nil {
panic(err)
}
fmt.Println(completion.Choices[0].Message.Content)
}来源示例 11（java）：
// [中文] 同页HTML恢复的Chat Completions工单分类版本；只返回Hardware、Software或Other三个英文标签之一。
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.chat.completions.ChatCompletionCreateParams;
ChatCompletionCreateParams params =
ChatCompletionCreateParams.builder()
.model("gpt-6-astra")
.addDeveloperMessage(
"You are an expert in categorizing IT support tickets. Given the support "
+ "ticket below, categorize the request as Hardware, Software, or Other. "
+ "Respond with only one of those words.")
.addUserMessage("My monitor won't turn on - help!")
.build();
client.chat().completions().create(params).choices().stream()
.flatMap(choice -> choice.message().content().stream())
.forEach(System.out::println);来源示例 12（csharp）：
// [中文] 同页HTML恢复的Chat Completions工单分类版本；只返回Hardware、Software或Other三个英文标签之一。
using OpenAI.Chat;
#pragma warning disable OPENAI001
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "gpt-6-astra";
ChatClient client = new(model, key);
string instructions =
"Categorize the IT support ticket as Hardware, Software, or Other. Respond with only one of those words.";
ChatCompletion completion = await client.CompleteChatAsync(
[
new DeveloperChatMessage(instructions),
new UserChatMessage("My monitor will not turn on. Help!"),
]
);
Console.WriteLine(completion.Content[0].Text);来源示例 13（ruby）：
## [中文] 同页HTML恢复的Chat Completions工单分类版本；只返回Hardware、Software或Other三个英文标签之一。
require "openai"
client = OpenAI::Client.new
instructions = <<~INSTRUCTIONS
You are an expert in categorizing IT support tickets. Given the support
ticket below, categorize the request as Hardware, Software, or Other.
Respond with only one of those words.
INSTRUCTIONS
completion = client.chat.completions.create(
model: "gpt-6-astra",
messages: [
{
role: :developer,
content: instructions
},
{
role: :user,
content: "My monitor won't turn on - help!"
}
]
)
puts(completion.choices.fetch(0).message.content)来源示例 14（bash）：
## [中文] 同页HTML恢复的Chat Completions工单分类版本；只返回Hardware、Software或Other三个英文标签之一。
curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-6-astra",
"messages": [
{
"role": "developer",
"content": "Categorize the following support ticket into one of Hardware, Software, or Other."
},
{
"role": "user",
"content": "My monitor wont turn on - help!"
}
]
}'接着通过 API建立评测。需要两项核心配置：
data_source_config：测试数据的模式。testing_criteria：判断模型输出是否正确的评分器。
来源示例 15（javascript）：
// [中文] 创建工单分类评测，定义ticket_text、correct_label和精确匹配评分；人工标准答案通过item引用，模型输出通过sample引用。
import OpenAI from "openai";
const openai = new OpenAI();
const evalObj = await openai.evals.create({
name: "IT Ticket Categorization",
data_source_config: {
type: "custom",
item_schema: {
type: "object",
properties: {
ticket_text: { type: "string" },
correct_label: { type: "string" },
},
required: ["ticket_text", "correct_label"],
},
include_sample_schema: true,
},
testing_criteria: [
{
type: "string_check",
name: "Match output to human label",
input: "{{ sample.output_text }}",
operation: "eq",
reference: "{{ item.correct_label }}",
},
],
});
console.log(evalObj);来源示例 16（python）：
## [中文] 创建工单分类评测，定义ticket_text、correct_label和精确匹配评分；人工标准答案通过item引用，模型输出通过sample引用。
from openai import OpenAI
client = OpenAI()
eval_obj = client.evals.create(
name="IT Ticket Categorization",
data_source_config={
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"ticket_text": {"type": "string"},
"correct_label": {"type": "string"},
},
"required": ["ticket_text", "correct_label"],
},
"include_sample_schema": True,
},
testing_criteria=[
{
"type": "string_check",
"name": "Match output to human label",
"input": "{{ sample.output_text }}",
"operation": "eq",
"reference": "{{ item.correct_label }}",
}
],
)
print(eval_obj)来源示例 17（java）：
// [中文] 创建工单分类评测，定义ticket_text、correct_label和精确匹配评分；人工标准答案通过item引用，模型输出通过sample引用。
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.evals.EvalCreateParams;
import com.openai.models.graders.gradermodels.StringCheckGrader;
import java.util.List;
import java.util.Map;
var eval =
client
.evals()
.create(
EvalCreateParams.builder()
.name("IT Ticket Categorization")
.dataSourceConfig(
EvalCreateParams.DataSourceConfig.Custom.builder()
.itemSchema(
EvalCreateParams.DataSourceConfig.Custom.ItemSchema.builder()
.additionalProperties(
Map.of(
"type", JsonValue.from("object"),
"properties",
JsonValue.from(
Map.of(
"ticket_text",
Map.of("type", "string"),
"correct_label",
Map.of("type", "string"))),
"required",
JsonValue.from(
List.of("ticket_text", "correct_label"))))
.build())
.includeSampleSchema(true)
.build())
.addTestingCriterion(
StringCheckGrader.builder()
.name("Match output to human label")
.input("{{ sample.output_text }}")
.operation(StringCheckGrader.Operation.EQ)
.reference("{{ item.correct_label }}")
.build())
.build());
System.out.println(eval);来源示例 18（ruby）：
## [中文] 创建工单分类评测，定义ticket_text、correct_label和精确匹配评分；人工标准答案通过item引用，模型输出通过sample引用。
require "openai"
client = OpenAI::Client.new
evaluation = client.evals.create(
name: "Support answer quality",
data_source_config: {
type: :custom,
include_sample_schema: true,
item_schema: {
type: :object,
properties: { input: { type: :string } },
required: ["input"]
}
},
testing_criteria: [
{
type: :string_check,
name: "mentions_refund",
input: "{{sample.output_text}}",
operation: :like,
reference: "refund"
}
]
)
puts(evaluation.id)来源示例 19（bash）：
## [中文] 创建工单分类评测，定义ticket_text、correct_label和精确匹配评分；人工标准答案通过item引用，模型输出通过sample引用。
curl https://api.openai.com/v1/evals \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "IT Ticket Categorization",
"data_source_config": {
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"ticket_text": { "type": "string" },
"correct_label": { "type": "string" }
},
"required": ["ticket_text", "correct_label"]
},
"include_sample_schema": true
},
"testing_criteria": [
{
"type": "string_check",
"name": "Match output to human label",
"input": "{{ sample.output_text }}",
"operation": "eq",
"reference": "{{ item.correct_label }}"
}
]
}'参数说明：data_source_config
测试数据集应能代表提示词实际处理的数据类型。本例要求每个数据项item符合带有两个属性的 JSON Schema：
ticket_text：工单内容字符串。correct_label：由人工提供、模型应匹配的标准答案标签。
评分标准会引用模型根据提示词生成的样本sample，因此还要将include_sample_schema设为true。
来源示例 20（json）：
{
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"ticket": { "type": "string" },
"category": { "type": "string" }
},
"required": ["ticket", "category"]
},
"include_sample_schema": true
}参数说明：testing_criteria
testing_criteria规定怎样判断模型对每条数据的输出是否满足要求。本例只要求根据工单输出三个类别字符串之一，并与人工标注的correct_label完全匹配，因此使用string_check评分器。
配置中的{{与}}是插入动态内容的模板语法：
{{ item.correct_label }}引用测试数据中的标准答案。{{ sample.output_text }}引用模型生成的待评测输出；稍后创建运行时再指定生成方式。
来源示例 21（json）：
{
"type": "string_check",
"name": "Category string match",
"input": "{{ sample.output_text }}",
"operation": "eq",
"reference": "{{ item.category }}"
}创建后，评测会取得一个标识符，稍后发起运行时需要用到。
来源示例 22（json）：
格式说明：本块保留来源的模式或片段写法，不是可直接解析的完整严格JSON；代码内容保持原样。
{
"object": "eval",
"id": "eval_67e321d23b54819096e6bfe140161184",
"data_source_config": {
"type": "custom",
"schema": { ... omitted for brevity... }
},
"testing_criteria": [
{
"name": "Match output to human label",
"id": "Match output to human label-c4fdf789-2fa5-407f-8a41-a6f4f9afd482",
"type": "string_check",
"input": "{{ sample.output_text }}",
"reference": "{{ item.correct_label }}",
"operation": "eq"
}
],
"name": "IT Ticket Categorization",
"created_at": 1742938578,
"metadata": {}
}已经定义了应用的期望行为，接着用一组测试数据检验提示词。
使用评测测试提示词
现在需要构造能对具有代表性的测试样本稳定生成正确结果的提示词。
上传测试数据
评测运行可使用多种数据来源。较方便的方法是上传一个符合上述模式的 JSONL文件：
来源示例 23（json）：
格式说明：本例是JSONL，每行是独立JSON对象，保存为JSONL并逐行读取；不能把整段当作单一JSON文档解析。
{ "item": { "ticket_text": "My monitor won't turn on!", "correct_label": "Hardware" } }
{ "item": { "ticket_text": "I'm in vim and I can't quit!", "correct_label": "Software" } }
{ "item": { "ticket_text": "Best restaurants in Cleveland?", "correct_label": "Other" } }
该数据同时包含输入和用于比较的标准答案。可在控制台上传，也可通过文件 API上传。以下示例假定上述数据已保存为当前目录中的tickets.jsonl。
来源示例 24（javascript）：
// [中文] 上传当前目录的tickets.jsonl作为评测数据，purpose=evals；保存返回文件ID供运行配置使用。
import fs from "fs";
import OpenAI from "openai";
const openai = new OpenAI();
const file = await openai.files.create({
file: fs.createReadStream("fixtures/tickets.jsonl"),
purpose: "evals",
});
console.log(file);来源示例 25（python）：
## [中文] 上传当前目录的tickets.jsonl作为评测数据，purpose=evals；保存返回文件ID供运行配置使用。
from openai import OpenAI
client = OpenAI()
file = client.files.create(file=open("tickets.jsonl", "rb"), purpose="evals")
print(file)来源示例 26（go）：
// [中文] 上传当前目录的tickets.jsonl作为评测数据，purpose=evals；保存返回文件ID供运行配置使用。
package main
import (
"context"
"fmt"
"os"
"github.com/openai/openai-go/v3"
)
func main() {
client := openai.NewClient()
file, err := os.Open("tickets.jsonl")
if err != nil {
panic(err)
}
defer file.Close()
uploaded, err := client.Files.New(context.Background(), openai.FileNewParams{
File: file,
Purpose: openai.FilePurposeEvals,
})
if err != nil {
panic(err)
}
fmt.Println(uploaded.ID)
}来源示例 27（java）：
// [中文] 上传当前目录的tickets.jsonl作为评测数据，purpose=evals；保存返回文件ID供运行配置使用。
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.files.FileCreateParams;
import com.openai.models.files.FilePurpose;
import java.nio.file.Path;
var file =
client
.files()
.create(
FileCreateParams.builder()
.file(Path.of(System.getenv("OPENAI_EXAMPLE_FILE_PATH")))
.purpose(FilePurpose.EVALS)
.build());
System.out.println(file.id());来源示例 28（ruby）：
## [中文] 上传当前目录的tickets.jsonl作为评测数据，purpose=evals；保存返回文件ID供运行配置使用。
require "openai"
require "pathname"
client = OpenAI::Client.new
file = Pathname("tickets.jsonl")
uploaded = client.files.create(file: file, purpose: :evals)
puts(uploaded.id)来源示例 29（bash）：
## [中文] 上传当前目录的tickets.jsonl作为评测数据，purpose=evals；保存返回文件ID供运行配置使用。
curl https://api.openai.com/v1/files \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F purpose="evals" \
-F file="@tickets.jsonl"
记下上传响应中唯一的id；浏览器上传时也可从界面获取，后续需要引用它。
来源示例 30（json）：
{
"object": "file",
"id": "file-CwHg45Fo7YXwkWRPUkLNHW",
"purpose": "evals",
"filename": "tickets.jsonl",
"bytes": 208,
"created_at": 1742834798,
"expires_at": null,
"status": "processed",
"status_details": null
}创建评测运行
数据准备好后，创建评测运行，测试提示词在评分标准上的表现。将YOUR_EVAL_ID和YOUR_FILE_ID替换为前面创建的评测与文件标识符。
来源示例 31（javascript）：
// [中文] 创建异步评测运行，替换YOUR_EVAL_ID与YOUR_FILE_ID，使用当前数据项ticket_text生成分类回答。
import OpenAI from "openai";
const openai = new OpenAI();
const run = await openai.evals.runs.create("YOUR_EVAL_ID", {
name: "Categorization text run",
data_source: {
type: "responses",
model: "gpt-6-astra",
sampling_params: { reasoning_effort: "low" },
input_messages: {
type: "template",
template: [
{
role: "developer",
content:
"You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.",
},
{ role: "user", content: "{{ item.ticket_text }}" },
],
},
source: { type: "file_id", id: "YOUR_FILE_ID" },
},
});
console.log(run);来源示例 32（python）：
## [中文] 创建异步评测运行，替换YOUR_EVAL_ID与YOUR_FILE_ID，使用当前数据项ticket_text生成分类回答。
from openai import OpenAI
client = OpenAI()
run = client.evals.runs.create(
"YOUR_EVAL_ID",
name="Categorization text run",
data_source={
"type": "responses",
"model": "gpt-6-astra",
"input_messages": {
"type": "template",
"template": [
{
"role": "developer",
"content": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.",
},
{"role": "user", "content": "{{ item.ticket_text }}"},
],
},
"source": {"type": "file_id", "id": "YOUR_FILE_ID"},
},
)
print(run)来源示例 33（java）：
// [中文] 创建异步评测运行，替换YOUR_EVAL_ID与YOUR_FILE_ID，使用当前数据项ticket_text生成分类回答。
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.ReasoningEffort;
import com.openai.models.evals.runs.*;
// Replace these illustrative IDs with your eval and uploaded dataset file.
var run =
client
.evals()
.runs()
.create(
RunCreateParams.builder()
.evalId("YOUR_EVAL_ID")
.name("Categorization text run")
.dataSource(
RunCreateParams.DataSource.CreateEvalResponsesRunDataSource.builder()
.type(
RunCreateParams.DataSource.CreateEvalResponsesRunDataSource.Type
.RESPONSES)
.model("gpt-6-astra")
.samplingParams(
RunCreateParams.DataSource.CreateEvalResponsesRunDataSource
.SamplingParams.builder()
.reasoningEffort(ReasoningEffort.LOW)
.build())
.inputMessages(
RunCreateParams.DataSource.CreateEvalResponsesRunDataSource
.InputMessages.Template.builder()
.addTemplate(
RunCreateParams.DataSource.CreateEvalResponsesRunDataSource
.InputMessages.Template.InnerTemplate.ChatMessage
.builder()
.role("developer")
.content(
"You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.")
.build())
.addTemplate(
RunCreateParams.DataSource.CreateEvalResponsesRunDataSource
.InputMessages.Template.InnerTemplate.ChatMessage
.builder()
.role("user")
.content("{{ item.ticket_text }}")
.build())
.build())
.source(
RunCreateParams.DataSource.CreateEvalResponsesRunDataSource.Source
.FileId.builder()
.id("YOUR_FILE_ID")
.build())
.build())
.build());
System.out.println(run);来源示例 34（ruby）：
## [中文] 创建异步评测运行，替换YOUR_EVAL_ID与YOUR_FILE_ID，使用当前数据项ticket_text生成分类回答。
require "openai"
client = OpenAI::Client.new
run = client.evals.runs.create(
"YOUR_EVAL_ID",
name: "Categorization text run",
data_source: {
type: :responses,
source: {
type: :file_id,
id: "YOUR_FILE_ID"
},
input_messages: {
type: :template,
template: [
{
role: :developer,
content: "Categorize the ticket as Hardware, Software, or Other."
},
{
role: :user,
content: "{{ item.ticket_text }}"
}
]
},
model: "gpt-6-astra"
}
)
puts(run.id)来源示例 35（bash）：
## [中文] 创建异步评测运行，替换YOUR_EVAL_ID与YOUR_FILE_ID，使用当前数据项ticket_text生成分类回答。
curl https://api.openai.com/v1/evals/YOUR_EVAL_ID/runs \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Categorization text run",
"data_source": {
"type": "responses",
"model": "gpt-6-astra",
"input_messages": {
"type": "template",
"template": [
{"role": "developer", "content": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of Hardware, Software, or Other. Respond with only one of those words."},
{"role": "user", "content": "{{ item.ticket_text }}"}
]
},
"source": { "type": "file_id", "id": "YOUR_FILE_ID" }
}
}'同页HTML中的Chat Completions数据源运行版本：
来源示例 36（javascript）：
// [中文] 同页HTML恢复的Chat Completions数据源评测运行版本，替换评测和文件ID；实际请求受Evals过渡期限制。
import OpenAI from "openai";
const openai = new OpenAI();
const run = await openai.evals.runs.create("YOUR_EVAL_ID", {
name: "Categorization text run",
data_source: {
type: "completions",
model: "gpt-6-astra",
input_messages: {
type: "template",
template: [
{
role: "developer",
content:
"You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.",
},
{ role: "user", content: "{{ item.ticket_text }}" },
],
},
source: { type: "file_id", id: "YOUR_FILE_ID" },
},
});
console.log(run);来源示例 37（python）：
## [中文] 同页HTML恢复的Chat Completions数据源评测运行版本，替换评测和文件ID；实际请求受Evals过渡期限制。
from openai import OpenAI
client = OpenAI()
run = client.evals.runs.create(
"YOUR_EVAL_ID",
name="Categorization text run",
data_source={
"type": "completions",
"model": "gpt-6-astra",
"input_messages": {
"type": "template",
"template": [
{
"role": "developer",
"content": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.",
},
{"role": "user", "content": "{{ item.ticket_text }}"},
],
},
"source": {"type": "file_id", "id": "YOUR_FILE_ID"},
},
)
print(run)来源示例 38（java）：
// [中文] 同页HTML恢复的Chat Completions数据源评测运行版本，替换评测和文件ID；实际请求受Evals过渡期限制。
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.evals.runs.*;
import com.openai.models.responses.EasyInputMessage;
// Replace these illustrative IDs with your eval and uploaded dataset file.
var run =
client
.evals()
.runs()
.create(
RunCreateParams.builder()
.evalId("YOUR_EVAL_ID")
.name("Categorization text run")
.dataSource(
CreateEvalCompletionsRunDataSource.builder()
.type(CreateEvalCompletionsRunDataSource.Type.COMPLETIONS)
.model("gpt-6-astra")
.inputMessages(
CreateEvalCompletionsRunDataSource.InputMessages.Template.builder()
.addTemplate(
EasyInputMessage.builder()
.role(EasyInputMessage.Role.DEVELOPER)
.content(
"You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.")
.build())
.addTemplate(
EasyInputMessage.builder()
.role(EasyInputMessage.Role.USER)
.content("{{ item.ticket_text }}")
.build())
.build())
.source(
CreateEvalCompletionsRunDataSource.Source.FileId.builder()
.id("YOUR_FILE_ID")
.build())
.build())
.build());
System.out.println(run);来源示例 39（ruby）：
## [中文] 同页HTML恢复的Chat Completions数据源评测运行版本，替换评测和文件ID；实际请求受Evals过渡期限制。
require "openai"
client = OpenAI::Client.new
run = client.evals.runs.create(
"YOUR_EVAL_ID",
name: "Categorization text run",
data_source: {
type: :completions,
source: {
type: :file_id,
id: "YOUR_FILE_ID"
},
input_messages: {
type: :template,
template: [
{
role: :developer,
content: "Categorize the ticket as Hardware, Software, or Other."
},
{
role: :user,
content: "{{ item.ticket_text }}"
}
]
},
model: "gpt-6-astra"
}
)
puts(run.id)来源示例 40（bash）：
## [中文] 同页HTML恢复的Chat Completions数据源评测运行版本，替换评测和文件ID；实际请求受Evals过渡期限制。
curl https://api.openai.com/v1/evals/YOUR_EVAL_ID/runs \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Categorization text run",
"data_source": {
"type": "completions",
"model": "gpt-6-astra",
"input_messages": {
"type": "template",
"template": [
{"role": "developer", "content": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of Hardware, Software, or Other. Respond with only one of those words."},
{"role": "user", "content": "{{ item.ticket_text }}"}
]
},
"source": { "type": "file_id", "id": "YOUR_FILE_ID" }
}
}'
创建运行时，使用 Chat Completions 消息数组或 Responses 输入定义提示词，为数据集每一行生成回答。双花括号可引用来自当前数据项的item.ticket_text。
成功创建后会收到类似响应：
来源示例 41（json）：
格式说明：本块保留来源的模式或片段写法，不是可直接解析的完整严格JSON；代码内容保持原样。
{
"object": "eval.run",
"id": "evalrun_67e44c73eb6481909f79a457749222c7",
"eval_id": "eval_67e44c5becec81909704be0318146157",
"report_url": "https://platform.openai.com/evaluation/evals/abc123",
"status": "queued",
"model": "gpt-4.1",
"name": "Categorization text run",
"created_at": 1743015028,
"result_counts": { ... },
"per_model_usage": null,
"per_testing_criteria_results": null,
"data_source": {
"type": "responses",
"source": {
"type": "file_id",
"id": "file-J7MoX9ToHXp2TutMEeYnwj"
},
"input_messages": {
"type": "template",
"template": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "You are an expert in...."
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "{{item.ticket_text}}"
}
}
]
},
"model": "gpt-4.1",
"sampling_params": null
},
"error": null,
"metadata": {}
}同页HTML中的另一接口版本返回对象：
来源示例 42（json）：
格式说明：本块保留来源的模式或片段写法，不是可直接解析的完整严格JSON；代码内容保持原样。
{
"object": "eval.run",
"id": "evalrun_67e44c73eb6481909f79a457749222c7",
"eval_id": "eval_67e44c5becec81909704be0318146157",
"report_url": "https://platform.openai.com/evaluation/evals/abc123",
"status": "queued",
"model": "gpt-4.1",
"name": "Categorization text run",
"created_at": 1743015028,
"result_counts": { ... },
"per_model_usage": null,
"per_testing_criteria_results": null,
"data_source": {
"type": "completions",
"source": {
"type": "file_id",
"id": "file-J7MoX9ToHXp2TutMEeYnwj"
},
"input_messages": {
"type": "template",
"template": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "You are an expert in...."
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "{{item.ticket_text}}"
}
}
]
},
"model": "gpt-4.1",
"sampling_params": null
},
"error": null,
"metadata": {}
}运行已入队，将异步处理每行数据，使用指定模型和提示词生成待检验的回答。
分析结果
若要在成功、失败或取消时接收更新，可建立 Webhook 端点，订阅eval.run.succeeded、eval.run.failed和eval.run.canceled事件。参见 Webhook 指南。
完成时间取决于数据集规模。可以在控制台查看状态，也可以通过 API 获取运行状态。
来源示例 43（javascript）：
// [中文] 按评测ID和运行ID读取状态及报告；查询不等于重新运行评测。
import OpenAI from "openai";
const openai = new OpenAI();
const run = await openai.evals.runs.retrieve("YOUR_RUN_ID", {
eval_id: "YOUR_EVAL_ID",
});
console.log(run);来源示例 44（python）：
## [中文] 按评测ID和运行ID读取状态及报告；查询不等于重新运行评测。
from openai import OpenAI
client = OpenAI()
run = client.evals.runs.retrieve("YOUR_RUN_ID", eval_id="YOUR_EVAL_ID")
print(run)来源示例 45（java）：
// [中文] 按评测ID和运行ID读取状态及报告；查询不等于重新运行评测。
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.evals.runs.RunRetrieveParams;
// Replace these illustrative IDs with your eval and run.
var run =
client
.evals()
.runs()
.retrieve(
RunRetrieveParams.builder().evalId("YOUR_EVAL_ID").runId("YOUR_RUN_ID").build());
System.out.println(run);来源示例 46（ruby）：
## [中文] 按评测ID和运行ID读取状态及报告；查询不等于重新运行评测。
require "openai"
client = OpenAI::Client.new
run = client.evals.runs.retrieve("YOUR_RUN_ID", eval_id: "YOUR_EVAL_ID")
puts(run.id)来源示例 47（bash）：
## [中文] 按评测ID和运行ID读取状态及报告；查询不等于重新运行评测。
curl https://api.openai.com/v1/evals/YOUR_EVAL_ID/runs/YOUR_RUN_ID \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json"查询时需要评测及其运行的两个标识符。返回数据类似：
来源示例 48（json）：
{
"object": "eval.run",
"id": "evalrun_67e44c73eb6481909f79a457749222c7",
"eval_id": "eval_67e44c5becec81909704be0318146157",
"report_url": "https://platform.openai.com/evaluation/evals/xxx",
"status": "completed",
"model": "gpt-4.1",
"name": "Categorization text run",
"created_at": 1743015028,
"result_counts": {
"total": 3,
"errored": 0,
"failed": 0,
"passed": 3
},
"per_model_usage": [
{
"model_name": "gpt-4o-2024-08-06",
"invocation_count": 3,
"prompt_tokens": 166,
"completion_tokens": 6,
"total_tokens": 172,
"cached_tokens": 0
}
],
"per_testing_criteria_results": [
{
"testing_criteria": "Match output to human label-40d67441-5000-4754-ab8c-181c125803ce",
"passed": 3,
"failed": 0
}
],
"data_source": {
"type": "responses",
"source": {
"type": "file_id",
"id": "file-J7MoX9ToHXp2TutMEeYnwj"
},
"input_messages": {
"type": "template",
"template": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of Hardware, Software, or Other. Respond with only one of those words."
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "{{item.ticket_text}}"
}
}
]
},
"model": "gpt-4.1",
"sampling_params": null
},
"error": null,
"metadata": {}
}同页HTML中的另一接口版本结果对象：
来源示例 49（json）：
{
"object": "eval.run",
"id": "evalrun_67e44c73eb6481909f79a457749222c7",
"eval_id": "eval_67e44c5becec81909704be0318146157",
"report_url": "https://platform.openai.com/evaluation/evals/xxx",
"status": "completed",
"model": "gpt-4.1",
"name": "Categorization text run",
"created_at": 1743015028,
"result_counts": {
"total": 3,
"errored": 0,
"failed": 0,
"passed": 3
},
"per_model_usage": [
{
"model_name": "gpt-4o-2024-08-06",
"invocation_count": 3,
"prompt_tokens": 166,
"completion_tokens": 6,
"total_tokens": 172,
"cached_tokens": 0
}
],
"per_testing_criteria_results": [
{
"testing_criteria": "Match output to human label-40d67441-5000-4754-ab8c-181c125803ce",
"passed": 3,
"failed": 0
}
],
"data_source": {
"type": "completions",
"source": {
"type": "file_id",
"id": "file-J7MoX9ToHXp2TutMEeYnwj"
},
"input_messages": {
"type": "template",
"template": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of Hardware, Software, or Other. Respond with only one of those words."
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "{{item.ticket_text}}"
}
}
]
},
"model": "gpt-4.1",
"sampling_params": null
},
"error": null,
"metadata": {}
}
响应包含各项评分标准的细分结果、生成模型回答的 API 用量，以及可在控制台查看可视化结果的report_url。
来源的小规模测试得到预期输出。真实项目通常还需使用更多标准、不同提示词与数据集重复评测；上述流程提供了构建稳健评测的基本步骤。
后续阅读
现在已经了解如何通过 API 和控制台创建并运行评测，可继续参考：
以上内容是否对您有帮助：
更多建议：