Skip to content

feat: Add reporter for vitest-evals#2142

Merged
Luca Forstner (lforst) merged 7 commits into
mainfrom
lforst/vitest-evals-reporter
Jun 23, 2026
Merged

feat: Add reporter for vitest-evals#2142
Luca Forstner (lforst) merged 7 commits into
mainfrom
lforst/vitest-evals-reporter

Conversation

@lforst

@lforst Luca Forstner (lforst) commented Jun 19, 2026

Copy link
Copy Markdown
Member

Adds a reporter for https://www.npmjs.com/package/vitest-evals

Setup:

// vitest.config.ts
import BraintrustVitestEvalsReporter from "braintrust/vitest-evals-reporter";
import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    reporters: [
      "vitest-evals/reporter",
      new BraintrustVitestEvalsReporter({
        projectName: "my-project",
      }),
    ],
  },
});

or

reporters: [
  "vitest-evals/reporter",
  ["braintrust/vitest-evals-reporter", { projectName: "my-project" }],
]

We are emitting these scorers:

{
  pass: 1 | 0,
  avg_score?: number | null,
  [judgeName]: number | null
}

@lforst Luca Forstner (lforst) marked this pull request as ready for review June 19, 2026 14:22
};

if (typeof evalMeta?.avgScore === "number" || evalMeta?.avgScore === null) {
scores.avg_score = evalMeta.avgScore;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we prefix with vitest? To reduce the chance of user collision if they decide to call their score avg_score or pass?

@lforst Luca Forstner (lforst) merged commit d4dd324 into main Jun 23, 2026
48 checks passed
@lforst Luca Forstner (lforst) deleted the lforst/vitest-evals-reporter branch June 23, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants