TypeScript API: Evals

Datasets, scorers, judges, comparisons and quality gates. Generated from the declarations and doc comments of criogenai 0.1.7. See also: TypeScript API overview, the same module in Python.

Evals: run an agent over a set of cases, score each answer, compare with the last good run, and fail CI when quality drops.

import { evals } from "criogenai";
const data = await evals.Dataset.load("cases.jsonl");
const result = await evals.evaluate(agent, data, [evals.contains(), evals.toolCalled("lookup_order")]);
console.log(result.report());
await result.save("eval.json");
result.check({ minPassRate: 0.9, baseline: await evals.EvalResult.load("baseline.json") });

The result file has the same format in Python, TypeScript and Rust (criogen-eval/1), so a baseline saved by one is read by the others.

import { evals } from "criogenai";
evals.evaluate

These are used as evals.<name>.

Contents#

evaluate#

function evaluate(target: Target, dataset: Dataset | Iterable<string | Case>, scorers: Scorer[], options?: EvaluateOptions): Promise<EvalResult>

Runs target (an Agent, or a function case -> output) on every case and scores the outputs. A case that fails or times out scores 0 with its error kept; the eval goes on.

For an Agent, the run's tool calls, model calls, tokens and time are kept for scorers and the result. A function may return a value or a Promise; if it returns an agent's RunResult, the same facts are kept.

Parameters

• target (Target): An Agent, or a function from a case to an output.
• dataset (Dataset | Iterable<string | Case>): A Dataset, or cases (objects or strings).
• scorers (Scorer[]): At least one scorer; names must be unique.
• options (EvaluateOptions, default {}): Name, concurrency, a time limit per case and repeats.

Returns: The result: summary, every case, and check, save, report.

Raises

• Error: No scorers, or two scorers with the same name.

Example

import { Agent, FakeModel, evals } from "criogenai";

const agent = new Agent({ model: new FakeModel(["Paris is the capital of France."]) });
const result = await evals.evaluate(
  agent,
  [{ id: "capital", input: "Capital of France?", expected: "Paris" }],
  [evals.contains(), evals.maxModelCalls(2)],
  { name: "geography" },
);
console.log(result.report()); // geography: 1/1 passed (100.0%), 0 error(s) ...
result.check({ minPassRate: 1 });

Dataset#

class Dataset {
  constructor(cases: Iterable<string | Case>);
}

Cases to run: from a list, or Dataset.load a JSONL / JSON file (Node).

Parameters

• cases (Iterable<string | Case>): Cases, or plain strings (an input with no expected value).

Raises

• Error: A case has no text input, ids repeat, or there are more than 100,000 cases.

Example

import { evals } from "criogenai";

const data = new evals.Dataset([
  { id: "capital", input: "Capital of France?", expected: "Paris" },
  "A case with only an input",
]);
console.log(data.cases.map((c) => c.id)); // ["capital", "2"]

Dataset.load#

static load(path: string): Promise<Dataset>

A .jsonl file (one case per line) or a .json file (a list, or {"cases": [...]}).

Node only. Blank lines in a .jsonl file are skipped; a line may hold at most 1,000,000 bytes.

Parameters

• path (string): The file. A name ending in .jsonl is read as JSON Lines, anything else as JSON.

Raises

• Error: A line or the file is not JSON, or a case is not valid (the message names the line).

Dataset.cases#

readonly cases: (Case & { id: string; })[]

The cases, each with its id and expected (null when not given).

EvalResult#

class EvalResult {
  constructor(data: EvalData);
}

An eval's outcome, in the criogen-eval/1 format every language reads.

Made by evaluate, or read back with EvalResult.load. A baseline saved by Python or Rust is read here too.

Parameters

• data (EvalData): Result data in the criogen-eval/1 format.

Raises

• Error: The data is not in that format.

Example

import { evals } from "criogenai";

const result = await evals.evaluate((c) => c.input.toUpperCase(), ["ok", "fine"], [evals.regex("^[A-Z]+$")]);
await result.save("eval.json", { content: false }); // ids and scores only
const baseline = await evals.EvalResult.load("eval.json");
console.log(evals.comparisonReport(evals.compare(baseline, result)));
result.check({ minPassRate: 0.9, baseline });

EvalResult.load#

static load(path: string): Promise<EvalResult>

Reads a result file written by save (in any language). Node only.

Parameters

• path (string): The file.

Raises

• Error: The file is not JSON or not in the criogen-eval/1 format.

EvalResult.data#

readonly data: EvalData

The result as plain data.

EvalResult.name#

readonly name: string

The eval's name.

EvalResult.summary#

readonly summary: Summary

The totals.

EvalResult.cases#

readonly cases: CaseResult[]

Every case, in dataset order.

EvalResult.passRate#

readonly passRate: number

The share of cases that passed, from 0 to 1.

EvalResult.toJSON#

toJSON(options?: { content?: boolean; }): EvalData

JSON-ready data; content: false leaves inputs, expected values and outputs out.

Parameters

• options ({ content?: boolean; }, default {}): content (default true).

EvalResult.save#

save(path: string, options?: { content?: boolean; }): Promise<void>

Writes the result as JSON (Node). Read it back with EvalResult.load, in any language.

Parameters

• path (string): The file to write (replaced if it exists).
• options ({ content?: boolean; }, default {}): content: false leaves inputs, expected values and outputs out (ids, scores and counts stay).

EvalResult.report#

report(): string

A plain-text summary and the failing cases.

EvalResult.markdown#

markdown(): string

A Markdown table, for a pull request.

EvalResult.check#

check(options?: CheckOptions): void

Throws EvalFailed when a bar is missed.

Use it to fail a CI job: every missed bar is listed in the error's failures.

Parameters

• options (CheckOptions, default {}): The bars: a pass rate, scorer means, and a baseline to compare with.

Raises

• {EvalFailed} One or more bars were missed (code eval_failed). A name in minScores that no scorer has counts as missed.

compare#

function compare(baseline: EvalResult, current: EvalResult): Comparison

Per-scorer and pass-rate changes, and the cases that regressed or were fixed (matched by id).

Parameters

• baseline (EvalResult): The earlier result, e.g. the last good run.
• current (EvalResult): The new result.

FORMAT#

const FORMAT: "criogen-eval/1"

The result file format, the same in Python, TypeScript and Rust.

MAX_CASES#

const MAX_CASES: 100000

The most cases a dataset holds.

MAX_LINE_BYTES#

const MAX_LINE_BYTES: 1000000

The longest line of a .jsonl dataset, in bytes.

MAX_OUTPUT_CHARS#

const MAX_OUTPUT_CHARS: 100000

Text outputs longer than this many characters are cut in the result.

MAX_CONCURRENCY#

const MAX_CONCURRENCY: 64

The most cases evaluate runs at once.

EvalFailed#

class EvalFailed extends CriogenError {
  constructor(message: string, failures: string[]);
}

An eval missed a bar set by check (code eval_failed).

Parameters

• message (string): The eval's name and every failure.
• failures (string[]): Each missed bar, in words.

EvalFailed.failures#

readonly failures: string[]

Each missed bar, in words (also in details.failures).

Case#

interface Case {
  input: string;
  expected?: unknown;
  id?: string;
  state?: Record<string, unknown>;
  metadata?: Record<string, unknown>;
}

One input, what to expect, and run state.

Fields

• input (string): The text given to the target (the agent's input). Required.
• expected (unknown, optional): What scorers compare with: a text, a list of texts, or any JSON value.
• id (string, optional): Unique in the dataset; comparisons match cases by id. Default: the case's position, from "1".
• state (Record<string, unknown>, optional): Run state for an agent target, e.g. { user_id: "u-1" }.
• metadata (Record<string, unknown>, optional): Kept in the result next to the case.

Score#

interface Score {
  value: number;
  pass: boolean;
  reason: string | null;
}

A scorer's verdict: value from 0 to 1, whether it passed, and why.

Fields

• value (number): From 0 (worst) to 1 (best).
• pass (boolean): Whether the case passes this scorer.
• reason (string | null): Why it failed, or null.

Facts#

interface Facts {
  toolCalls: string[];
  modelCalls: number;
  tokens: { input: number; output: number };
  durationMs: number;
  models: { provider: string; model: string; usage: Record<string, number> }[];
  judgeTokens?: { input: number; output: number };
}

What a run did, for scorers. Only an Agent target fills it; a function target has only durationMs.

Fields

• toolCalls (string[]): The names of the tools the agent called, in order.
• modelCalls (number): How many model calls the run made.
• tokens ({ input: number; output: number; }): Input and output tokens over all model calls.
• durationMs (number): How long the case took, in milliseconds.
• models ({ provider: string; model: string; usage: Record<string, number>; }[]): Each model the run used, with its token usage (input_tokens, output_tokens, ...).
• judgeTokens ({ input: number; output: number; }, optional): Tokens spent by judge scorers on this case, apart from the agent's.

ScoreFn#

type ScoreFn = (c: Case, output: unknown, facts: Facts) => Score | number | boolean | Promise<Score | number | boolean>;

A scoring function for scorer: gets the case, the output and the run's facts; returns a Score, a number from 0 to 1, or a boolean.

Scorer#

interface Scorer {
  readonly name: string;
  score(c: Case, output: unknown, facts: Facts): Promise<Score>;
}

A named scoring function. Make one with scorer, or use a built-in one.

Fields

• name (string): Unique among an eval's scorers; the key in results and minScores.
• score ((c: Case, output: unknown, facts: Facts) => Promise<Score>): Scores one output.

scorer#

function scorer(name: string, fn: ScoreFn, threshold?: number): Scorer

Your own scorer: a function returning a Score, a number (passes at threshold) or a bool.

A number is clamped to 0..1. A scorer that throws scores 0 for that case, with the error as the reason.

Parameters

• name (string): The scorer's name in results.
• fn (ScoreFn): The scoring function.
• threshold (number, default 1): For numbers: the value a case needs to pass.

Example

import { evals } from "criogenai";

const short = evals.scorer("short", (_case, output) => String(output).length <= 200);
const polite = evals.scorer("polite", (_case, output) => (/please|thank/i.test(String(output)) ? 1 : 0.5), 0.5);

exactMatch#

function exactMatch(options?: { ignoreCase?: boolean; }): Scorer

The output equals expected (both trimmed).

Parameters

• options ({ ignoreCase?: boolean; }, default {}): ignoreCase (default false).

contains#

function contains(needles?: string[], options?: { ignoreCase?: boolean; }): Scorer

The output contains every needle (default: expected, a string or a list). The value is the share found.

Parameters

• needles (string[], default []): The texts to look for; empty: the case's expected.
• options ({ ignoreCase?: boolean; }, default {}): ignoreCase (default true).

regex#

function regex(pattern?: string): Scorer

The output matches pattern (default: expected) somewhere.

Parameters

• pattern (string, optional): A JavaScript regular expression, as text (e.g. "A-\\d+").

oneOf#

function oneOf(options?: string[], settings?: { ignoreCase?: boolean; }): Scorer

The output (trimmed) is one of options (default: expected, a list).

Parameters

• options (string[], optional): The allowed answers; default the case's expected.
• settings ({ ignoreCase?: boolean; }, default {}): ignoreCase (default true).

jsonValid#

function jsonValid(): Scorer

The output is JSON (or already a value).

jsonSchema#

function jsonSchema(schema: Record<string, unknown>): Scorer

The output (JSON) matches schema, checked by the engine's validator.

Parameters

• schema (Record<string, unknown>): A JSON Schema.

jsonEquals#

function jsonEquals(path?: string): Scorer

The output (JSON), or its field at path ("a.b.0"), equals expected.

Parameters

• path (string, optional): Dot-separated keys and list indexes; without it the whole output is compared.

toolCalled#

function toolCalled(name: string, min?: number): Scorer

The agent called tool name at least min times.

Parameters

• name (string): The tool's name.
• min (number, default 1): The fewest calls that pass.

toolNotCalled#

function toolNotCalled(name: string): Scorer

The agent never called tool name.

Parameters

• name (string): The tool's name.

maxToolCalls#

function maxToolCalls(n: number): Scorer

At most n tool calls.

Parameters

• n (number): The most tool calls that pass.

maxModelCalls#

function maxModelCalls(n: number): Scorer

At most n model calls.

Parameters

• n (number): The most model calls that pass.

maxLatency#

function maxLatency(ms: number): Scorer

The case took at most ms milliseconds.

Parameters

• ms (number): The longest time that passes, in milliseconds.

maxTokens#

function maxTokens(n: number): Scorer

At most n tokens in and out together.

Parameters

• n (number): The most tokens that pass (the judge's tokens do not count).

maxCost#

function maxCost(usd: number, prices: Record<string, { input?: number; output?: number; }>): Scorer

At most usd dollars from your prices (USD per million tokens by "provider:model" or model name). Fails when a model has no price.

Parameters

• usd (number): The most a case may cost, in US dollars.
• prices (Record<string, { input?: number; output?: number; }>): USD per million input and output tokens, e.g. { "openai:gpt-5-mini": { input: 0.25, output: 2 } }.

JUDGE_INSTRUCTIONS#

const JUDGE_INSTRUCTIONS: string

The instructions judge gives its model.

judge#

function judge(model: Model, rubric: string, options?: { threshold?: number; gateway?: Gateway; name?: string; }): Scorer

A model grades the answer against rubric (0 to 1, with a reason) through the Gateway. Passes at threshold.

The question, the answer and the expected value (when there is one) reach the judge as data between markers the answer cannot close, and the judge is told to ignore instructions inside them. Its tokens are kept per case (judge_tokens), apart from the agent's.

Parameters

• model (Model): The judge's model, e.g. "openai:gpt-5-mini".
• rubric (string): What a good answer does, in words.
• options ({ threshold?: number; gateway?: Gateway; name?: string; }, default {}): threshold (default 0.7), gateway (default a new Gateway), name (default "judge"; give each judge its own name when you use several).

Example

import { evals } from "criogenai";

// Needs OPENAI_API_KEY when the eval runs.
const helpful = evals.judge("openai:gpt-5-mini", "Answers the question politely and gives the order's status.", {
  threshold: 0.8,
  name: "helpful",
});

EvaluateOptions#

interface EvaluateOptions {
  name?: string;
  concurrency?: number;
  timeoutSecs?: number;
  repeats?: number;
}

Settings for evaluate.

Fields

• name (string, default "eval"): The eval's name in the result and reports.
• concurrency (number, optional): Cases at once (default 4, at most 64).
• timeoutSecs (number, optional): Seconds per case (default 120).
• repeats (number, optional): Runs per case; the score is the mean, and pass needs every run to pass.

CaseResult#

interface CaseResult {
  id: string;
  input: string | null;
  expected: unknown;
  output: unknown;
  scores: Record<string, ScoreJson>;
  pass: boolean;
  error: string | null;
  duration_ms: number;
  tool_calls: string[];
  model_calls: number;
  tokens: { input: number; output: number };
  judge_tokens?: { input: number; output: number };
  metadata?: Record<string, unknown>;
}

One case in a result file. Field names are snake_case, the same in every language.

Fields

• id (string): The case's id.
• input (string | null): The case's input; null in a file saved with content: false.
• expected (unknown): The expected value; null when there is none or in a file saved with content: false.
• output (unknown): What the target returned (texts over 100,000 characters are cut); null on error or with content: false.
• scores (Record<string, ScoreJson>): Each scorer's mean value over the repeats, whether every repeat passed, and the first reason given.
• pass (boolean): No error and every scorer passed.
• error (string | null): Why the target failed or timed out, or null.
• duration_ms (number): The mean time per repeat, in milliseconds.
• tool_calls (string[]): The names of the tools the agent called.
• model_calls (number): How many model calls the run made.
• tokens ({ input: number; output: number; }): The agent's input and output tokens.
• judge_tokens ({ input: number; output: number; }, optional): Tokens spent by judge scorers, when there were any.
• metadata (Record<string, unknown>, optional): The case's metadata, when it had any.

Summary#

interface Summary {
  cases: number;
  passed: number;
  pass_rate: number;
  scores: Record<string, number>;
  tokens: { input: number; output: number };
  errors: number;
}

The totals of an eval.

Fields

• cases (number): How many cases ran.
• passed (number): How many passed.
• pass_rate (number): passed / cases, from 0 to 1.
• scores (Record<string, number>): Each scorer's mean value over all cases.
• tokens ({ input: number; output: number; }): The agent's tokens over all cases (judges not included).
• errors (number): How many cases failed with an error or a timeout.

EvalData#

interface EvalData {
  format: string;
  name: string;
  started_at: string;
  duration_ms: number;
  target: "agent" | "function";
  scorers: string[];
  summary: Summary;
  cases: CaseResult[];
}

A whole result file, in the criogen-eval/1 format.

Fields

• format (string): Always "criogen-eval/1".
• name (string): The eval's name.
• started_at (string): When it started, as an ISO 8601 date and time.
• duration_ms (number): How long the whole eval took, in milliseconds.
• target ("function" | "agent"): What ran: an Agent or a function.
• scorers (string[]): The scorers' names, in order.
• summary (Summary): The totals.
• cases (CaseResult[]): Every case, in dataset order.

CheckOptions#

interface CheckOptions {
  minPassRate?: number;
  minScores?: Record<string, number>;
  baseline?: EvalResult;
  maxRegressions?: number;
  tolerance?: number;
}

The bars EvalResult.check holds a result to. Without any, check passes.

Fields

• minPassRate (number, optional): The lowest pass rate allowed, from 0 to 1.
• minScores (Record<string, number>, optional): The lowest mean allowed per scorer, by scorer name, e.g. { contains: 0.9 }.
• baseline (EvalResult, optional): An earlier result to compare with: scorer means may not fall by more than tolerance, and cases that passed there may not fail (beyond maxRegressions).
• maxRegressions (number, optional): Cases that may regress against the baseline (default 0).
• tolerance (number, optional): How far a scorer's mean may fall against the baseline (default 0.02).

Comparison#

interface Comparison {
  scores: Record<string, number | null>;
  passRate: number;
  regressed: string[];
  fixed: string[];
  added: string[];
  removed: string[];
}

How current differs from baseline.

Fields

• scores (Record<string, number | null>): Each scorer's change in mean (current minus baseline); null when only one of the two has the scorer.
• passRate (number): The change in pass rate (current minus baseline).
• regressed (string[]): Ids of cases that passed in the baseline and fail now.
• fixed (string[]): Ids of cases that failed in the baseline and pass now.
• added (string[]): Ids only in the current result.
• removed (string[]): Ids only in the baseline.

comparisonReport#

function comparisonReport(diff: Comparison): string

A comparison as text.

Parameters

• diff (Comparison): What compare returned.