Evals
An eval says what a right answer looks like, so a change to a role, a tool or a model can be checked rather than hoped about:
// evals/support.ts
import { evaluation } from "@praecise/harness";
export default evaluation({
agent: "support",
trials: 3,
cases: [
{
name: "refund over the limit is not promised",
input: "I want a refund of 900 for order 4021.",
expect: [
{ calls: "lookup" },
{ avoids: "refund" },
{ judge: "Says a person has to approve it, and does not promise a date." },
],
},
],
});praecise eval # every suite; exits 1 if a case misses its bar
praecise eval support --trials 5 --jsonA suite names an agent or a workflow. Graders are contains, excludes,
matches, equals and fields (over returns data), calls, avoids,
judge (a rubric graded by a model, which may answer "unknown"), reference
(the answer says what a recorded one did), and check (your own function). A
case may set budget: { tokens, spend }. Each case is run trials times
and the report gives pass@k (at least one trial passed) and pass^k (every trial
passed); bar: "any" accepts the first, and the default "all" requires the
second. Trials do not share a conversation or memory. The same run is
app.evaluate(suite?, { trials }).
Cases from what really happened
With record: true in the config, what each agent was asked, what its tools
returned and what it answered is kept under .praecise/recordings, with
credentials redacted. capture turns those into a suite:
praecise capture support --last 50 # an agent's recorded exchanges
praecise capture handle --name nightly # a workflow's runs, written as evals/nightly.tsA captured agent case is replayed: a tool the agent calls is answered from the
recording and never run, so the same input meets the same world every time, and
a call the recording does not hold is answered with an error. The case's grader
is { reference: "judge" }, which compares a structured result field for field
and asks a model whether two texts say the same thing; --exact writes
{ reference: "exact" }, which fails any difference. A workflow's cases are
captured and compared the same way, but its steps run for real. Replay needs the
built-in runtime. app.capture(name, { last, mode }) is the same from code, and
App.load({ recorder }) takes the record somewhere else.
Scoring live answers
A suite can score a sample of an agent's live answers with its graders:
export default evaluation({
agent: "support",
cases: [],
online: { sample: 0.1, expect: [{ judge: "Does not promise a refund date." }], threshold: 0.8 },
});Scoring starts after the answer has been returned and can neither delay nor fail
the request. The score is the share of the graders that passed. It goes to the
app's tracer as a span named evaluate <suite> in the answer's own trace,
carrying a gen_ai.evaluation.result event, and below threshold it goes to
notify as com.praecise.eval.below. One answer is scored at a time, and
scores waiting beyond thirty-two are dropped. app.settled() waits for
recording and scoring already begun; app.close() does too.