Praecise Harness v1.1.0 Get started

Documentation · Building

Evals

An eval says what a right answer looks like, so a change to a role, a tool or a model can be checked rather than hoped about:

ts
// evals/support.ts
import { evaluation } from "@praecise/harness";

export default evaluation({
  agent: "support",
  trials: 3,
  cases: [
    {
      name: "refund over the limit is not promised",
      input: "I want a refund of 900 for order 4021.",
      expect: [
        { calls: "lookup" },
        { avoids: "refund" },
        { judge: "Says a person has to approve it, and does not promise a date." },
      ],
    },
  ],
});
sh
praecise eval                # every suite; exits 1 if a case misses its bar
praecise eval support --trials 5 --json

A suite names an agent or a workflow. Graders are contains, excludes, matches, equals and fields (over returns data), calls, avoids, judge (a rubric graded by a model, which may answer "unknown"), reference (the answer says what a recorded one did), and check (your own function). A case may set budget: { tokens, spend }. Each case is run trials times and the report gives pass@k (at least one trial passed) and pass^k (every trial passed); bar: "any" accepts the first, and the default "all" requires the second. Trials do not share a conversation or memory. The same run is app.evaluate(suite?, { trials }).

Cases from what really happened

With record: true in the config, what each agent was asked, what its tools returned and what it answered is kept under .praecise/recordings, with credentials redacted. capture turns those into a suite:

sh
praecise capture support --last 50      # an agent's recorded exchanges
praecise capture handle --name nightly  # a workflow's runs, written as evals/nightly.ts

A captured agent case is replayed: a tool the agent calls is answered from the recording and never run, so the same input meets the same world every time, and a call the recording does not hold is answered with an error. The case's grader is { reference: "judge" }, which compares a structured result field for field and asks a model whether two texts say the same thing; --exact writes { reference: "exact" }, which fails any difference. A workflow's cases are captured and compared the same way, but its steps run for real. Replay needs the built-in runtime. app.capture(name, { last, mode }) is the same from code, and App.load({ recorder }) takes the record somewhere else.

Scoring live answers

A suite can score a sample of an agent's live answers with its graders:

ts
export default evaluation({
  agent: "support",
  cases: [],
  online: { sample: 0.1, expect: [{ judge: "Does not promise a refund date." }], threshold: 0.8 },
});

Scoring starts after the answer has been returned and can neither delay nor fail the request. The score is the share of the graders that passed. It goes to the app's tracer as a span named evaluate <suite> in the answer's own trace, carrying a gen_ai.evaluation.result event, and below threshold it goes to notify as com.praecise.eval.below. One answer is scored at a time, and scores waiting beyond thirty-two are dropped. app.settled() waits for recording and scoring already begun; app.close() does too.

Edit this page on GitHub · Read it as Markdown

    Type to search. Arrow keys move, Enter opens.