# Evals

An eval says what a right answer looks like, so a change to a role, a tool or a
model can be checked rather than hoped about:

```ts
// evals/support.ts
import { evaluation } from "@praecise/harness";

export default evaluation({
  agent: "support",
  trials: 3,
  cases: [
    {
      name: "refund over the limit is not promised",
      input: "I want a refund of 900 for order 4021.",
      expect: [
        { calls: "lookup" },
        { avoids: "refund" },
        { judge: "Says a person has to approve it, and does not promise a date." },
      ],
    },
  ],
});
```

```sh
praecise eval                # every suite; exits 1 if a case misses its bar
praecise eval support --trials 5 --json
```

A suite names an `agent` or a `workflow`. Graders are `contains`, `excludes`,
`matches`, `equals` and `fields` (over `returns` data), `calls`, `avoids`,
`judge` (a rubric graded by a model, which may answer "unknown"), `reference`
(the answer says what a recorded one did), and `check` (your own function). A
case may set `budget: { tokens, spend }`. Each case is run `trials` times
and the report gives `pass@k` (at least one trial passed) and `pass^k` (every trial
passed); `bar: "any"` accepts the first, and the default `"all"` requires the
second. Trials do not share a conversation or memory. The same run is
`app.evaluate(suite?, { trials })`.

## Cases from what really happened

With `record: true` in the config, what each agent was asked, what its tools
returned and what it answered is kept under `.praecise/recordings`, with
credentials redacted. `capture` turns those into a suite:

```sh
praecise capture support --last 50      # an agent's recorded exchanges
praecise capture handle --name nightly  # a workflow's runs, written as evals/nightly.ts
```

A captured agent case is replayed: a tool the agent calls is answered from the
recording and never run, so the same input meets the same world every time, and
a call the recording does not hold is answered with an error. The case's grader
is `{ reference: "judge" }`, which compares a structured result field for field
and asks a model whether two texts say the same thing; `--exact` writes
`{ reference: "exact" }`, which fails any difference. A workflow's cases are
captured and compared the same way, but its steps run for real. Replay needs the
built-in runtime. `app.capture(name, { last, mode })` is the same from code, and
`App.load({ recorder })` takes the record somewhere else.

## Scoring live answers

A suite can score a sample of an agent's live answers with its graders:

```ts
export default evaluation({
  agent: "support",
  cases: [],
  online: { sample: 0.1, expect: [{ judge: "Does not promise a refund date." }], threshold: 0.8 },
});
```

Scoring starts after the answer has been returned and can neither delay nor fail
the request. The score is the share of the graders that passed. It goes to the
app's `tracer` as a span named `evaluate <suite>` in the answer's own trace,
carrying a `gen_ai.evaluation.result` event, and below `threshold` it goes to
`notify` as `com.praecise.eval.below`. One answer is scored at a time, and
scores waiting beyond thirty-two are dropped. `app.settled()` waits for
recording and scoring already begun; `app.close()` does too.
