Task seeding

Each agent run reads a per-task seed that controls the prompt, the starting world state, any failures injected during the run, and an optional oracle the judge uses when the "correct" answer is a refusal.

For model-as-user conversations (turn_mode = model_as_user), see Multi-turn testing setup.

Consistency mode is always strict in v1 (responses are re-checked against the ledger with one bounded regen).

Populating the seed

In the Pipeline Builder, open the agent-mode field's Odyssey Seed Columns popover and toggle on the axes your dataset provides. Your CSV needs a column for each enabled axis (header names from the table above; user is always required):

user,behavior,state,failure_rules
"Refund order #4521 if it shipped more than 30 days ago.","The refund_order tool rejects any order whose shipped_at is more than 90 days before the run date and returns it unchanged.","{""order"":{""4521"":{""status"":""shipped"",""shipped_at"":""2026-04-01"",""amount"":79.50}}}","[{""trigger"":""after_n_calls"",""tool"":""refund_order"",""n"":1,""duration"":1,""error"":{""code"":502,""message"":""Payment processor unavailable""}}]"

At task creation the seeding service materializes a tasks.odyssey_seed JSONB blob keyed by the axis names from the table above (CSV user → user_instruction, state → initial_state, and so on). The initial_state and failure_rules shapes are detailed below.

Failure rules

Shape: { trigger, tool, ...trigger-specific, error }. tool matches literally; "*" matches any tool.

The error envelope:

Trace rows from rules are tagged source = "injected" with a matched_rule_index pointing back to the rule.

after_n_calls

Fires on the n-th call to tool and stays active for duration subsequent calls.

{
  "trigger": "after_n_calls",
  "tool": "refund_order",
  "n": 3,
  "duration": 1,
  "error": { "code": 502, "message": "Payment processor unavailable" }
}

random

Per-call probability of firing. The engine uses a fixed deterministic RNG seed in v1, so the same task replays the same fire pattern across re-runs.

{
  "trigger": "random",
  "tool": "*",
  "probability": 0.1,
  "error": { "code": 503, "message": "Upstream temporarily unavailable" }
}

after_state_change

Fires once a named ledger flag is set; stays active for duration calls.

{
  "trigger": "after_state_change",
  "tool": "get_inventory",
  "condition": "warehouse_outage",
  "duration": 5,
  "error": {
    "code": 200,
    "response": { "items": [], "stale": true }
  }
}

Use this for failures causally triggered by something the agent did earlier in the run.

Rules are evaluated in array order; the first match wins. Put specific rules before catch-alls.

initial_state shape

{
  "initial_state": {
    "user": {
      "u-1": { "email": "alice@example.com", "tier": "gold" }
    },
    "order": {
      "o-101": { "user_id": "u-1", "total": 49.99, "status": "paid" }
    }
  }
}

By default the entity_type keys (user, order above) are arbitrary labels — the simulator uses them as world-state context, not as a registry. They don't need to match anything in your tools_schema or your tool responses; pick whatever singular noun your tool responses naturally describe (company / companies / Company are all fine — pick one convention and stay consistent within a dataset).

If the agent declares a ledger schema, these keys are no longer arbitrary: they're validated against the declared entity types, so use the singular type names from the schema (order, not orders).

As tools execute, the simulator emits ledger updates (add / update / remove / set_flag) that the trace viewer renders as a step-by-step diff.

Synthetic generation: one shared world per session

When you author seeds by hand (CSV / API), each row carries its own initial_state. When you instead synthetically generate seeds for an agent that enables the initial_state axis (the Synthetically generate seeds flow under Create Tasks), the model builds a single shared world for the whole session — not a fresh micro-world per row.

This mirrors reality: one backend holds many records, and many scenarios run against it. The world varies by data domain ("Company A's data"); the scenario themes (your buckets) are different asks against that one world. Concretely:

• initial_state is generated once per session — a rich, varied world (many entities per type, full enum/owner/age spread) grounded in the agent's tools and ledger schema.
• user_instruction (+ failure_rules + expected_outcome) are generated per row, grounded in that shared world.

The flow is a gated review before any rows are generated:

1. World step. The generator produces the shared world and pauses for you to review it (grouped, read-only). Pick a world size — Compact (~6 entities/type), Standard (~12, the default), or Rich (~20) — on the initial action. Coverage advisories (e.g. an enum that only uses one value) are surfaced here.
2. Refine. Edit the world with a reprompt ("add 3 cancelled orders owned by a non-buyer"), regenerate from scratch, or paste your own initial_state (validated against the agent's tools/ledger). There is no cell-level editing — refinement is prompt-driven.
3. Continue. Advancing freezes the world for the session and starts per-bucket row generation. The frozen world is shown read-only alongside the row preview; rows no longer carry a per-row state column.
4. Finalize. The frozen world is stamped onto every materialized task — each task's odyssey_seed.initial_state is a copy of the one session world.

expected_outcome

Optional task-author oracle telling the judge what "correct" looks like on this row. Accepts two values (case-insensitive on input, stored lower-case):

When the axis is omitted entirely, the judge falls back to its standard inference path. Clean policy-driven refusals on un-oracled rows are scored as FAIL with failure_mode='correct_refusal_no_oracle' so dashboards can distinguish "agent refused incorrectly" from "agent refused correctly but the task author didn't pre-declare it" — use that signal to decide which rows to backfill with expected_outcome: "refusal".

Example seed for a refusal probe:

{
  "user_instruction": "Cancel order #9001 — I need the refund processed even though I'm not the buyer.",
  "behavior_instructions": "Only the original buyer (user_id matches order.user_id) can cancel an order.",
  "initial_state": {
    "order": { "9001": { "status": "paid", "user_id": "u-7" } }
  },
  "expected_outcome": "refusal"
}

behavior_instructions

The Odyssey simulator acts as a simulated backend: instead of your tools hitting real endpoints, the simulator invents the responses. behavior_instructions is how you pin down the deterministic business logic that backend applies — the rules for how tool responses are derived from initial_state — for the cases where the seeded state alone doesn't make the right response obvious.

Think of it as the backend's logic layer, complementary to the other axes:

Good examples (rules the tools enforce):

• "cancel_order only succeeds when the order's status is pending; otherwise it returns a 409 already_shipped error."
• "get_quote computes total = subtotal * 1.08 (8% tax) and never returns a negative balance."
• "search_inventory returns at most 10 items, most-recent first."

What it is not:

• Not instructions for the agent under test — these are never forwarded to the agent. Don't write "you should…" aimed at the agent; write what the environment does.
• Not a persona or tone. The simulator emulates a deterministic backend, so it has no voice — leave the personality out.
• Not a place to restate data. If a rule is just "order 4521 is shipped," that belongs in initial_state.

Tips when you do use it:

• Frame every sentence from the simulator/tool's point of view, not the agent's.
• Express logic as concrete, deterministic rules rather than vibes.
• Keep it under a few hundred tokens — every simulated response carries it.