Trajectories

trajectories is the canonical trajectory data and conversion layer for Control Tower.

A trajectory is a structured record of agent behavior for a sample:

  • environment/main-task/side-task context
  • action/tool-call history
  • timing/cost/monitor metadata
  • optional run linkage (run_id) and eval linkage (eval_log_url, sample ID)

This package handles:

  • core data model + validation (trajectory.py)
  • conversion from eval logs/tool calls/paths/IDs (convert.py, from_eval_log.py, from_tool_calls.py)
  • storage and retrieval (filesystem/API helpers in io.py, api.py) and Mongo document building (mongo_docs.py)
  • analysis utilities (analysis.py)
  • trajectory-focused CLI commands (cli.py)
  • TUI support for interactive trajectory review/edit (in control_tower.live.tui)

CLI Overview

Top-level command group:

  • ct traj ... (wired from control_tower.trajectories.cli:traj_cli)

Primary commands:

  • ct traj make <TRAJ_LIKE> [--sample-id ID] [-e ENV] [-t MAIN_TASK] [-s SIDE_TASK] [--save-path PATH]

    • Converts trajectory-like input into one trajectory JSON.
    • TRAJ_LIKE can be a file path, directory, URL, trajectory ID, run ID, or eval-like input.

  • ct traj list [--fs] [-e ENV] [-t MAIN_TASK] [-s SIDE_TASK] [--tag TAG] [--limit N]

    • Lists trajectories from the API (default) or the local filesystem (--fs).

  • ct traj download [<trajectory_id_or_viewer_url>] [--eval] [-e ENV] [-t MAIN_TASK] [-s SIDE_TASK] [--tag TAG] [-n N] [--dataset NAME] [--dataset-version V] [-o OUTPUT] [--overwrite]

    • Downloads one trajectory (by ID or viewer URL); bulk-downloads the latest N matching trajectories when filter options are passed instead of an ID (max 500, default 20); or downloads every trajectory in a dataset with --dataset NAME (no count limit).
    • --eval downloads the trajectory's full .eval archive instead of the bundle JSON.
    • --dataset fetches the dataset's full trajectory ID list, then downloads those trajectories by ID in batches, so the whole dataset arrives completely (no auth or Mongo URI required) instead of truncating on a single oversized stream. Add --dataset-version V to download a specific version (default: latest).
    • Accepts a trajectory ID from the configured API/MongoDB or a viewer trajectory URL (e.g. https://basharena-data.vercel.app/trajectories/<id>). See Downloading from the Viewer.

  • ct traj upload <path_to_trajectory_json>

    • Uploads a local trajectory JSON to MongoDB.

  • ct traj patch <traj> ...

    • Generates a patch from a trajectory using live tooling.

  • ct traj apply <eval_log_file> ...

    • Applies file edits from an eval log into an environment codebase.

  • ct traj tui <trajectory_path> ...

    • Opens the trajectory TUI for interactive review/editing.

  • ct traj top-sus <TRAJ_LIKE> [--limit N] [--min-score X] [--env ENV] [--full] [--json]

    • Shows top suspicious actions using monitor scoring attached to actions.

Cost fields on a Trajectory

Each converted Trajectory carries both uncached-equivalent cost and actual API spend. Inspect response-cache reads preserve the original call's usage metadata, so their cost_usd remains the price the call would incur without the Inspect replay while actual_cost_usd is $0. Provider prompt-cache discounts remain part of both prices because those are real API pricing, not local response-cache replays.

  • Per-call: action.timing.model_calls[i].cost_usd is uncached-equivalent cost and actual_cost_usd is actual spend. Both are priced per call using pricing.select_tier(input_tokens), so tiered models (e.g. Gemini) pick the right tier per call.
  • Agent total: Trajectory.agent_cost is the per-model uncached-equivalent sum; Trajectory.actual_agent_cost is the per-model actual-spend sum and actual_agent_cost_usd is its scalar total.
  • Legacy aggregate: Trajectory.cost: TrajectoryCost carries uncached-equivalent total_usd; the input/output/cached breakdown still comes from aggregate token_usage (an approximation for tiered models).
  • Structured breakdown: cost_breakdown.traj_task_cost carries total_usd / by_model alongside actual_total_usd / actual_by_model.

Monitor costs live on each MonitorResult.cost_usd (see Cost Tracking in function-registry.md). Helpers on Trajectory:

  • trajectory.monitor_cost() -> dict[str, float] — per-model sum across action and trajectory monitor results. Ensemble-safe: walks members rather than double-counting the parent.
  • trajectory.total_cost() -> dict[str, float] — agent + monitor, per model.
  • trajectory.total_cost_usd() -> float — grand total.

Cost tracking is automatic; MonitorResult.cost_usd=None means a call used an unpriced model, and the helpers skip those silently.

How trajectories fits with other CLI groups

  • ct run ... runs experiments and can produce trajectories.
  • ct runs ... manages run-level containers around groups of trajectories.
  • ct traj ... is the direct trajectory manipulation and analysis surface.

Typical workflows

Convert input to a trajectory JSON:

  • uv run ct traj make logs/sample.eval -o data/traj/sample.json

List trajectories from API:

  • uv run ct traj list -n 20

Download one trajectory:

  • uv run ct traj download <trajectory_id>
  • uv run ct traj download https://basharena-data.vercel.app/trajectories/<id> (viewer URL)
  • uv run ct traj download <trajectory_id> --eval (full .eval archive instead of bundle JSON)

Bulk-download the latest matching trajectories:

  • uv run ct traj download -e web_scraping --tag baseline -n 100

Download every trajectory in a dataset (latest version, or a specific one):

  • uv run ct traj download --dataset LaStraj
  • uv run ct traj download --dataset LaStraj --dataset-version 2

Inspect suspicious actions:

  • uv run ct traj top-sus data/run_trajs/<run_id> --limit 15