---
title: "How to Evaluate AI Agents: An Engineering Framework"
slug: evaluating-ai-agents-framework
description: "Learn how to evaluate autonomous AI agents using ephemeral Docker sandboxes, bounded loops, and runtime protocol execution for reliable benchmarks."
date: 2026-07-15
last_reviewed: 2026-11-02
status: published
type: knowledge-base
author: [liyam-flexer]
image: evaluating-ai-agents-framework
image_alt: "Cover graphic: How to Evaluate AI Agents: An Engineering Framework, over a dark blue network of connected points."
tags: [ai agents, model evaluation, model context protocol, agent evaluation, software engineering, docker]
---

# How to Evaluate AI Agents: An Engineering Framework

**In short:** Single-turn prompt benchmarks measure text generation quality, not autonomous agent capability. Robust evaluation of tool-using, multi-turn AI agents requires isolated ephemeral Docker sandboxes, hard step ceilings to prevent infinite loops, and deterministic runtime execution grading (such as a Model Context Protocol handshake succeeding) rather than text similarity or LLM-as-a-judge scoring alone. The TypeScript below is an illustrative harness sketch, not a tested library.

Autonomous systems that read files, invoke external APIs, execute bash commands, and iterate based on environmental feedback cannot be measured with traditional static question-and-answer datasets. When an agent loops indefinitely, hallucinates tool parameters, or leaves an engineering task partially completed without error handling, text-based graders fail to catch the malfunction.

This engineering blueprint details the architecture for multi-turn agent evaluation: sandboxed execution containers, stateful agent loops, and automated protocol compliance grading using the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/introduction), an open-source standard [introduced by Anthropic on 25 November 2024](https://www.anthropic.com/news/model-context-protocol), as a machine-verifiable benchmark deliverable. All code samples are illustrative: they show the structure of a harness and have not been run as a package.

---

## Why Single-Turn Benchmarks Fail for Autonomous Agents

The difference between single-turn model evaluation and multi-turn agent evaluation is architectural. Standard LLM evaluations (like MMLU, GSM8K, or HumanEval; agentic benchmarks such as [SWE-bench](https://arxiv.org/abs/2310.06770) already move toward real repositories) treat language models as stateless pure functions: one prompt enters, one string leaves, and a regex or semantic similarity check scores the output. 

Autonomous agents, by contrast, operate inside stateful environments where actions produce irreversible side effects.

```
SINGLE-TURN EVALUATION (Stateless):
[ Prompt ] ───► [ LLM Generation ] ───► [ String Similarity / Regex Match ]

MULTI-TURN AGENT EVALUATION (Stateful World State):
                 ┌────────────────────────────────┐
                 ▼                                │
[ Task Goal ] ──► [ Reason ] ──► [ Action / Tool ] ──► [ Docker Environment ]
                                                        │ (Filesystem, Net, Proc)
                                                        ▼
[ Final Runtime Grade ] ◄─── [ Live Execution ] ◄───────┘
```

The table below contrasts the technical boundaries between text evaluation and agent runtime verification:

| Evaluation Dimension | Single-Turn Benchmark | Multi-Turn Agent Evaluation |
|---|---|---|
| **Input Surface** | Static text prompt | Task specification, schema definitions, ephemeral workspace |
| **Operational Loop** | Single forward inference pass | Reason → select tool → parse output → manage state → iterate |
| **Execution Environment** | Stateless execution | Persistent container state (filesystem, network, processes) |
| **Grader Verification** | Exact string match, regex, LLM-as-judge | Runtime execution: exit codes, test suites, protocol compliance |
| **Failure Modes** | Factual error, poor style, token truncation | Infinite loops, workspace corruption, tool schema violations |

Grading an agent on text output alone misses production-critical failures. An agent can generate syntactically clean TypeScript code that fails to compile, or produce a server entry point that crashes immediately on startup. True agent capability can only be measured by evaluating the resulting **world state**.

For broader context on agentic reasoning and autonomy boundaries, see [AI Developer Tools Compared](./ai-developer-tools-compared.html) and [Why My AI Agents Need Permission Slips](./my-ai-agents-dont-need-to-be-smarter-they-need-permission-slips.html), as well as the encyclopedia overview on [Agent Architectures](../chapters/ch-20.html).

---

## The Benchmark Deliverable: Building an MCP Server

To construct a repeatable agent benchmark, the task deliverable must have a deterministic, machine-verifiable pass condition without requiring subjective human grading. 

Building a server that complies with MCP is a reasonable engineering benchmark for four reasons:

1. **Deterministic Pass/Fail Criterion**: Over stdio, an MCP server either answers a protocol probe with a valid JSON-RPC response or it does not. Which probe applies depends on the protocol revision: revisions up to `2025-11-25` use an `initialize` handshake, while the [`2026-07-28` revision](https://modelcontextprotocol.io/specification/2026-07-28/server/discover) drops the handshake and has clients call `server/discover` (or send versioned requests directly). The grader below accepts either, and your task statement should say which revision and SDK the agent must target.
2. **Multi-Step Dependency Chain**: The agent must inspect requirements, scaffold a Node.js or Python environment, write schema-compliant code, install dependencies, compile TypeScript, and launch the process.
3. **Realistic Developer Workflow**: Implementing protocol specifications mirrors production engineering assignments far more accurately than solving toy algorithmic puzzles.
4. **Gradual Capability Stacking**: The test harness can incrementally assess advanced capabilities, such as `tools/list` schema validation, error response formatting, and SQL query execution.

---

## Agent Evaluation Harness Architecture

The harness separates orchestration, isolation, and evaluation into decoupled layers. The assumed setup is Node 22 with `tsconfig.json` set to CommonJS (the sketch uses `__dirname`) and `@types/node` installed:

```text
agent-eval-harness/
├── docker/
│   └── sandbox.Dockerfile      # Ephemeral sandbox container definition
├── tasks/
│   └── mcp-sqlite-server/
│       ├── task.json           # Instructions, timeouts, and resource budgets
│       ├── scaffold/           # Boilerplate plus pre-installed node_modules (no network in sandbox)
│       └── tests/              # Private test suites withheld from the agent
├── src/
│   ├── sandbox.ts              # Container lifecycle and process supervisor
│   ├── agent.ts                # Step-bounded execution loop
│   ├── tools.ts                # Sandboxed tool definitions
│   ├── grader.ts               # Protocol handshake and functional validator
│   └── run.ts                  # Test harness entrypoint
├── package.json
└── tsconfig.json
```

---

## 1. Ephemeral Sandbox Management (`sandbox.ts`)

Every benchmark run requires a completely isolated, clean environment to prevent state contamination between attempts. Running the agent's commands inside Docker containers with a mounted ephemeral directory gives reproducible execution and limits what the agent can touch. The sketch below uses argument arrays (`spawnSync`, `execFileSync`) rather than interpolating strings into a host shell, so agent-supplied paths and commands are not expanded by the host:

```typescript
import { execFileSync, spawn, spawnSync, ChildProcess } from 'child_process';
import * as fs from 'fs';
import * as path from 'path';

export interface SandboxConfig {
  taskId: string;
  scaffoldPath: string;
}

export interface RunResult {
  ok: boolean;
  output: string;
}

export class Sandbox {
  private containerName: string;
  public hostVolumePath: string;

  constructor(config: SandboxConfig) {
    this.containerName = `agent-eval-${config.taskId}-${Date.now()}`;
    this.hostVolumePath = path.resolve('/tmp/evals', this.containerName);

    // Copy the starter scaffold (including pre-installed node_modules) into an isolated directory
    fs.mkdirSync(this.hostVolumePath, { recursive: true });
    fs.cpSync(config.scaffoldPath, this.hostVolumePath, { recursive: true });
  }

  public start(): void {
    // No network, no extra capabilities, bounded memory and process count
    execFileSync('docker', [
      'run', '-d',
      '--name', this.containerName,
      '-v', `${this.hostVolumePath}:/workspace`,
      '-w', '/workspace',
      '--network', 'none',
      '--cap-drop', 'ALL',
      '--memory', '1g',
      '--pids-limit', '256',
      'node:22-alpine',
      'tail', '-f', '/dev/null',
    ]);
  }

  /** Run a command inside the container; never throws, returns combined output. */
  public run(argv: string[], timeoutMs = 15_000): RunResult {
    const r = spawnSync('docker', ['exec', this.containerName, ...argv], {
      timeout: timeoutMs,
      encoding: 'utf-8',
    });
    return { ok: r.status === 0, output: `${r.stdout ?? ''}${r.stderr ?? ''}` };
  }

  /** Start a long-lived process inside the container with piped stdio. */
  public spawn(argv: string[]): ChildProcess {
    return spawn('docker', ['exec', '-i', this.containerName, ...argv]);
  }

  /** Resolve a workspace-relative path and refuse anything that escapes the mount. */
  public resolveInside(relPath: string): string {
    const full = path.resolve(this.hostVolumePath, relPath);
    if (full !== this.hostVolumePath && !full.startsWith(this.hostVolumePath + path.sep)) {
      throw new Error(`Path escapes workspace: ${relPath}`);
    }
    return full;
  }

  public cleanup(): void {
    try {
      execFileSync('docker', ['rm', '-f', this.containerName]);
      fs.rmSync(this.hostVolumePath, { recursive: true, force: true });
    } catch (e) {
      console.error(`Cleanup failed for container ${this.containerName}:`, e);
    }
  }
}
```

Two critical architectural guardrails are enforced:
- `--network none` (plus `--cap-drop ALL`, memory and PID limits): isolates the agent from the network. The task scaffold must therefore ship its dependencies (for example the MCP SDK and TypeScript in `node_modules`), because `npm install` cannot reach a registry.
- Combined stdout and stderr in `run`: when commands fail (for example `tsc` type errors), returning the diagnostics gives the agent the feedback it needs to self-correct.

This is a baseline, not a complete security boundary. Docker isolation depends on host configuration; for untrusted model output consider a stronger sandbox (gVisor, a microVM) and never mount anything sensitive.

---

## 2. Bounded Agent Execution Loop (`agent.ts`)

Unbounded agent loops risk runaway token consumption when an agent gets stuck in cyclic errors. The evaluation loop must enforce a hard step ceiling. This sketch also appends every result to one growing string, so cost grows with each step; a real harness should use the provider's message and tool-use API and cap context size:

```typescript
export interface Tool {
  name: string;
  description: string;
  execute(args: Record<string, unknown>): Promise<string>;
}

export interface AgentAction {
  reasoning: string;
  toolName: string;
  toolArgs: Record<string, unknown>;
}

export class AgentScaffold {
  private tools: Map<string, Tool>;
  private readonly maxSteps: number;

  constructor(tools: Tool[], maxSteps = 20) {
    this.tools = new Map(tools.map(t => [t.name, t]));
    this.maxSteps = maxSteps;
  }

  public async executeTask(instructions: string): Promise<'completed' | 'halted'> {
    let step = 0;
    let context = `Task Directive:\n${instructions}\n\nAvailable Tools: ${[...this.tools.keys()].join(', ')}\n`;

    while (step < this.maxSteps) {
      step++;

      // Dispatch prompt and context to model client
      const action = await this.callLLM(context);

      if (action.toolName === 'final_submit') {
        console.log(`Agent declared completion at step ${step}.`);
        return 'completed';
      }

      const tool = this.tools.get(action.toolName);
      const output = tool
        ? await tool.execute(action.toolArgs).catch((e: Error) => `Execution error: ${e.message}`)
        : `Unknown tool: ${action.toolName}`;

      context += `\n[Step ${step}] Action: ${action.toolName} | Args: ${JSON.stringify(action.toolArgs)}\nResult:\n${output}\n`;
    }

    console.warn(`Step budget exhausted (${this.maxSteps} steps). Halting.`);
    return 'halted';
  }

  private async callLLM(context: string): Promise<AgentAction> {
    // Placeholder: call your model provider's SDK here and map its tool-call output to AgentAction.
    throw new Error('LLM client not configured.');
  }
}
```

---

## 3. Sandboxed Tool Definitions

Agents interact with the sandbox through a minimal, strictly typed surface:

```typescript
import * as fs from 'fs';
import * as path from 'path';
import { Sandbox } from './sandbox';
import { Tool } from './agent';

export function createAgentTools(sandbox: Sandbox): Tool[] {
  return [
    {
      name: 'run_command',
      description: 'Execute a shell command inside the workspace container and return stdout/stderr.',
      execute: async (args) => sandbox.run(['sh', '-c', String(args.command)], 60_000).output,
    },
    {
      name: 'write_file',
      description: 'Write string content to a workspace file, creating directories as needed.',
      execute: async (args) => {
        const target = sandbox.resolveInside(String(args.path));
        fs.mkdirSync(path.dirname(target), { recursive: true });
        fs.writeFileSync(target, String(args.content));
        return `Successfully wrote ${String(args.path)}`;
      },
    },
    {
      name: 'read_file',
      description: 'Read and return the raw text contents of a workspace file.',
      execute: async (args) => fs.readFileSync(sandbox.resolveInside(String(args.path)), 'utf-8'),
    },
    {
      name: 'list_directory',
      description: 'List files up to depth 2 at the given workspace path.',
      execute: async (args) =>
        sandbox.run(['sh', '-c', 'find "$1" -maxdepth 2 | sort', 'sh', String(args.path ?? '.')]).output,
    },
    {
      name: 'final_submit',
      description: 'Signal that the assignment is finished and submit the workspace for grading.',
      execute: async () => 'Submission registered.',
    },
  ];
}
```

The file tools operate on the mounted host directory, which is why `resolveInside` rejects paths that escape it. A thrown error (for example reading a missing file) is caught by the agent loop and reported back as `Execution error`.

---

## 4. Automated Runtime Grader (`grader.ts`)

Once the agent loop terminates, the grader checks the final workspace. It runs four checks, all inside the container so agent-written code never executes on the host: dependencies present, TypeScript compiles, an entry point exists, and the server answers a protocol probe over stdio.

The probe follows the MCP docs. For legacy revisions, the [lifecycle spec](https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle) has the client send `initialize` (with `protocolVersion`, `capabilities` and `clientInfo`), read the response, then send `notifications/initialized`. For the [`2026-07-28` revision](https://modelcontextprotocol.io/specification/2026-07-28/server/discover), a client sends `server/discover` with version and client metadata in `_meta` and gets a result with `supportedVersions`. The sketch sends both and passes if either succeeds. Per the [stdio spec](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/stdio), messages are newline-delimited and a server must not write anything to stdout that is not a valid MCP message, so any non-JSON stdout line fails the check.

```typescript
import { ChildProcess } from 'child_process';
import * as fs from 'fs';
import * as path from 'path';
import { Sandbox } from './sandbox';

export interface SubTestResult {
  name: string;
  passed: boolean;
  message: string;
}

const ENTRY_CANDIDATES = ['build/index.js', 'dist/index.js', 'index.js'];

export class MCPGrader {
  constructor(private readonly sandbox: Sandbox) {}

  public async grade(): Promise<{ score: number; results: SubTestResult[] }> {
    const results: SubTestResult[] = [];

    results.push(this.verifyDependencies());
    results.push(this.verifyCompilation());
    results.push(this.verifyEntryPoint());
    results.push(await this.verifyProtocolHandshake());

    const passedCount = results.filter(r => r.passed).length;
    const score = Math.round((passedCount / results.length) * 100);

    return { score, results };
  }

  private verifyDependencies(): SubTestResult {
    // Network is disabled, so check that the declared dependencies are already installed
    const r = this.sandbox.run(['npm', 'ls', '--depth=0'], 30_000);
    return { name: 'Dependencies Present', passed: r.ok, message: r.ok ? 'npm ls reports no problems.' : 'npm ls reported missing or invalid dependencies.' };
  }

  private verifyCompilation(): SubTestResult {
    const r = this.sandbox.run(['./node_modules/.bin/tsc', '--noEmit'], 60_000);
    return { name: 'Type Compilation', passed: r.ok, message: r.ok ? 'TypeScript compiled with zero errors.' : 'TypeScript compilation produced errors.' };
  }

  private findEntry(): string | undefined {
    return ENTRY_CANDIDATES.find(p => fs.existsSync(path.join(this.sandbox.hostVolumePath, p)));
  }

  private verifyEntryPoint(): SubTestResult {
    const entry = this.findEntry();
    return entry
      ? { name: 'Entry Point Check', passed: true, message: `Entry found at ${entry}` }
      : { name: 'Entry Point Check', passed: false, message: 'No entry point located.' };
  }

  private verifyProtocolHandshake(): Promise<SubTestResult> {
    const name = 'MCP Protocol Probe';
    const entry = this.findEntry();
    if (!entry) {
      return Promise.resolve({ name, passed: false, message: 'Missing entry point file.' });
    }

    return new Promise(resolve => {
      const proc: ChildProcess = this.sandbox.spawn(['node', entry]);
      let buffer = '';
      let settled = false;

      const finish = (result: SubTestResult) => {
        if (settled) return;
        settled = true;
        clearTimeout(timer);
        proc.stdin?.end(); // closing stdin is the stdio shutdown signal
        proc.kill();
        resolve(result);
      };

      const timer = setTimeout(
        () => finish({ name, passed: false, message: 'No valid response within 5000ms.' }),
        5_000
      );

      proc.on('error', err => finish({ name, passed: false, message: `Spawn failed: ${err.message}` }));

      proc.stdout?.on('data', (chunk: Buffer) => {
        buffer += chunk.toString();
        const lines = buffer.split('\n');
        buffer = lines.pop() ?? ''; // keep any partial line for the next chunk

        for (const line of lines) {
          if (!line.trim()) continue;
          let msg: any;
          try {
            msg = JSON.parse(line);
          } catch {
            finish({ name, passed: false, message: `Non-JSON line on stdout: ${line.slice(0, 80)}` });
            return;
          }
          if (msg.id === 'discover-1' && msg.result?.supportedVersions) {
            finish({ name, passed: true, message: `server/discover OK. Versions: ${msg.result.supportedVersions.join(', ')}` });
            return;
          }
          if (msg.id === 1 && msg.result?.protocolVersion) {
            proc.stdin?.write(JSON.stringify({ jsonrpc: '2.0', method: 'notifications/initialized' }) + '\n');
            finish({ name, passed: true, message: `initialize OK. Protocol version: ${msg.result.protocolVersion}` });
            return;
          }
        }
      });

      // Modern probe (2026-07-28 revision)
      proc.stdin?.write(JSON.stringify({
        jsonrpc: '2.0',
        id: 'discover-1',
        method: 'server/discover',
        params: {
          _meta: {
            'io.modelcontextprotocol/protocolVersion': '2026-07-28',
            'io.modelcontextprotocol/clientInfo': { name: 'eval-harness', version: '1.0.0' },
            'io.modelcontextprotocol/clientCapabilities': {},
          },
        },
      }) + '\n');

      // Legacy handshake (2025-11-25 and earlier)
      proc.stdin?.write(JSON.stringify({
        jsonrpc: '2.0',
        id: 1,
        method: 'initialize',
        params: {
          protocolVersion: '2025-11-25',
          capabilities: {},
          clientInfo: { name: 'eval-harness', version: '1.0.0' },
        },
      }) + '\n');
    });
  }
}
```

A server that supports only one era will return a JSON-RPC error to the other probe; this sketch ignores that error and waits for the matching success. The MCP spec recommends probing `server/discover` first and falling back only on specific outcomes; a production grader should follow the [stdio backward-compatibility rules](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/stdio) exactly.

---

## 5. End-to-End Orchestration (`run.ts`)

The evaluation lifecycle connects provisioning, loop execution, grading, and mandatory cleanup within a `try/finally` block:

```typescript
import { Sandbox } from './sandbox';
import { AgentScaffold } from './agent';
import { createAgentTools } from './tools';
import { MCPGrader } from './grader';
import * as path from 'path';
import * as fs from 'fs';

async function main() {
  const taskPath = path.resolve(__dirname, '../tasks/mcp-sqlite-server');
  const taskConfig = JSON.parse(fs.readFileSync(path.join(taskPath, 'task.json'), 'utf-8'));

  const sandbox = new Sandbox({
    taskId: taskConfig.id,
    scaffoldPath: path.join(taskPath, 'scaffold'),
  });

  console.log('Initializing ephemeral sandbox...');
  sandbox.start();

  let exitCode = 0;
  try {
    const tools = createAgentTools(sandbox);
    const agent = new AgentScaffold(tools, 20);

    console.log('Executing agent loop...');
    const status = await agent.executeTask(taskConfig.instructions);
    console.log(`Agent run status: ${status}`);

    console.log('Grading environment artifacts...');
    const grader = new MCPGrader(sandbox);
    const { score, results } = await grader.grade();

    results.forEach(r => console.log(`[${r.passed ? 'PASS' : 'FAIL'}] ${r.name}: ${r.message}`));
    console.log(`Aggregate Capability Score: ${score}/100`);

    exitCode = score === 100 ? 0 : 1;
  } finally {
    console.log('Teardown: cleaning container and host mount...');
    sandbox.cleanup();
    process.exit(exitCode);
  }
}

main().catch(err => {
  console.error('Fatal harness error:', err);
  process.exit(2);
});
```

---

## Failure Modes a Runtime Grader Can Catch

The list below is a set of failure modes this kind of harness is designed to surface. It is the author's engineering expectation, not a measured result: no model comparison or failure rates are reported here, and you should record your own.

```
┌──────────────────────────┬───────────────────────────────┐
│ 1. Circular Debugging    │ Repeating failed shell syntax │
│                          │ without inspecting stdout     │
├──────────────────────────┼───────────────────────────────┤
│ 2. Schema Non-Compliance │ Malformed JSON-RPC envelope   │
│                          │ or missing required fields    │
├──────────────────────────┼───────────────────────────────┤
│ 3. Build Phantom Finish  │ Declaring final_submit before │
│                          │ compiling TypeScript          │
├──────────────────────────┼───────────────────────────────┤
│ 4. Stdout Pollution      │ Logging to stdout instead of  │
│                          │ stderr on a stdio server      │
└──────────────────────────┴───────────────────────────────┘
```

1. **Circular debugging loops**: Without early-exit triggers, an agent that hits a missing package can retry variants of the same broken command until it exhausts its step budget.
2. **Schema non-compliance**: The server returns a malformed envelope or omits fields such as `protocolVersion` or `supportedVersions`, so the probe never succeeds.
3. **Build artifact phantoms**: The agent writes `src/index.ts`, calls `final_submit`, and never runs the build. The source can look fine to an LLM judge, but the grader finds no executable file.
4. **Stdout pollution**: On stdio, the [spec says](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/stdio) a server must not write anything to stdout that is not a valid MCP message, and may use stderr for logging. A stray `console.log("Starting server...")` breaks parsing, which the grader above reports as a non-JSON stdout line.

---

## Frequently asked questions

**What is the difference between single-turn and multi-turn agent evals?**
Single-turn evaluations score one prompt-response pair, often with string matching or an LLM judge. Multi-turn agent evaluations place a model in an environment with tools, let it act over several steps, and grade the resulting state with executable checks such as builds, tests and protocol probes. The second approach catches side effects and unfinished work that text graders miss.

**Why are Docker sandboxes preferred over mock environments?**
Containers give a real filesystem, processes and permissions, so failures that mocks hide (missing binaries, path issues, process lifecycle) show up. They also keep agent-generated code off the host. Docker is not a perfect boundary, so combine it with no network, dropped capabilities, resource limits, and a stronger sandbox for untrusted output.

**Why is MCP a good benchmark deliverable?**
An MCP server has an objective pass condition: over stdio it must answer a protocol request with a valid JSON-RPC response. The task also exercises reading a specification, scaffolding a project, building it and running it. Check which protocol revision your task targets, since the 2026-07-28 revision replaced the initialize handshake with server/discover.

**How should partial task completion be scored?**
Split the objective into independent checks, such as dependencies, compilation, entry point and protocol probe. Score each pass or fail and report both the aggregate and the per-check results. Equal weights, as in the sketch above, are a simplification; weight checks by how much each matters to your use case.

**What is a good step budget for agent benchmarks?**
There is no standard value. The 20-step ceiling in this sketch is an arbitrary default. Pick a budget by running your own tasks, looking at how many steps successful runs need, and setting the cap somewhat above that, while also bounding wall-clock time and token spend.

---

## Sources

1. [Model Context Protocol: Introduction](https://modelcontextprotocol.io/introduction) and Anthropic's announcement, [Introducing the Model Context Protocol](https://www.anthropic.com/news/model-context-protocol) (25 November 2024).
2. [MCP specification, 2025-11-25: Lifecycle](https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle) (initialize handshake).
3. [MCP specification, 2026-07-28: server/discover](https://modelcontextprotocol.io/specification/2026-07-28/server/discover), [Transports](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports) and [stdio](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/stdio).
4. [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk) (v2 uses the `@modelcontextprotocol/server` package; v1 uses `@modelcontextprotocol/sdk`). The harness above does not call the SDK; it only probes a finished server over stdio.
5. Jimenez et al., [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770) (ICLR 2024), cited as an example of benchmarking agents on real repository tasks.