Can AI Models Direct Each Other? Organizational Structure as a Probe into Training Limitations | AI Paper Digest

TL;DR Highlight

Having an expensive AI direct a cheap AI can achieve performance on par with the expensive AI alone — at a fraction of the cost, but only when there's a real capability gap between them.

Who Should Read

Backend/ML engineers designing LLM-based coding agents or multi-agent pipelines. Developers who need to optimize cost-performance tradeoffs and are deciding which model to assign to which role.

Core Mechanics

Claude Sonnet 4.6 (manager) + GPT-5-mini (worker) achieves 62% resolution rate, matching Sonnet alone (60%), while saving ~5x strong-model tokens
Adding structure without a capability gap backfires — pairing GPT-5-mini with itself drops performance from 44% to 42%
Manager doing 'review only' yields +2pp; manager 'directly orchestrating exploration and planning' yields +11pp — active direction matters far more than simple review
Giving the manager repository access actually hurts performance — it causes hallucinations like 'already fixed.' Restricting the manager to text-only input forces proper abstract reasoning
Current models are trained to 'do everything solo,' making them poor at delegation and scoped execution
Pipeline code must serve as the organizational structure — phase transitions, round limits, and prompt strategy switches must not be left to the model

Evidence

"MANAGERWORKER (Sonnet 4.6 manager + GPT-5-mini worker) 62% vs Strong Direct (Sonnet alone) 60% — strong-model tokens reduced ~78% from ~30K to 6.6K (200 instances); Weak→Weak (GPT-5-mini manager+worker) 42% vs Weak Direct (GPT-5-mini alone) 44% — structure introduction causes -2pp with worst total token usage at 75K (50 instances); Simple Loop (review only) 53% vs Full Pipeline (exploration+planning+implementation) 62% — structured orchestration contributes additional +9pp; Guided-then-Strict strategy outperforms Strict-only by +10pp (52%→64%), reducing empty patches from 8 to 0 (50-instance ablation)"

How to Apply

"In coding agent pipelines that require cost optimization, assign the strong model exclusively to text-only (no file access) analysis, planning, and review, while the cheaper model handles actual code execution and file exploration; Design the worker prompt strategy as 'guided on round 1, strict on retries' — round 1 lets the worker explore actual code structure freely, and on failure the manager issues precise corrective instructions; Have the manager decompose tasks into no more than 3 exploration subtasks at a time, and require workers to return only natural-language summary reports — this enables iterative exploration without context window explosion"

Code Example

snippet

# Phase 1: Manager analyzes issue (text-only, no repo access)
ANALYSIS_PROMPT = """
You are a senior software engineer analyzing a GitHub issue.

## Issue
{problem_statement}

## Your Task
Identify 2-3 specific exploration tasks that a junior
engineer should perform to gather the information needed
to fix this issue.

Each task should be a focused investigation:
- "Find the X method in Y file and report its signature"
- "Check how Z handles the case when W is None"

Output each task on its own line starting with "TASK: "
"""

# Phase 4 Round 1: Guided (worker has autonomy)
IMPL_GUIDED_PROMPT = """
You are fixing a bug in '{repo}'.

## Analysis & Plan
{plan}

## Your Task
1. Read the relevant file(s) to understand current code structure.
2. Make the changes described in the plan.
3. Run 'git diff' to produce your patch.

Use your judgment on the exact implementation.
Do NOT modify test files. Keep changes minimal.
"""

# Phase 4 Round 2+: Strict (follow corrections exactly)
IMPL_STRICT_PROMPT = """
A senior engineer has reviewed your previous attempt.

## Corrected Instructions -- follow these EXACTLY
{prior_feedback}

## Rules
1. Apply ONLY the changes described above.
2. Do NOT modify test files.
3. Output the complete git diff as your final answer.
"""

Terminology

SWE-bench LiteA benchmark measuring how well AI resolves real GitHub issues. Tasks involve fixing bugs in real Python projects by producing code patches.

monolithic agentA single AI agent that handles analysis, planning, code writing, and execution all by itself. Most current coding agents like Claude Code and OpenHands follow this approach.

training distributionThe range of data patterns a model has seen during training. Models perform well within this range but tend to behave unpredictably outside it.

delegationAssigning a task to another agent rather than doing it yourself. The ability to give instructions at the right level of abstraction — e.g., 'find method Y in file X and report back.'

scoped executionThe ability to execute only within a given scope and report results. Performing exactly what was instructed without considering the big picture — current AI tends to drift beyond its assigned scope.

Pareto frontierThe set of optimal combinations where neither cost nor performance can be improved without sacrificing the other. Used here to analyze the tradeoff between token cost and resolution rate.

ablationAn experiment that removes or simplifies a specific component to measure its contribution. E.g., 'How much does performance drop if we remove the structured exploration phase?'

Related Papers

Related Resources

MANAGERWORKER GitHub Repository (SWE-bench eval infrastructure)

Original Abstract (Expand)

Can an expensive AI model effectively direct a cheap one to solve software engineering tasks? We study this question by introducing ManagerWorker, a two-agent pipeline where an expensive "manager" model (text-only, no code execution) analyzes issues, dispatches exploration tasks, and reviews implementations, while a cheap "worker" model (with full repo access) executes code changes. We evaluate on 200 instances from SWE-bench Lite across five configurations that vary the manager-worker relationship, pipeline complexity, and model pairing. Our findings reveal both the promise and the limits of multi-agent direction: (1) a strong manager directing a weak worker (62%) matches a strong single agent (60%) at a fraction of the strong-model token usage, showing that expensive reasoning can substitute for expensive execution; (2) a weak manager directing a weak worker (42%) performs worse than the weak agent alone (44%), demonstrating that the directing relationship requires a genuine capability gap--structure without substance is pure overhead; (3) the manager's value lies in directing, not merely reviewing--a minimal review-only loop adds just 2pp over the baseline, while structured exploration and planning add 11pp, showing that active direction is what makes the capability gap productive; and (4) these behaviors trace to a single root cause: current models are trained as monolithic agents, and splitting them into director/worker roles fights their training distribution. The pipeline succeeds by designing around this mismatch--keeping each model close to its trained mode (text generation for the manager, tool use for the worker) and externalizing organizational structure to code. This diagnosis points to concrete training gaps: delegation, scoped execution, and mode switching are skills absent from current training data.