Orchestration Principles

A condensed set of operating principles for multi-agent systems. Each rule names a failure mode and the design move that mitigates it. Treat agents as unreliable distributed workers: non-deterministic, occasionally slow, capable of confidently wrong output. The discipline that makes microservices survivable — explicit contracts, bounded blast radius, observability, least privilege — applies directly.

Prefer the Simplest Pattern That Works

Most "multi-agent" problems are a sequential pipeline or a single supervisor with a few workers. Reserve peer-to-peer and hierarchical graphs for cases where you can articulate why a simpler topology fails.

# Start here. Add agents only when a single one demonstrably can't cope.
result = await agent.run(task)
if result.needs_specialist:
    result = await supervisor.delegate(task, workers=[search, synth])

See /courses/05-agent-orchestration/lesson-02/ for the full pattern catalogue.

Make Handoffs Explicit

A handoff is an API boundary; define a schema for what crosses it. Never pass an entire conversation transcript when a structured summary will do — token cost compounds and downstream agents over-anchor on irrelevant detail.

from pydantic import BaseModel

class ResearchHandoff(BaseModel):
    query: str
    findings: list[str]
    sources: list[str]       # provenance travels with the claim
    confidence: float        # let the next agent reason about trust

# Validate at the boundary, before the next agent ever sees it.
payload = ResearchHandoff.model_validate(coordinator_output)

Treat shared-state writes the same way: a blackboard is eventually consistent, so give every entry a key, a writer ID, and a version. See /courses/05-agent-orchestration/lesson-03/.

Design for Partial Failure

Some sub-tasks will succeed while others fail; the system must make progress with partial results and never leave a workflow wedged. Make agent steps idempotent so retries are safe, and decide explicitly what a partial result is worth.

import random, asyncio

async def retry(call, attempts=4, base=0.5, budget_left=lambda: True):
    for i in range(attempts):
        if not budget_left():
            raise BudgetExhausted()
        try:
            return await call()                      # call must be idempotent
        except TransientError:
            if i == attempts - 1:
                raise
            await asyncio.sleep(base * 2**i + random.random())

Workflow engines (Temporal, Prefect, Airflow) give you durable retries, timeouts, and resumable state — wrap each non-deterministic agent call as an activity so a crash resumes from the last completed step rather than re-running the whole graph. See /courses/05-agent-orchestration/lesson-06/.

Bound Total Cost

Non-deterministic agents loop, re-plan, and re-fetch. Without hard limits a single run can consume unbounded tokens and dollars. Bound the dimensions below independently and fail closed when any is hit.

class Budget:
    def __init__(self, max_tokens, max_steps):
        self.tokens_left = max_tokens
        self.steps_left = max_steps

    def charge(self, used_tokens):
        self.tokens_left -= used_tokens
        self.steps_left -= 1
        if self.tokens_left <= 0 or self.steps_left <= 0:
            raise BudgetExhausted()   # fail closed, never loop on empty

Optimise within the budget by routing cheap tasks to faster/smaller models, caching repeated context (prompt caching), and caching identical sub-task results. See /courses/05-agent-orchestration/lesson-09/.

Keep Traces

A multi-agent run is a distributed trace; without correlation you cannot answer "which agent caused this output?" Propagate one run ID through every call, tool invocation, and handoff, and emit structured events — not prose logs.

from opentelemetry import trace
tracer = trace.get_tracer("orchestrator")

with tracer.start_as_current_span("agent.synthesis") as span:
    span.set_attribute("run.id", run_id)
    span.set_attribute("agent.name", "synthesis")
    out = await synth.run(payload)
    span.set_attribute("tokens.total", out.usage.total_tokens)

Persist enough to replay a failed run deterministically against recorded inputs. See /courses/05-agent-orchestration/lesson-08/.

Isolate Agent Permissions

Each agent should hold the narrowest set of tools and credentials its job requires; a compromised or hallucinating agent must not reach beyond its lane. Authenticate agent-to-agent calls and log every privileged action.

search_agent = Agent(
    name="search",
    tools=[web_search],          # read-only, no filesystem, no payments
    instructions="Retrieve and cite. Never write to shared stores.",
)
writer_agent = Agent(
    name="writer",
    tools=[write_doc],           # scoped to one output location
    instructions="Persist the approved draft only.",
)

Least-privilege tooling, agent-to-agent authentication, and audit logging are production gates covered in /courses/05-agent-orchestration/lesson-10/.

Related References