Agent orchestration patterns

Some tasks are too big for one AI agent, so developers split them across several. Someone then has to decide which agent runs when. They also have to decide what happens if one agent stops partway. The common answers have names, and public guides from Anthropic and Microsoft describe them.

One after another

The simplest arrangement passes the work along a line. Microsoft's guide to agent orchestration says the sequential pattern "chains AI agents in a predefined, linear order". Each agent works on what the agent before it produced. Microsoft adds that people also call this a pipeline, prompt chaining or linear delegation.

Anthropic's guide, published on 19 December 2024, describes prompt chaining in the same way. A developer splits a task into a sequence of steps, "where each LLM call processes the output of the previous one". Anthropic suggests adding checks between the steps to keep the process on track.

Side by side, then gathered

When parts of a task do not depend on each other, they can run at the same time. Microsoft's concurrent pattern "runs multiple AI agents simultaneously on the same task". Its other names include fan-out/fan-in and scatter-gather.

Anthropic calls this parallelization and describes two forms. In sectioning, a developer breaks a task into independent subtasks that run in parallel. In voting, the developer runs the same task several times to get a range of outputs.

A supervisor and its workers

Some tasks cannot be split up in advance. In Anthropic's orchestrator-workers pattern, "a central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results". Anthropic suggests it for complex tasks where the developer can't predict the subtasks needed.

The OpenAI Agents SDK documents a related idea, the handoff. Its documentation says handoffs "allow an agent to delegate tasks to another agent".

When a step fails

Several agents bring the usual troubles of any distributed system. Microsoft names node failures, network partitions, message loss and cascading errors. Its first piece of advice is to "Implement timeout and retry mechanisms."

In Microsoft's Retry pattern, the application waits between attempts, and the wait can grow each time. The same guide asks whether the step is idempotent, which means safe to run twice. "If so, it's inherently safe to retry." Where finished steps have to be undone, the Compensating Transaction pattern describes a workflow that "rewinds through the completed steps and reverses each step".

Temporal's documentation describes another way to recover. Temporal keeps an Event History, "a complete, ordered log of everything that has already happened in a Workflow". It uses that history to bring a run back to the state it was in before it stopped.

A person in the loop

Some steps need a person to approve them before they take effect. Microsoft writes that several orchestration patterns support a human in the loop (HITL).

Where approval is mandatory, Microsoft advises saving the run's state at that checkpoint. The run can then resume without repeating the agents' earlier work. Anthropic's guide says agents "can then pause for human feedback at checkpoints or when encountering blockers".

Where the research is

Two papers on arXiv give a wider view. "Large Language Model based Multi-Agents: A Survey of Progress and Challenges" asks how these agents are profiled and how they communicate. The AutoGen paper describes an open-source framework whose agents "converse with each other to accomplish tasks".

Useful links