Operational Notes: Orchestrating Multi-Agent CLI Councils Inside Herdr

Terminal multiplexing frameworks like Herdr bridge coordinating agents and autonomous worker CLIs (such as Codex, Grok, and Anti-Gravity). This allows you to spin up multiple agents from different providers and have them all talk to each other in real time. This is pretty amazing! However, treating agent terminal sessions like standard shell scripts introduces distributed-state failure modes: silently swallowed execution commands, premature completion signals, and unmessageable background processes.

These operating notes document the mechanics, harness failure modes, and recovery patterns discovered while orchestrating a three-seat prompt review council across parallel terminal panes.

This memo starts with advice for us humans and culminates a prompt you can drop into your agent to help it operate in the herdr environment.

1. Worker Topology: Headless Execution vs. Messageable Agents

There are three ways to invoke a worker CLI within a managed pane, but only one yields an agent you can interact with mid-flight:

[Coordinating Agent]
       │
       ├── herdr pane run ... (Headless / Detached: stdin closed; no interactive steerability)
       ├── background shell ... (Orphan process: unmonitored; lifecycle untracked)
       └── herdr agent start ... (Messageable: managed lifecycle states; interactive stdin)

Ingestion Pattern	Command Structure	Lifecycle Observability	Mid-Flight Steering
Headless Pane Run	herdr pane run <pane> "codex exec ..."	None (blind terminal execution)	No. Stdin is closed. Mid-task interventions require killing or brittle disk-session resumes (codex exec resume, grok -c).
Background Shell	sh -c "agy ... &"	Process-level only	No. Completely unmanaged by the multiplexer.
Interactive Herdr Agent	herdr agent start <name> --kind <kind> --pane <pane> -- <flags>	Discrete states: idle, working, blocked, done	Yes. Accepts follow-up turns via herdr agent prompt <name> "..." --wait.

Why Messageability Matters

During our benchmark runs, a worker-generated parsing toolkit failed against real-world reviewer outputs. Because the worker had been initialized via herdr agent start, remediation required a single instruction via herdr agent prompt containing the failing sample. The worker reviewed the patch, identified seven defects, repaired two, generated regression tests, and completed the fix without rebuilding context from scratch. A headless one-shot worker would have required a total re-briefing.

2. Runtime Failure Modes in Terminal Multiplexing

Automated terminal monitoring fails when orchestrators rely on standard shell heuristics:

• Leftover Pane Processes Swallow Launches: herdr pane run operates only when the target pane sits at an active shell prompt. If a prior step left a streaming process running (e.g., tail -F output.log), the new command is swallowed silently. The pane logs no error, launches no process, and leaves the output files empty. Always inspect pane process state via herdr pane process-info and force clearance with ctrl+c before dispatching.

• The clear; Mask: Prefixing automated pane commands with clear; can discard standard output streams entirely in certain virtual terminal emulators without surfacing runtime errors. Never prepend clear commands.

• State Flicker Between Tool Calls: On long-running jobs, herdr agent wait or agent state monitors can momentarily flicker to done while an agent transitions between sequential tool calls. Reading outputs on this transient state yields truncated payloads. For operations taking longer than two minutes, mandate that a non-working state hold for 60 seconds (four checks at 15-second intervals) before releasing the lock.

• Process Name Drift: Never evaluate agent health using pgrep on the CLI binary name. Provider wrapper CLIs spawn child processes under runtime-specific names (Node runtimes, Python virtualenvs, helper binaries), generating false completion signals. Track execution via explicit filesystem completion sentinels written by the invocation wrapper.

3. CLI Harness Quirks

Each frontier agent CLI exhibits distinct runtime behaviors that must be handled prior to launching automated reviews:

Codex (codex exec)

• Stdin Blocking: By default, Codex polls stdin and hangs indefinitely on "Reading additional input from stdin" unless piped from null: </dev/null.

• Output Isolation: Redirect cleanly using -o <path> to prevent interactive logs, search queries, and reasoning traces from contaminating downstream file parsers.

Grok (grok)

• Narration Bleed: Grok prepends unformatted status narration to stdout ahead of Markdown headers. Pipelines parsing Grok outputs require a mechanical regex strip (^## ) to discard conversational preamble before downstream ingestion.

• Permission Flags: Run with explicit permission bypasses (e.g., --permission-mode acceptEdits or plan) depending on whether the seat is auditing or modifying code.

Anti-Gravity (agy)

• Flag Incompatibilities: The --effort flag accepts low|medium|high on Gemini models, but throws unhandled exceptions on Claude models (e.g., claude-opus-4-6-thinking). Omit --effort entirely when dispatching Claude workloads.

• Silent Artifact Dumping: Certain models running under Anti-Gravity output a brief narrative summary to stdout while saving the actual deliverable to an internal directory (~/.antigravity/...). Prompt instructions must explicitly mandate printing the full unabridged report directly to stdout.

• Silent Quota Exhaustion: Rate limits or token quotas often terminate executions midway while printing only a single error line to stderr. Always inspect stderr size before trusting short outputs.

4. Workspace Safety: Clean Snapshotting

When coordinating review agents while human developers or secondary coding agents operate in the same repository, never grant review agents direct git tree access. Git branch switching, index locks, or unstaged diffs will cause race conditions.

Isolate reviewers into ephemeral scratchpads using tar snapshots:

# Export an immutable tree snapshot to an isolated scratchpad
git archive <target-sha> . | tar -x -C /tmp/scratchpad/src-<target-sha>

# Verify repository working directory remains completely untouched
git status --porcelain

5. Drop-In Coordinator System Directive

Copy the block below directly into the system prompt of any LLM agent tasked with coordinating workers inside Herdr:

=== HERDR WORKER COORDINATION DIRECTIVES ===

1. ENVIRONMENT & INITIALIZATION
- Verify environment: check `test "$HERDR_ENV" = 1`.
- Allocate one pane per seat at startup: `herdr tab create --label <seat> --no-focus`.
- Name panes after worker seats (`herdr pane rename`), never transient task names.
- Never run smoke tests as raw background subshells from the coordinator pane.

2. WORKER DISPATCH CONTRACT
- ALL messageable agents MUST be started using:
  `herdr agent start <name> --kind <kind> --pane <pane> -- <cli-flags>`
- Dispatch tasks and await completion using:
  `herdr agent prompt <name> "<prompt-text>" --wait`
- DO NOT use `herdr pane run` or raw background bash commands (`&`) for tasks 
  requiring corrections; these run headless with stdin closed.
- Issue mid-task adjustments via `herdr agent prompt`, NOT by attempting disk 
  resumes (`codex exec resume`, `grok -c`).

3. PANE HYGIENE & VALIDATION
- Before dispatching to a pane, inspect: `herdr pane process-info --pane <id>`.
- Clear lingering tasks (e.g., active `tail -F`): `herdr pane send-keys <id> ctrl+c`.
- NEVER execute commands with a `clear;` prefix.
- Validate agent completion against explicit sentinel files or stable agent states.
- Beware of state flicker: verify that `done` holds stable for 60 seconds on complex jobs.

4. SEAT-SPECIFIC HARNESS RULES
- CODEX: Append `</dev/null` to every `codex exec` invocation to block stdin hangs.
  Capture output via `-o <file>`.
- GROK: Sanitize stdout by stripping all text preceding the first Markdown header (`## `).
- ANTI-GRAVITY: Do NOT pass `--effort` to Claude models. Explicitly instruct the 
  model to output its complete text to stdout rather than writing internal artifacts.
- GLOBAL: Run review seats in explicit read-only or plan modes. Disable the local 
  sandbox restrictions when external API or socket calls are required.

5. CONCURRENCY & REPOSITORY PROTECTION
- NEVER allow review agents direct git access in shared working trees.
- Export immutable source trees using:
  `git archive <sha> . | tar -x -C <scratchpad>/src-<sha>`
===========================================

---

Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email jason.vertrees@gmail.com.