The Problem
Big Band is the tool I lead at Pickford for turning a ticket into a reviewed, landed change. An operator labels a ticket in our issue tracker; Big Band starts an LLM coding agent in a persistent workspace, has a second agent from a different model provider review the result, routes findings back for repair, and hands a clean pull request to a human. The implementation half of that loop worked sooner than I expected. The merge half is where the time went.
Agents are good at producing many small, dependent pull requests. A single refactor can come back as a stack dozens of PRs deep, with branches fanning out from the middle. Each PR is reviewable on its own, which is the point, but someone still has to land the whole thing on main in the right order without losing work, approvals or attribution. That someone was a human, merging by hand, and humans became the bottleneck.
Hand-merging is also where the quiet failures live. GitHub's view of a stack is eventually consistent, squash merges rewrite history under the PRs that depend on them, and every PR in a stack can pass CI while the combination does not. None of these fail loudly. They look like success.
How a Stack Lands
The merge side has two parts. A merge queue owns order: which ticket goes next, and whether it is still allowed to. Big Band owns the merge itself: rebasing, validating and landing the stack, then reporting what happened. They talk through tracker labels and one authenticated heartbeat, nothing else. The queue sends no refs and no job payload; Big Band rereads everything it needs from GitHub.
1. Getting into the queue
A ticket becomes eligible only after the automated review passes and a human approves the exact head that was independently reviewed. Approval moves the ticket to a queued state but never merges anything. Queue entries carry sparse ranks so they can be reordered, and no manual move can put a ticket ahead of something it depends on.
2. Dispatch rereads the world
The queue keeps a cache of GitHub and tracker data so the review UI is fast, but the cache is never allowed to authorise a merge. A dispatch round takes a lock, reads a bounded window of candidates, releases the lock while it fetches live approval and dependency state, then retakes the lock and rereads its own rows before claiming anything. A failed upstream read holds the candidate; it is never treated as an empty answer. An approval dismissed by a force-push after enqueue holds the entry rather than removing it. Only one entry is ever in flight.
3. The doorbell
The queue does not merge. It writes a namespaced command label on the ticket, and Big Band picks it up, marks the run as merging, and starts sending heartbeats. Labels are not transactional, so the queue revalidates the attempt after every external write, and the live label set, not a local field, is the source of idempotence. A stale worker cannot finalise a later attempt.
4. Walk and record the stack
Big Band derives the stack from live GitHub: topology, head SHAs, reviews and the base branch. Trunk branches always terminate a chain, so a release PR cannot accidentally adopt unrelated stacks. Every layer is written to durable state before a worker launches, which matters later: if a parent merges and GitHub retargets its child to main, the recorded layers still show what the stack was.
5. Validate the union, not the parts
Per-PR CI only sees one PR's diff, so a break that spans two PRs in different parts of the codebase is invisible until everything is in one tree. Before merging anything, Big Band merges every leaf into a single integration tree, runs the full unit suite and the generated-artifact freshness checks on it, and diffs the failures against the same suite on main to separate what the stack caused from what was already broken. This is the cheapest step in the pipeline and the one that catches the most.
The required-versus-advisory split comes from the repository's ruleset API, not from what the checks UI shows. Merging with a known-advisory check red is honest; merging with a required check red or unreported is refused.
6. Merge down, top first
Landing walks a strict reverse-topological order computed up front. Each PR merges into its parent's branch, and the bottom PR's final squash is the single commit that reaches main. Every approval in the chain is preserved and every PR is recorded as merged, rather than closed in favour of a fresh consolidation branch that would throw away all of its approvals.
Between steps there is no timer. After each merge Big Band polls until the PR reports merged and its parent reports a known mergeability, then asserts that the parent's diff against base grew by at least the child's file count. Only then does it take the next layer.
7. Audit and hand back
When the walk finishes, the host process (not the agent) verifies every layer on live GitHub: merged, with the expected head, base and merge commit. It then writes an outcome label for the queue, plus a comment saying how much landed and, if anything is left, what to fix and how to retry. A ticket moves to done only after GitHub reports the merge commit.
Where humans sit
Agents no longer merge to main themselves by default. Creating a merge request for the queue counts as the human review step, so an agent that was not explicitly told to merge asks first. After that, recovery from a moved base, a conflict or a failed attempt belongs to the queue and Big Band. A person is pulled back in only when the report says one is needed. The top-down procedure is also written down as a guide, so a human can run it by hand as an audited break-glass path when the queue is bypassed.
Triage: Red, Yellow, Green
Once agents can produce PRs faster than people can read them, the scarce resource is human attention. I wanted a signal on every open PR that answers one question: does this change a promise that something outside the change depends on?
Size turned out to be a bad proxy. The smallest contract break I measured in our repository was 14 lines, a renamed field on a payload published to a message stream. The largest safe change was 8,716 lines of brand-new package. So the classifier reads the diff instead.
| Tier | What a reviewer should do | What drives it |
|---|---|---|
| Red | Full review | A mechanical rule fired with a citable diff line, or the semantic read is confident the change alters a contract. |
| Yellow | Quick scan | The semantic read leans toward a contract change but stops short of the threshold, or thinks only supporting files changed. Yellow never gates a merge. |
| Green | Land as is | No rule fired and the semantic read does not escalate. |
Two halves that miss different things
The mechanical half is a set of pure functions over unified diff text. It flags removed public fields, removed routes, schemas switched to reject unknown fields, and migrations whose upgrade path drops or alters something. Every flag carries the verbatim diff line that triggered it, so a reviewer checks evidence instead of trusting a verdict. The rules got narrower each time a looser version measured wrong: a field removed inside a test is not a contract break, and a table dropped in a migration's downgrade never runs on a forward deploy.
The semantic half asks an LLM a handful of typed questions about the diff and gets probabilities back rather than prose: how likely it alters a contract, how likely it breaks a consumer, and which surface it touches. It exists for breaks that leave no syntactic marker, like a renamed stream topic, a changed cache-key format, or a symbol dropped from a module's exports.
Two properties held the design together. The semantic read is escalate-only: it can add a reviewer but never clear one, because only the mechanical half produces citations a person can verify. And it degrades to nothing: no key, a timeout, rate limiting or a malformed reply all fall back to the mechanical verdict. A triage signal that sits in front of a merge queue must not block it, or escalate everything, when a provider has a bad day.
Stacks complicate attribution
A stacked PR's diff against main includes everything below it, so a break in the bottom PR would paint the whole tower red. Verdicts are therefore computed against the parent's head and keyed on both the PR's head SHA and its parent's head SHA. A force-push to the parent changes what the child authored without moving the child's own head, and the key catches that. A verdict whose head has since moved is not served at all. In the queue view an entry shows the worst tier across its PRs, and the diff pane shows the evidence: the basis, each cited line, and the semantic read as percentages.
The triage stack was built, measured and shown working in the review UI, but it was parked before landing on main while other merge work took priority.
When Things Go Wrong
Most of the rules above exist because a hand-merge of a large agent-produced tower failed in a way that looked like success. The guide Big Band enforces is written as a list of those failures, each paired with the concrete incident that motivated it.
The silent stale squash
Merging a child updates its parent's branch, but GitHub takes a few seconds to recompute the parent's diff. Merge the next layer too early and you either get an honest "not mergeable" error, or, much worse, the merge succeeds against a stale head and squashes only that PR's own diff. In one tower that is exactly what happened: everything below one layer received 74 files instead of 491, with no error anywhere. The only signal was a file count someone had to check on purpose. That is why the pipeline polls for settled state and asserts content propagation after every merge.
Orphaned children
Squashing a parent rewrites history, so a child still carrying the parent's original commits conflicts. Top-down order avoids this, but any interruption reorders the work. The fix is to treat the order as a resumable queue rather than a loop: on interruption, recompute, retarget any child whose parent has merged onto the branch the parent merged into, and replay only the child's own commits.
Partly landed stacks
A run can die after merging the top layers into their parents but before main moves. Big Band now follows merged layers down to the first open PR and resumes from there. It refuses to resume from any shape that could land something twice: everything merged, the bottom merged, a merged layer below an open one, or a closed PR. Each previously merged layer must still check out on live GitHub before the new attempt trusts it.
Retry policy by failure kind
Not every failure deserves the same response, so retries are budgeted by kind.
| Failure | Response |
|---|---|
| Base branch moved | Retry without limit; does not count as an attempt |
| Test failure | Retry once |
| Real error | Up to three attempts within one run |
| Merge conflict | No automatic retry; back to a human |
| Big Band's own trouble | Park the ticket as unavailable; write no outcome |
Before a fresh attempt, the host rechecks its run fence and proves on GitHub that nothing landed. The last row is the one I like most. When the problem is Big Band's (a crashed worker, a stalled agent, a workspace error, a provider quota, GitHub being unreadable) the ticket keeps its merge command, gains an "unavailable" marker, and waits for someone to ring again. Infrastructure trouble is not a verdict on the ticket, so it should not be recorded as one. A layer that GitHub reports closed is a ticket problem, not infrastructure, and fails instead of parking.
Liveness
Because only one entry is ever in flight, a dead worker would stall the whole queue. Big Band sends a heartbeat every minute; once one has arrived, the queue's watchdog treats a beat older than three minutes as death and frees the slot. Before any heartbeat arrives, it falls back to a fifteen-minute no-progress timeout.
Fail toward caution
The same rule runs through every component: missing data holds work. A failed read is not an empty read, a cached hint never authorises a merge, safe mode suppresses automated writes while leaving human review alone, and a dashboard with nothing on it says "I may be broken" rather than looking idle.
Results
The clearest number is the one that started the merge work: an 80-PR tower, a 26-deep trunk with a 24-wide fan, merged to main by hand. It surfaced seven distinct failure modes, and recovering from each cost either a full re-derivation of the content path or a rebuild. All of it was recoverable only because no branch had been deleted, so keeping branches is now part of the procedure too.
The approval math alone justified merging down. A consolidation branch for one reviewed chain would have traded 28 approvals for 1 and lost the merged-status trail. Merging down through the chain kept all of them and still put a single commit on main.
For triage, I measured both halves over all 292 open PRs at the time, scored against 40 hand-adjudicated PRs containing 29 confirmed contract breaks, with every citation checked against the real git diff.
| Recall | False flags | PRs escalated | |
|---|---|---|---|
| Mechanical rules | 24 / 29 (83%) | 2 | 44 / 292 (15%) |
| Semantic read | 23 / 29 (79%) | 2 | — |
| Union (red tier) | 29 / 29 (100%) | 3 | 81 / 292 (28%) |
Neither half cleared 83% on its own; together they caught every confirmed break at the cost of one extra false flag. Even at its most cautious, the union leaves roughly seven in ten open PRs green. I also found that 13% of open PRs carried file changes they did not author because of stacking, which is why verdicts are keyed on the parent's head.
What I Learned
Encode the procedure you have already debugged
Our first plan for landing was ambitious: a native, transactional stack merge for routine 50- to 100-PR stacks, frozen candidates, one cumulative test run per stack, durable leases and a post-merge audit. We cancelled it. What shipped instead was the top-down procedure I had already been through by hand, with every failure turned into a check. The ambitious design wasn't wrong, but it was a design for problems we had imagined. The guide was built from problems we had actually had, and it was useful the day it was written.
Most merge failures look like success
The stale squash, the per-PR-green-but-union-broken stack and the consolidation branch that quietly drops approvals all report success. The defence is an assertion after every step that checks the outcome, not the exit code.
Keep the boundary narrow
Labels and a heartbeat feel primitive for connecting two services, but the narrow boundary is what made restarts safe. Neither side trusts the other's payload; both reread GitHub. Agents never hold tracker or merge credentials, and every state change happens in the host process, so a confused agent cannot land anything on its own.
Make the cautious signal cheap to check
Triage only helps if reviewers believe it. Citing the exact diff line behind every red, and letting the LLM escalate but never clear, turned an opaque score into something a person can dispute in a few seconds.