In May 2016 I wrote a short post called Bots Club. One idea, one line: bots should and can work collaboratively as we humans do. Bot to bot, no human in the channel. I built three Hubot demos for my team — bots processing a ticket, bots handling an alert, bots watching cost — and the point was that if bots could hand work to each other, we’d get our time back.
Ten years later the idea is no longer a demo-day curiosity. The reasoning, the tool plumbing and the memory all exist, and agents can genuinely research, correlate and critique each other’s work. So I want to take the idea one step further than I did in 2016, and talk about the part nobody was ready for back then: what has to be true before you let a club of bots near production.
The capability is no longer the blocker. Agents are good enough to be useful — not good enough to be trusted unsupervised, which is exactly why the controls are now the interesting problem, and the only thing standing between a clever demo and something you’d run on a Tuesday afternoon.
The four things that were missing
| 2016 gap | What fills it now |
|---|---|
| Regex intent matching — a bot only understands what you scripted | LLM reasoning — a bot can act sensibly on a message nobody wrote a branch for |
| Every integration hand-coded per bot | MCP — write one tool server, any agent can use it |
| No memory between messages; every bot woke up amnesiac | Persistent memory and learned corrections that survive restarts |
| Bot A blindly trusts Bot B’s output | An independent evaluator that grades another bot’s work before it ships |
The fourth row is the one I completely missed in 2016, and it turns out to be the load-bearing one.
What this looks like when it’s actually running
Every week a job produces the AI briefing on this blog. It is not one bot, and it is not two. It’s three parts, and the shape of the split is the whole point:
- A drafter agent researches the week, then writes the post and its source list to two files on disk. It holds no publishing tool whatsoever — not “instructed not to publish”, but configured without the capability. It could not publish if it decided to.
- An evaluator agent on a different model reads those two files cold, with no knowledge of how the draft was produced. It grades six content dimensions one to five with a floor of four, checks that every claim maps to a named source, and writes its verdict to a third file.
- The publisher is not an agent at all. It’s a deterministic script with no model in it. It reads the verdict and the draft, and publishes only if every dimension clears the floor, there are zero unsourced claims, the format checks pass, and the verdict post-dates the draft it’s judging. Anything missing, stale or unparseable and it saves as a draft and tells me.
That third part is the one I’d most want to explain to my 2016 self. It started life as a third agent, and I retired it within the same session. Deciding “does this verdict pass, yes or no” is an if-statement. A model asked to make that call is non-deterministic, can be talked into it by how the request is phrased, and costs real money per run. Don’t put a model where a conditional belongs.
The evaluator earns its keep, too. Reading the actual file rather than a summary, it caught an unsourced claim about a corporate merger that hadn’t happened, two outlets cited in the body but absent from the source list, and a date the sources didn’t support. The first draft scored three out of five on grounding. Two revision passes cleared it.
And one detail took me several failed attempts to get right: the evaluator has to read the artifact, not a description of it. Earlier versions handed it a condensed summary, and it correctly refused to grade — “I can’t verify what I can’t read” is the right answer from a reviewer, even though it looks like a failure. Passing files instead of prose is what turned the review from theatre into a real gate.
IMPORTANT: there is still a hole in this, and I’d rather name it than pretend otherwise. The drafter can write files anywhere, which means it could in principle write the verdict file itself and forge its own pass. The gate trusts a file the drafter can reach. Closing it properly needs path-scoped writes, or a gate that runs the evaluation itself instead of reading someone else’s homework. Knowing where your control is thin is the difference between a control and a decoration.
The controls you need before production
In 2016 the objection to Bots Club was always the same: “nice, but I’m not letting that touch anything.” That objection was correct, and the instinct at the time was to answer it with more approval steps. Approvals are the weakest available answer. Here’s what actually holds.
1. Start read-only. Make writes a separate project.
A read-only agent that is wrong produces a bad recommendation, which a human can still reject. A write-capable agent that is wrong has already changed something. Both are real risks; only one of them needs a rollback. That difference deserves different rollout timelines, and almost every genuinely useful thing a bot club can do for an operations team is a read — correlate, explain, attribute, summarise, rank. Ship that first and collect months of evidence before anyone writes a line of remediation code.
2. Capability asymmetry, not trust
When writes eventually arrive, split the job. The investigator holds read-only credentials and roams widely — metrics, config, change history, log patterns. The actor holds a narrow write allowlist plus read access to the specific resources it can write, because an agent that mutates state it cannot inspect is more dangerous, not less: it can’t check preconditions, can’t confirm the target matches what the finding described, and can’t verify its own change landed.
Be precise about what this buys you, because it’s easy to oversell. It does not make either agent harmless. The actor holds write permissions, so it is the dangerous one by definition, and an investigator with broad read over operational data is its own exposure. What the split actually gives you is two things, from two different mechanisms: the set of possible mutations is fixed by the allowlist rather than by a model’s judgment, and a conclusion has to survive a hand-off before it can become an action.
And the finding itself is untrusted input. An actor that executes whatever it’s handed is a confused deputy — an injection landing in the investigator reaches production through a perfectly compliant actor. The actor validates preconditions itself and refuses anything outside its allowlist, however confidently it was asked.
IMPORTANT: don’t ask an agent to behave. Constrain what it can reach. Prompts are guidance; permissions are enforcement.
3. Don’t put a model where a conditional belongs
Agents are for judgment under ambiguity. Gates are not ambiguous. “Every score is at least four and the format check passed” is an if-statement, and an if-statement is deterministic, free, auditable and impossible to talk out of its answer. The moment you implement that same gate as an agent, you’ve added a component that can be persuaded by framing and that will occasionally decide differently on identical input.
Look at any bot club you’re designing and find the decisions that are actually rules. Move those out of the models.
4. Per-agent identity and short-lived, scoped credentials
One shared service account across the club destroys your audit trail — you can prove “a bot did this” and nothing more. Give every agent its own identity, its own role and its own scope, so the log answers which agent did what, under whose delegated authority, with credentials that expire. This is unglamorous IAM work and it’s the difference between an auditable system and a shrug.
5. A data classification boundary at the edge of the club
Decide explicitly what data is allowed into agent context, and enforce it before the model sees anything. Metrics, infrastructure state, deployment history and log patterns are usually fine — but check rather than assume, because config files carry secrets and metric dimensions carry customer identifiers more often than people expect. Customer records, credentials and raw log bodies are out. Redact at the tool boundary, not in the prompt: a rule the model is asked to follow is a rule that will eventually be broken.
6. Treat every input as untrusted
This one has no 2016 equivalent and it catches people out. A ticket description, an alert annotation, a log line, a web page an agent fetched — all of it is data, not instructions. Anyone who can file a ticket can write “ignore your previous instructions” into it. If your triage bot reads that ticket and your actor bot trusts your triage bot, you’ve handed a stranger a path into your automation. State the boundary explicitly in every agent, and keep the read side and the act side separated so a successful injection still can’t reach a write.
7. An adversary inside the club
Here’s the failure mode people underestimate: bots agreeing with each other, confidently and wrongly. One agent proposes something plausible, the next accepts the premise and builds on it, and four hops later you have a beautifully argued conclusion resting on an error nobody checked. A chain with no adversary in it isn’t collaboration — it’s a longer, more expensive path to the same mistake, with more confidence attached.
So put a dissenter in the room, and make it structurally different: different model, different information, different tools, different incentive. Two agents sharing a model and a context share their blind spots too, so a second pass buys you far less than it appears to — you’ll catch sloppiness and miss the errors that come from the way the model reasons.
And be honest about the limit: removing a reviewer’s ability to act bounds what its mistakes can do directly, but its worst failure is waving through something it should have caught. A lenient reviewer is more damaging than a harsh one, and no permission model fixes that. That’s what the rubric and the fixture set are for.
8. Grounding rules and an honest “I don’t know”
Every claim in an agent’s output should be traceable to a tool result, with the query that produced it. And the club needs a legitimate way to return insufficient evidence — because if the only accepted output is a confident answer, you’ve trained the system to invent one. An analysis bot that says “I could not determine the cause; here are the three things I checked and the two I couldn’t access” is more valuable than one that always names a culprit.
9. Bounded loops, budgets and a kill switch
Two bots that can each trigger the other will, eventually. Cap iterations, cap tool calls, cap spend per run, set a wall-clock timeout, and make sure one person can stop the whole club without a deploy. A runaway agent loop is the modern equivalent of a cron job that spawns cron jobs, except it also bills you per token.
10. Replayable transcripts
Every message between agents, every tool call, every input and output, stored and readable after the fact. Not for compliance theatre — for debugging. When the club reaches a wrong conclusion, “which step went wrong” is only answerable if the whole exchange survived. This is also the artefact that earns you permission to expand scope, because you can show your work.
11. Evaluation as a regression suite
Keep a set of past cases with known answers and re-run the club against them whenever you change a prompt, a model or a tool. Non-deterministic systems drift, and without a fixture set you’ll find out from a bad decision instead of a failing test.
Where to actually start: read-only, non-sensitive, high-value
These are the use cases I’d put a bot club on first. All of them are read-only against non-sensitive operational data, all of them end in a recommendation for a human, and all of them are worth real money on day one.
| Use case | The club | Reads | Output |
|---|---|---|---|
| Bug triage | Classifier, Code Locator, Duplicate Hunter, Skeptic | Ticket text, repo, recent commits, existing tickets | Draft comment: severity, likely component, suspected duplicates, confidence, dissent |
| Alert analysis | Correlator, Change Reviewer, Skeptic | Alarm history, metrics, deployment timeline, log patterns | Narrative attached to the alert: what fired, what changed, ranked candidate causes |
| Cost anomaly detection | Detector, Attributor, Risk Assessor | Cost and usage data, tags, resource inventory | Ranked anomalies with the resource and the change that likely caused them |
| Root cause analysis | Timeline Builder, Hypothesis Generator, Skeptic, Foreman | Metrics, logs, config history, change records | Draft RCA timeline with evidence per claim and gaps named explicitly |
Notice the shape they share. In every one, the club’s job is to compress a wide, tedious read into a short, argued brief — and the human still makes the call. Nothing is mutated. Nothing sensitive is read.
The downside of being wrong is a misleading brief a human can catch, rather than a change you have to roll back. That’s a genuine reduction in severity, not an absence of it: a confident, wrong root cause can still send an incident response down the wrong path, and a wrong duplicate call can close a real bug. Which is why the output has to show its evidence and not just its conclusion — a brief you can audit in a minute is the difference between a useful club and a plausible one.
It’s not a small ambition, either. Correlating an alert against a deployment timeline and a fortnight of metrics is exactly the work that eats an on-call engineer’s evening, and it parallelises beautifully across specialists.
I already have the simplest version of the cost case running: a daily job that reports what my AWS account spent yesterday and where. It’s useful, but it’s a report — I read it, I nod, I close it. Adding a second bot that attributes each movement to a cause, and a third that vetoes the reckless suggestions, is what turns a report into a decision queue. Same data, same read-only permissions, dramatically more value.
Graduating to writes
Eventually you’ll want the club to act. The path there isn’t a leap, it’s a ladder, and each rung should be earned with evidence from the rung below:
- Read and report. Output goes to a human. Months, not weeks.
- Write to itself. Comments, labels, draft documents, ticket annotations — reversible and non-operational, though worth remembering that auto-labelling can still misroute a real issue and comments are often customer-visible.
- Propose an executable change. The club produces the exact command or pull request but cannot run it. A human clicks.
- Act inside a tiny allowlist. One or two specific, reversible, well-understood operations, on non-production first, with the investigator/actor split enforced and every action logged.
IMPORTANT: most teams should stop at rung three for a long time, and that’s not timidity. Rung three already captures most of the value, because the expensive part of operations work is the diagnosis, not the keystroke.
One clause I’d add to the 2016 post
I stand by what I wrote ten years ago. Bots should work collaboratively as we humans do. What I’d add now, having watched it actually run, is the two parts of human collaboration we tend to leave out of the picture: we argue with each other, and none of us has the keys to everything.
The bots can talk to each other now. Whether you let them into production comes down to whether you built the controls first.
The Homer Simpson ending still stands, though. That bit I got right.
Jackie Chen's IT Workshop 