Last updated: May 1, 2026
Teaching AI to Read the Room: How We Built an Automated Incident Classifier
Written by Daniel Anderson · 10 minutes read
After 18 months and ~800 incidents, our Rootly classification fields were mostly empty, no one fills in six drop-downs at 2am. This post walks through how Tilt built an automated incident classifier: a scheduled .NET pipeline that scrapes the Slack channel, sends the conversation to Claude, and writes back the cause, vendor, component, and team, while quietly backing off any field a human corrected.
At Tilt, we use Rootly (our incident management platform) and Slack for handling production incidents. Something breaks, a Slack channel gets created, engineers jump in, and the conversation plays out. Diagnosis, mitigation, resolution. The usual.
What doesn’t happen reliably is the part that follows: classifying the incident. What was the cause? Did the error come from our systems or from an external integration? Which team owns the fix?
No one wants to bother with collating incident data at 2am after dealing with a complicated production issue. But that data is critical for understanding trends in system reliability.
So we engineered a solution using AI.
🔍 The Problem: Incomplete Data, Incomplete Picture
Let me paint the picture. We use Rootly to manage incident data. When you create an incident, there are a slew of fields you can populate: cause, forced/unforced, owning team, vendor, component, and more. After about a year and a half we’d gathered roughly 800 incidents. On review, the vast majority were barely classified. The old field system wasn’t helping either. Causes had cryptic prefixes, there was no clean way to distinguish “which vendor caused this” from “which internal system broke”, and half the fields were just empty.
And honestly, you can’t blame anyone. You’ve just spent hours on a production incident, it’s late, you’re tired, and now you’re expected to fill in six confusing drop menus worth of information before you can close the tab? On top of all that, there’s no forcing function and certainly no reward for following through. The missing value isn’t recognised until weeks or even months later when someone asks a question no one can answer.
Our engineering leadership kept asking the platform team questions like “which vendors are causing us the most incidents?” and the honest answer was… we don’t really know.
That needed to change.
Why bother classifying deeply?
If you don’t have the data, you can’t hope to see the bigger picture of what’s really going wrong in your systems. Without it, you’re going off gut feel and whatever’s fresh in memory, which will inevitably fall into the recency bias trap and completely undermine any long-term effort to improve.
With proper classification you have the ability to:
- See patterns over time, not just react to what’s on fire right now
- Figure out where the pain is coming from, not just where it hurts today
- Make real decisions about where to spend engineering effort next quarter
- Track whether the work you have done is actually reducing incidents in the areas you targeted
This may feel like busywork, but it’s foundational. It builds a feedback loop between incidents and engineering investment so you can fix the most valuable things first.
💬 The Approach: Read the Slack Conversation
Before building anything, we needed to figure out our source of truth. Every incident at Tilt plays out in a Slack channel. It’s the centralised, complete context of the incident: engineers discussing the problem, sharing graphs, posting logs, working through root cause in real time. By the time an incident is resolved, that channel is basically the full story of what happened and why. The data was already there, written for us.
So the idea was straightforward: scrape the Slack conversation, send it to Claude (via AuggieCLI, our AI agent command-line tool), and have it classify the incident into a clean set of fields.
📋 Designed with zero overlap - each field answers exactly one question
Here’s what we landed on:
| Field | What it answers | Examples |
|---|---|---|
| Cause | Why did it happen? | Infrastructure Failure, Code Bug, Bad Deployment |
| Sub-cause | What specifically? | Failed Server, Query Timeouts, Certificate Expiry |
| Vendor | External party involved? | Cloud providers, payment processors, etc. |
| Component | Internal system affected? | API, Redis, database, etc. |
| Domain | Business area? | Platform, Credit, Cash Advance |
| Team | Who owns it? | The team responsible for the affected system |
| Culpability | Self-inflicted or external? | Forced (external) / Unforced (we caused it) |
| Summary | What happened? | Root cause, impact, and resolution |
This breakdown took a few iterations to get right. We originally had Vendor and Component as a single field, but quickly realised they needed to be separate. Some incidents involve both - a cloud provider goes down and takes one of our internal services with it. You want to be able to ask “which vendors cause us the most pain?” and “which internal systems break the most?” independently.
We also separated Domain from Team. Teams restructure, people move around, names change. But business domains are stable. Splitting them means you can track reliability trends across domains without the data going stale every time a reorg happens.
🏗️ The Architecture: A Console App in a Pipeline
Once we had our classification model, we moved to building the system. At Tilt, we like keeping things simple, so we settled on a .NET console app running as a scheduled Azure DevOps pipeline - two familiar building blocks we were already using. Here’s the flow:
The entire system boils down to four tasks:
- Channel Scrape - we find all the incident channels in Slack (including archived ones), temporarily unarchive them if needed, and extract all the message content
- AI Analysis - the messages are sent to Claude with a structured prompt that includes the allowed field values, cause-to-subcause mappings, and rules for tricky edge cases
- Rootly Update - the AI’s classifications are written back to Rootly
- Stale Incident Ping - a daily nudge for incidents that have been sitting open for too long
Since this is just a pipeline, we were able to operate it in two modes. In Backfill mode, we chewed through all 785 historical incidents from the beginning of time, working through them in chunks. Then in Reconcile mode, the pipeline runs every morning and picks up newly resolved incidents as they come in.
⚡ Avoiding Reclassified Incidents
Revisiting already-processed incidents would be a pretty big waste of time, so we built in a simple checkpoint mechanism. Each incident gets a timestamp saved after processing - the last Slack message we saw. On the next run, we check with Slack: “anything new since this timestamp?” If the answer is no, we skip it entirely. No AI call, no API writes, nothing wasted. This means a daily run scanning 800 channels but only finding 3 new incidents completes in minutes.
🧠 Lessons from the AI
🎯 Getting the AI to Pick the Right Answer
Our first attempt was basically “here are the messages, classify this incident.” That didn’t go well. The AI kept making up field values, rephrasing our options into something that sounded similar but didn’t actually exist in Rootly, or picking causes that weren’t real.
The fix was putting the allowed values right next to the final instruction, not at the start of the conversation. When you send 200 Slack messages in batches, the context from the beginning gets pushed far back in the window. The AI loses track of what the options were. Moving the field definitions to the end, right before “now classify this”, made a significant difference.
The other thing that helped was adding descriptions to sub-causes. Without them, the AI was just guessing from the name. “Maintenance” could mean “we did scheduled maintenance and it broke something” or “the server needed maintenance because it was dying.” One line of description per sub-cause cleared up most of the ambiguity.
🤝 Respecting Human Corrections
This was important to get right. If somebody fixes something the AI got wrong, that fix needs to stick. No one wants to come back the next day and find the AI overwrote their correction.
The way it works: the system saves which fields it wrote and what values it set. Next run, it checks what’s currently in Rootly. If the value changed since the AI last wrote it, that means a human deliberately changed it. The AI backs off.
| Scenario | What happens |
|---|---|
| Field is empty | AI fills it in |
| Human set it, AI has no record | AI overwrites (humans get initial classification wrong too) |
| AI set it, value unchanged | AI can refine with new analysis |
| AI set it, human changed it | AI skips. Correction sticks. |
So it’s self-correcting. Run it, spot-check, fix what’s wrong, and the AI learns to leave those alone going forward.
📢 The Friendly Nudge
Here’s the catch with AI classification: it only works on resolved incidents. If the conversation is still going, the root cause might not be known yet. The resolution isn’t there. You’d get a half-baked classification.
So what about incidents that sit in “mitigated” or “started” for weeks? That data just doesn’t exist.
Our fix: if an incident has been open for more than 14 days, we post a daily @here in the Slack channel. “This incident has been open for 32 days. Is it possible to close it?” Every day. Until someone resolves it or cancels it.
It’s a bit annoying by design. And since it runs under my account, people probably think I’m personally nagging them every morning. I’m not. But I’m also not sorry about it 😛
📊 The Results
We backfilled all 785 historical incidents across 15 batches, scaling up the batch size as we got more confident in the output. The daily reconcile pipeline has been running since, picking up new resolved incidents automatically.
For the first time, we can actually answer questions about our incidents with data. Which vendors cause us the most problems. Which internal systems break the most. Whether our incidents are mostly self-inflicted or things outside our control.
☕ The system processes about 15-20 incidents per run, takes around 10 minutes, and costs roughly the price of a coffee in AI API calls.
Not bad for replacing a bunch of manual work that wasn’t happening anyway.
🔄 What We’d Do Differently
- Start with the field taxonomy. We changed the fields several times during development. Getting the classification model nailed down first would have saved a lot of prompt engineering back and forth.
- Test with real incidents early. Fake test data doesn’t surface the weird edge cases. First time we ran it against a real 200-message incident channel, we found three bugs in the ownership logic.
- Don’t fight the Slack API. Rate limits on archiving and unarchiving are aggressive. We added retry logic, but honestly the real fix was just slowing down and being patient.
Interested in how we built this, or thinking about doing something similar? Happy to chat about AI-powered incident classification, prompt engineering for structured output, or building systems that know when to get out of the way.
About Daniel Anderson
Experienced Platform Engineer at Tilt with 12+ years building backend systems, real-time streaming infrastructure, and cloud-native applications. Previously Staff Software Engineer at Dolby Laboratories, working on streaming APIs, transcoder development, and real-time delivery using WebRTC, RTMP, and SRT protocols. Has a passion for AI-driven development workflows, scalable distributed systems, and solving complex infrastructure problems.
