Six days from pitch to an AI on call

I was hired as a UX Analyst. Nothing in the team or the company investigated an alert on its own, so overnight outages waited hours for a person. On 16 June I pitched a triage playbook; on 22 June my pipeline posted its first full verdict. I built and ran it alone until the backend and AI teams moved it onto the company's monitoring platform, where it grew to cover 939 alert types.

Role
UX Analyst. Designed how AI verdicts are written, trusted and delivered
Team
Solo MVP; backend and AI teams on the platform
Company
Multi-market digital platform
Timeline
June - October 2026
Scale
939 alert recipes, 21 market groups
Sentinel pipeline: PostHog alert, CDP webhook, Sentinel worker, AI triage and Slack verdict, with the evidence the worker gathered for one deposit alert

At a glance

The story in 60 seconds

The 2:46 AM alert, the relay I drew and built in six days, the first full verdict, and the growth to 939 recipes.

My role
UX Analyst. I designed how the AI's verdicts are written, checked and delivered, and built its first version alone.
What I was responsible for
The triage method, the first pipeline, the platform requirements, and from 21 July the quality of the AI's analysis. The platform itself belonged to the backend and AI teams; Panther was co-built with my lead and the Head of UX/DX; my part was the design of its review loop and the quality of the verdicts it reads.
What zero looked like
Alerts fired into Slack at any hour. No automated investigation existed in the team or the company. An analyst opened Claude Code (an AI assistant), ran a prompt from memory and pasted a verdict by hand, if anyone got to it.
What was at stake
Deposit, withdrawal and login failures across many markets. An alert nobody reads at 2 AM is an outage nobody handles until morning.
The key decision
Replace the coded rules engine I had already built and tested with an AI agent working from my playbook, because the coded rules broke on the first real shift in normal levels and a model reading the same evidence could adapt. To keep that safe, the AI could only read data; a plain script I could audit did all the posting.
What shipped
A triage playbook and a working AI pipeline that I ran myself for three weeks. Then the method and 939 recipes on the Sentinel platform, an audit catalogue of 47 automated checks, and the verdict-quality loop inside Panther, the escalation agent.

Impact

  • 6 daysPitch to first full verdict, soloPlaybook pitched to the AGM on 16 June. First full verdict on 22 June, same conclusion as the AGM's manual triage
  • 59 of 66Live alerts diagnosed by my MVPPipeline log, 16 June - 9 July, before any platform existed. The rest were flagged to a person for manual review
  • 1 → 939Triage recipesOne deposit alert on 16 June; eight journeys and 21 market groups on 11 September

Dated snapshots from logs and exports, not ongoing rates. Accuracy against a labelled sample is the next thing to measure. See Evidence.

The MVP

The signal arrived. The answer took a person.

I had joined as a UX Analyst: funnels, dashboards, insights. Nobody in the team or the company had built a system that investigates an alert on its own, and I had never deployed a webhook (an automatic message from one tool to another) service, written a scheduler or run an unattended AI agent. I had the queries and the judgement. I started from those and learned each missing piece when it blocked me.

This is where it all started. On 29 May my section lead credited me, with the CTO, COO and AGM in copy, for rebuilding our weekly report workflow with Claude: about 90 minutes every week down to 15-20 minutes end to end, with a human review step kept in. Nine minutes later a second AGM asked to tap me for AI learnings sharing.

Where it started, 29 May

The weekly report shout-out, and a second AGM's reply under it.

Slack post from the UX Lab section lead crediting Robert for rebuilding the weekly report workflow with Claude and Slack MCP, from about 90 minutes a week to 15-20 minutes, copied to the CTO, COO and AGM

The shout-out. 90 minutes a week down to 15-20, with a human approval step.

Slack reply from an AGM: let's tap Robert for AI learnings sharing

Nine minutes later. "let's tap @Robert (UX Lab) for ai learnings sharing"

That trust bought me a hearing for a bigger problem: alerts that waited hours for a person. On 16 June I brought the AGM a fix. I had turned my manual triage habit into a playbook: five fixed queries, a decision matrix, four test cases, read-only, nothing posted by the tool. The AGM read it and said: switch to auto, now.

The pitch, 16 June

My message to the AGM and two leads: a playbook tested on four real cases, read-only, never auto-posting. Project and market codes removed.

Slack message from Robert on 16 June proposing a first version of the alert triage playbook, tested on four real cases, read-only
16 June · The pitch
A read-only playbook, tested on four real cases. The AGM: switch to auto, now.
17-21 June · The relay, built alone
A Cloudflare Worker mailbox, a poller and a headless agent (an AI run with no screen and no person), each the first I had built. Every component up; every live alert silently rejected.
22 June · First full verdict
Replayed on the 02:46 alert after the fix. Same conclusion as the AGM's manual triage.

One alert, one inspectable verdict

Pick a date, then select a stage to see which part of the reply it produced. Values from real verdicts; provider and market codes removed.

PostHog alert

Deposit funnel success rate below 44%

PH market · hourly check · previous hour below the lower threshold

Previous hour12.3%
Lower threshold44%
Alert triageThread reply
1Verdict
Transient, recovered (system) · PH market · 1:00-2:00
Payment routing and failover failure. The primary provider went dark for the hour and failover providers could not cope; recovered by 2:00.
2Bottom line
Success per click 13.2% (the alert's 12.3% is the funnel rate) against a ~55% same-hour, seven-day baseline, more than 10 standard errors below.
What broke
  • Submit to success 27.8% (baseline ~78%), the dominant leak
  • Clicks per user 2.74 (baseline ~1.5): heavy retrying
  • Primary provider: 1,819 orders to 0; three failover providers at 13-30% success
3Scope
This market only. Sibling markets normal in the same hour (47.5%, 40.8%). Not platform-wide.
Confidence
High · posted automatically in the alert thread
4Owner · action
Payments and platform routing. Post-incident ticket: why the primary provider stopped, and why failover ran at 13-30%.

Auto-generated triage (testing). Verify before actioning.

  1. 1The conclusion comes first, as a named incident class from the playbook.
  2. 2Evidence is measured against a baseline, not just restated from the alert.
  3. 3Scope rules out a platform-wide failure before anyone is paged.
  4. 4The reply ends with who acts next and what to ask.

The first full verdict, 22 June 2026, replayed on the real alert after a fix. Wording lightly condensed.

My first pipeline had no model in it. A Cloudflare Worker (a small cloud script) caught the alert from PostHog, the analytics tool, ran four SQL queries and walked a coded three-rung decision tree. It had 70 tests. It broke on the first real baseline it met.

Decision
An agent working from a playbook.
Rejected
The coded rules engine I had already built and tested.
Why
The matrix was right; the arithmetic around it was brittle. A model reading the same evidence could absorb a shifted baseline without a code change.
What it cost
The certainty of code I could test line by line.
Decision
The model never touches Slack.
Rejected
Letting the agent post its own verdicts.
Why
Unattended runs stalled on permission prompts, so the model ran with permissions bypassed. That made one line non-negotiable: it could read analytics and nothing else. A plain poller script I could audit did the posting.
What it cost
A more capable agent. Its only write action was mine.

The relay, as I drew it in June

My first architecture sketch. A Cloudflare Worker catches the alert, a poller on my Mac checks every 60 seconds, and a headless agent runs the playbook with read-only analytics access. High confidence posts to the alert thread; medium or low becomes a draft for a person. A run that fails twice is posted as a manual-review notice, never dropped, and work that waits over 15 minutes pages me.

Robert's first architecture flowchart: PostHog alert fires, Catcher mailbox on a Cloudflare Worker, Scheduler poller every 60 seconds, the Agent with read-only PostHog access, a confidence gate, then Messenger auto-post, draft for human, or needs manual review; a watchdog branch after 15 minutes

On 22 June I wrote in my own review: "the agentic path has never once worked end-to-end." For five days every component had been up, and every live alert had been rejected.

Five silent days, 17-22 June

Every component was up. The webhook body was missing one field, and no error ever reached Slack.

  1. Each live fire
    1. PostHog sends the webhook
    2. Worker rejects it400: the body omits the required alert ID
    3. Mailbox stays emptyThe poller runs with nothing to pick up
    4. No verdictAnd no error
  2. Checks added after
    1. Staleness watchdog: work that arrived but got no answer
    2. Transport check: "1 of 6 PostHog fires never reached the triage inbox in the last hour."

The trace, and the check that came out of it

Left, my 22 June fix review: every fire confirmed, every fire rejected. Right, the transport check still on watch in September.

Fix decisions review from 22 June: the alerts fired, the verdicts never came because the Worker rejected every fire; a boundary trace from PostHog to Slack

22 June. "The alerts fired. The verdicts never came."

Pipeline health watchdog message on 2 September: 1 of 6 PostHog fires never reached Sentinel in the last hour, owner Robert

2 September. The watchdog catches a lost fire, owner Robert.

I fixed it and replayed the 2:46 AM alert. The verdict posted at 8:50: provider outage, failover overwhelmed, one market, already recovered. The AGM had triaged the same incident by hand at 8:02. Same conclusion.

02:46 · The alert fires
Success 12.3% against a 44% threshold. Live fires are still being rejected.
08:02 · A person investigates
The AGM's manual triage, written without the agent.
08:50 · The same conclusion
The AI verdict, replayed after the fix, with baseline, scope, owner and action attached.

Two investigations, one answer

22 June. Left, the AGM's manual triage. Right, the replayed verdict. Both found the same provider outage, scoped to one market and already recovered.

The AGM's manual triage thread at 8:02 AM on 22 June: verdict REAL, transient, already recovered

08:02, by hand. "verdict: REAL, transient, already recovered"

The replayed verdict at 8:50 AM on 22 June: transient and recovered system failure, a payment provider went dark and failover providers could not cope, confidence High

08:50, replayed. Provider dark, failover overwhelmed, one market, confidence High.

Three days later the AGM wrote: "the session replay didn't work - need to check again, AI is easy mistake this." The relay had produced a bad replay link. Therefore the evidence rules got stricter, not looser: no link the agent had not verified. "Too much text" was the other half of the same complaint, so the verdict was cut from 2,028 characters to 925.

The first break

25 June. The AGM flags a broken replay link; the no-unverified-link rule follows in the same thread.

Slack thread where the AGM reports that the session replay link in an AI verdict did not work

In three weeks the relay answered 66 unique alerts with 59 verdicts and 8 manual-review notices, and 15 times the model refused to invent a verdict without evidence. It worked. But it only worked while my laptop was open.

Running is not the same as working.

Handoff and scale

The MVP outgrew my laptop, so I gave it away.

Handing it over, on paper

The PRD I wrote as requester, the scope the backend team built, and the ownership table that kept the analysis with me.

PRD for Alert Triage Automation, webhook-first architecture: target go-live 7 July 2026, owner the backend team, requester Robert (UX Lab), status released; the problem section names Robert's poller on his Mac
PRD new flow and Scope V1: PostHog alert, webhook, analysis engine with runbook and Claude API, decision gate, then seven items the backend team builds
Ownership table: Robert owns Sentinel recipes and prompt standard; the backend owns the Sentinel service

On 25 June the AGM opened a thread on automating investigation for every alert. It ran to 56 replies and settled where the MVP would live next.

Decision
Move the relay into the monitoring platform the backend lead's team was already building, renamed Sentinel that week.
Rejected
Spending July making my poller survive sleep, restarts and my holidays.
Why
Their platform brought a database, a UI and many teams at once, and I stopped being the single point of failure.
What it cost
The pipeline became theirs. The method stayed mine: I wrote the PRD as requester, and from 21 July I owned the quality of the AI's analysis.

On 2 July the company's training team ran a knowledge-sharing session on AI adoption. I presented the triage agent as one of four featured use cases, and the internal "Effective Claude AI Use Cases" page lists it as "Auto Triaging Deposit Alerts with an AI Agent".

Company-wide, 2 July

The training team's thanks to the five speakers, 26 reactions, and the internal page that lists the use case alongside three other teams'.

Slack kudos post thanking the five speakers of the AI Adoption and Effective Use Cases session, Robert among them, with 26 reactions

The session. Five speakers, 26 reactions.

Internal Effective Claude AI Use Cases page listing four featured use cases, including Auto Triaging Deposit Alerts with an AI Agent by Robert, UX Lab

The listing. "Auto Triaging Deposit Alerts with an AI Agent (By Robert - UX Lab)"

Go-live was 9 July: 48 alerts across deposit, gameplay and registration, one webhook, no laptop. The first card landed at 00:11, Real/High (a real incident, high confidence): a payment provider in a Bangladesh market succeeded on 1 of 54 orders. I confirmed it by hand that afternoon.

The first autonomous card

9 July, 00:11. Real/High on a payment provider failing 53 of 54 orders.

The first automated Sentinel card in Slack, a Real and High verdict on a payment provider failure

Sentinel, five stages

The production shape after handoff: PostHog alert, webhook transport, the Sentinel worker, AI triage, Slack verdict. The same read-only agent and confidence gate from my June sketch, now with a database and a UI behind it. Market and provider names replaced.

Five-stage architecture diagram: Source PostHog alert, Transport webhook, Platform Sentinel worker, Analysis AI triage, Output Slack verdict, with a sample verdict card

From one alert to 939 recipes

  1. 558 JulyThe handbook, before go-live
  2. 995 AugustFive journeys baselined
  3. 48226 August240 promotion recipes added
  4. 93911 SeptemberEight journeys, 21 market groups

Recipes and configured alerts are different units. These figures describe coverage, not a count of successful diagnoses.

Decision
Govern the recipes with automated checks.
Rejected
Review by eye, recipe by recipe.
Why
Nobody can read 939 prompts, and a batch update spreads a bad instruction as fast as a good one. I built an audit catalogue (47 checks by September), a batch updater that rewrote all 939 recipes with no format loss, and a revert path.
What it cost
Tooling time instead of new coverage, and the audit checks what a recipe says, not whether it is right.

One batch update, checked by machine

Outdated prompts 876 → 0; recipes that reason badly 726 → 8. The 9 September audit, over the 896 recipes live that day.

Audit tables: prompt-version-behind findings went from 876 to 0; Tier 2 recipes that reason badly went from 726 to 8

On 27 July the daily scan went live: one AI report a day on every journey's funnels, errors and alerts. That same day, a backend engineer asked for the pattern on their own team's system: "This is an example that we need to setup onto our turbine." Exactly one month later, on 27 August, the scan reached full output, and the AGM's reply was three words: "finally it works".

The daily scan lands

Day one, another team asks for it. One month later, the AGM's verdict.

Slack reply from a backend engineer: This is an example that we need to setup onto our turbine
Slack reply from the AGM: finally it works

One of the heaviest alerts users, globally

PostHog's team, 10 September.

Slack message from PostHog's team: you guys are one of our heaviest alerts users globally, so we're going to pay some special attention for your alerts

Still evolving

Each gap we find becomes the next version.

Scale showed what one laptop never could. In week one we had limited auto-send to High-confidence verdicts to keep the channel quiet. But the verdict format rates an incident that is still happening as Medium, because it may change. A delivery sample I ran on 24 September showed that rule was holding back most real-incident verdicts.

Decision
Panther, a second agent that reads each verdict and decides RAISE (send to the team) or HOLD (keep it back), based on impact and ownership, with three named human approvers.
Rejected
Also sending Medium verdicts.
Why
A looser gate brings back the noise that led to the High-only rule in July. Escalation became an explicit, inspectable call instead of a confidence flag.
What it cost
A second agent to keep honest, and people between a verdict and operations.

Panther was co-built with my lead and the Head of UX/DX; my part was the loop's structure and the quality of the verdicts it reads. In its first full week the team raised 11 incidents, up from 6, and noise reaching the alert channel fell from 20% to 14%, according to Panther's weekly report, reviewed by the team.

Panther, RAISE or HOLD

The escalation workflow: impact and ownership checks, three human approvers, and an ops report on raise. Weekly figures are from Panther's report, reviewed by the team.

Notion page describing the Panther RAISE or HOLD escalation workflow

My section lead's review asked the question that matters most: "Has it caught a real incident?" It has, several times, including a two-week SMS failure the client later confirmed. The next version measures whether a person saw each verdict in time, one row per alert, written from the card's buttons.

Evidence

Dated snapshots, not a scoreboard.

Every number on this page comes from a log, an export or a report I can point to. They describe different things on different dates, so they stay separate.

SourceSampleDateResult
Pitch to live1 playbook, 4 test cases16-22 JunPitched 16 June, read-only, tested on four real cases. First full verdict 22 June, same conclusion as the AGM's manual triage.
Relay log66 unique alerts16 Jun - 9 Jul67 outputs: 59 verdicts and 8 manual-review notices. 14 auth failures, 15 refusals to verdict without evidence.
First full verdict1 alert22 JunSame conclusion as the AGM's independent manual triage (08:50 against 08:02).
Recognition1 session2 JulCompany knowledge-sharing session, four featured use cases. Listed on the internal Effective Claude AI Use Cases page.
Recipe audit896 recipes9 SepRecipes on an outdated prompt version: 876 to 0.
OTP outage1 incident18 SepFirst alert 02:02, first verdict 02:08. The manual scan raised it three days later.
Delivery sample200 events24 SepOf 156 real-incident verdicts, 128 were held back by the High-only rule. Led to Panther.
Panther weekly report1 weekLate Sep11 incidents raised (from 6); noise in the alert channel 14% (from 20%). Report reviewed by the team.

What these results don't claim. Not measured: verdict accuracy against ground truth, time saved per alert, verdict-to-action time, incidents with no alert at all (for example a 12 August backlog of 131 open orders where nothing fired), or money and members protected. Recipe and alert counts are different units.

Reflections

What I learned, and what's next.

  1. Start small, read-only, and earn the right to automate.The first playbook posted nothing and was simple enough to check. The AGM read it and said: switch to auto, now.

  2. No map is not a blocker.I was hired to analyse product data. There was no triage system to copy, and I had never deployed a worker or run an unattended agent. Starting from the queries I knew, and learning each missing piece the day it blocked me, got from pitch to live in six days.

  3. Hand off the pipeline, keep the method.Giving the relay to the platform team ended my laptop as a single point of failure and freed me to own verdict quality across 939 recipes.

  4. Measure delivery, not just output.A verdict counts when a person reads it in time. Sampling delivery found the held-back verdicts and led to Panther.

What I'd do differently

Sample delivery and build a labelled set of right and wrong verdicts in the first month, before scaling, so quality grows with coverage.

What's next

First results from the seen-in-time log, and a labelled sample so accuracy can finally be measured.

In June I thought the product was the verdict. By September it was clear the product is the moment a person reads it.

Next case studyAdaptive Coder: Two jobs, one screen