GTM Signal Agent
Synthetic benchmark Synthetic data: Ledgerly
Finds the accounts that need a seller’s attention, has Claude check the evidence and draft the next step, and puts every draft in a human review queue. Three eval suites score every prompt change.
- Status
- Write-up ready, code not public yet
- Relevant to
- AI and agents, GTM Systems
Results · Synthetic data: Ledgerly
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.of next-half-year churners flagged by the risk score at 30.8% precision, 3.3× the base rateSynthetic benchmark
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.of expansions surfaced as a seat conversation by the v2 score, up from 6.2% for the v1 ruleSynthetic benchmark
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.AUC for the churn risk score, with weights and cutoff set on earlier snapshots onlySynthetic benchmark
- Implementation factA fact about what was built and tested, not a business outcome.offline, including an end-to-end run against the real MCP serverImplementation fact
At a glance
- Question
- Which accounts need a seller’s attention now, and can an agent draft the next step safely?
- Data
- Ledgerly’s synthetic customers as of March 31, 2026. The agent sees nothing after that date; the following six months are the answer key.
- Compared with
- Single-rule signals such as usage drops or overdue invoices, and the v1 seats-near-cap rule for expansion.
- How it was tested
- Score the signal detector against what actually happened, then run three eval suites with confidence intervals and a held-out split.
- Result
- The risk score flagged 74.5% of churners at 30.8% precision. Expansion recall rose from 6.2% for the v1 rule to 57.8%.
- What it doesn’t show
- No live-model results yet: every agent score comes from an offline mock, so the evals test the harness and the detector, not Claude.
The problem
A follow-up agent is easy to demo and hard to trust. Without evals you can’t tell whether a prompt change made the drafts better or worse, or whether the agent obeys an instruction planted in a customer email.
What I built, in detail
- Point-in-time signals from one DuckDB query, plus churn and expansion scores whose per-driver contributions add up exactly to the score, so the agent can tell a seller why.
- A manual tool-use loop on the Messages API that must finish with a strict-schema action. Nothing is ever sent: drafts go to a review queue where a person approves, rejects or edits.
- Governed CRM mode: the agent reads the CRM over MCP, an approved draft becomes a pending change request, and a different person approves it in that CRM. The agent’s own approval is denied by the server.
- Core, adversarial and real-estate eval suites with 95% Wilson intervals, a held-out split that logs every look, a regression gate in CI and a prompt-to-eval diff with McNemar’s test.
How I checked it
- Every detector metric is scored against the hidden outcomes for the six months after the as-of date, which the agent never sees.
- Decisions made after seeing test-period results are disclosed, including a v2 expand play that doubled severe misses from 3 to 6 before I changed the policy on training data.
Limitations
- No live-model results yet. Every agent score comes from an offline rules-engine mock, which tests the harness and the detector, not Claude.
- Synthetic data from one company, and small eval suites: 40, 16 and 12 scenarios, so the intervals are wide on purpose.
Where it came from
My daily follow-up agent drafts check-ins for my clients but never sends. This is the B2B version, with the evals I wish I had for mine. Read the AI CRM case study.
Stack
TypeScript, Anthropic SDK, MCP SDK, DuckDB, Vitest, GitHub Actions