Churn Early Warning
Synthetic benchmark Synthetic data: Ledgerly
Flags the customers most likely to cancel in the next 90 days, explains each flag in plain English with the numbers behind it, and proves it with an out-of-time back-test and a second company the model never saw.
- Status
- Write-up ready, code not public yet
- Relevant to
- Data and analytics, AI and agents
Results · Synthetic data: Ledgerly
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.AUC for the shipped model, against 0.59 for a rules health scoreSynthetic benchmark
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.AUC on a held-out company never used for training or tuningSynthetic benchmark
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.of churned ARR flagged at least 30 days ahead when ranked by ARR at riskSynthetic benchmark
At a glance
- Question
- Which customers are most likely to cancel in the next 90 days, and why?
- Data
- Ledgerly (synthetic) for training and tuning, and a second synthetic company never used for either.
- Compared with
- A rules-based customer-health score.
- How it was tested
- An out-of-time back-test on point-in-time monthly snapshots, plus a leakage test that rebuilds the data as it would have been exported on each day.
- Result
- AUC 0.88 against 0.59 for the rules score, and 0.87 on the held-out company. Ranked by ARR at risk, 70% of churned ARR was flagged at least 30 days ahead.
- What it doesn’t show
- Three of four flags don’t churn within 90 days. The data is synthetic, so the AUCs describe this simulation, not a real customer base.
The problem
A health score is not a churn forecast. A typical rules-based CS score tracks health but barely ranks who leaves next quarter.
What I built, in detail
- Point-in-time monthly snapshots where every feature uses only what was knowable that day.
- A contractual renewal gate plus a sign-constrained logistic regression, so each score splits exactly into per-feature reasons an agent can quote.
- JSON reason cards for agents and a CSM-ready table, with evidence IDs that link to the ticket.
- A leakage test by deletion: rebuild the raw data as it would have been exported on a given day and check every feature comes out identical.
How I checked it
- I injected three leaks to make sure the leakage test bites. It caught all three.
- The results that don’t flatter it are written down too: three of four flags don’t churn within 90 days.
Where it came from
My follow-up agent looks for past clients who have gone quiet. This is the rigorous, back-tested version. Read the AI CRM case study.
Stack
DuckDB SQL, Python, pandas, scikit-learn, scipy, matplotlib, pytest
More screens
