CRM Dedupe Toolkit
Synthetic benchmark Synthetic data: Ledgerly
Finds duplicate accounts and contacts in CRM exports, decides which record survives, and produces merge files a Salesforce admin can act on, each with its reasons.
- Status
- Code on GitHub
- Relevant to
- Data and analytics, GTM Systems
Results · Synthetic data: Ledgerly
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.account auto-merge precision on a company the model never sawSynthetic benchmark
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.account F1 after steward review of 23 pairs, against 0.63 for a same-domain baselineSynthetic benchmark
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.contact recall after review, at 99.8% precisionSynthetic benchmark
- Synthetic benchmarkSynthetic data: Ledgerly. Measured against a hidden answer key in that data. It shows the method works there, not a production result.contact merges kept the right surviving recordSynthetic benchmark
At a glance
- Question
- Which CRM accounts and contacts are duplicates, and which record should survive a merge?
- Data
- Ledgerly’s synthetic CRM exports with a hidden list of true duplicates, and a second company the model never saw.
- Compared with
- Matching on the same website domain.
- How it was tested
- Normalize and block candidate pairs, train a small logistic model on 400 steward-reviewed pairs, and score merges against the answer key.
- Result
- Account F1 of 0.97 after steward review of 23 pairs, against 0.63 for the domain baseline, and 100% auto-merge precision on the unseen company.
- What it doesn’t show
- The duplicates and the answer key come from the same synthetic generator, so these scores describe that data, not a real CRM.
The problem
No single field is enough. Matching on website alone would merge 114 pairs of different companies that share a domain pattern, and matching on name alone is worse.
What I built, in detail
- Normalization for names, domains, states, emails, phones and nicknames, then blocking that cuts 4.9M possible account comparisons to about 2,700 candidates while keeping 99% of true duplicates.
- A logistic regression on field agreement levels with readable weights, trained on 400 steward-reviewed pairs. Every merge row lists its top reasons.
- Union-find clustering, survivorship rules, and merge-plan, review-queue, re-parent and golden-record files.
- An unsupervised Fellegi-Sunter model came first. Territory-correlated fields broke its independence assumption, so I switched to a small labeled sample.
Stack
Python, pandas, RapidFuzz, scikit-learn, pytest, GitHub Actions