Blog · AutoCollect
Why every collections change should be tested against a control group.
A before-and-after chart can make almost any collections change look like a success. A control group is the only fair way to know.
Collections teams change things all the time: a new reminder message, a different call order, a new dialler setting, an AI voice agent, a revised field route. Then, a month later, someone puts a before-and-after chart on the screen and the room decides whether it worked.
We think that habit causes more bad decisions in collections than any single tool or strategy. Here is why, and what to do instead.
Why before-and-after charts mislead
Collections results move for reasons that have nothing to do with your change:
- Calendar effects. Salary dates, festivals, harvest seasons and month-end all shift when borrowers can pay.
- Portfolio mix. A new batch of disbursals from a different segment or geography changes the bucket’s behaviour.
- Upstream changes. A tighter credit policy six months ago shows up as a cleaner Bucket 1 today.
- Team changes. A strong manager joins, attrition hits the agent floor, a target changes.
- Regression to the mean. A bad month is often followed by a better one, change or no change.
Put a real improvement on top of a bad calendar month and it can look like a failure. Put a useless change on top of a good month and it looks like a win. Teams then scale the wrong things and drop the right ones.
What a control group fixes
A control group is a randomly chosen share of accounts that keeps the current process while the rest get the new one, in the same bucket, over the same days. Both groups feel the same salary dates, the same festival, the same portfolio mix. The difference between them is the effect of the change, and very little else.
It does not need to be complicated. The essentials:
- Pick one bucket and one metric before you start, such as the cure rate in the early bucket.
- Split accounts randomly, not by branch, agent or amount, so neither group is easier.
- Keep the control on today’s process, run honestly, not starved of attention.
- Run long enough to cover at least one full cycle of due dates.
- Decide the threshold in advance: what difference would make you switch?
The usual objections
“We cannot leave borrowers untreated.” You are not. The control group gets exactly what every borrower got last month. Nobody is denied a reminder or a call; they receive the existing treatment.
“It slows us down.” A few weeks of measured testing is slower than switching everything on today. It is much faster than spending a year on a change that was not working, or abandoning one that was.
“The numbers are too small.” Sometimes they are, in a small portfolio. Then the honest answer is that you cannot yet tell, and the test runs longer. That is still better than guessing from a chart.
Measure more than the headline
Cure rate is the obvious metric, but a change that raises cures by pushing harder on borrowers is not a win. Alongside it, compare the two groups on:
| Measure | Why it matters |
|---|---|
| Promises kept vs promises taken | Promises that break are activity, not collection |
| Complaints and disputes | Conduct cost of the new approach |
| Contact-hour and script breaches | Whether the change stays inside the rules |
| Agent time per cure | Whether people were freed for harder cases |
How we run it
This is how we deliver AutoCollect. In the first weeks we map your portfolio, buckets, policy and scripts and take a cure-rate baseline. One bucket then goes live, measured against a control group, before later buckets, field and other portfolios join. Inside the test, accounts are ranked by propensity to pay, reminders and payment links go out on WhatsApp, voice agents take promises within contact hours, and every call is summarised and scored, so the conduct side of the comparison is measured as carefully as the cures.
If you want the background, our comparison of propensity-to-pay and amount-based call lists explains what is usually being tested, and the collection efficiency and cure rate calculator helps you settle the metric before you start. Everything else is in the AutoCollect guides.
Questions
What is a control group in collections?
A randomly chosen share of accounts in the same bucket that keeps the current process while the rest receive the new one, so the difference between the groups shows the effect of the change.
How long should a collections test run?
At least one full cycle of due dates, and longer if the portfolio is small and the difference between groups is not yet clear.
Is it unfair to borrowers in the control group?
No. They receive exactly the treatment every borrower received before the test; nobody is denied reminders or calls.
See it on your own data. AutoCollect — The right borrower, the right channel. Book a 30-minute working session with an engineer.