A recovery job, and then the hardening that followed it. I was handed an unfamiliar production payout system belonging to a client of a US managed service provider. Five days after first seeing it, two stuck batches of patient refunds were moving again.

The two batches had been stranded for 21 and 19 days. One had been sent to the bank three times and rejected at load each time, the last time with every state code corrected. The other was uploaded once and never registered at all. The working theory was the state codes. It was wrong. Comparing the rejected files record by record against three files the bank had accepted disproved that theory and isolated four defective records instead: two nine-digit phone numbers, one accented city name and one six-digit ZIP.

Corrected values came from the client the same afternoon. Two refunds already paid another way and one deleted on the client side came out of scope. Corrected files for 113 and 171 refunds were validated clean against the batch rows, six source rows were fixed, both files were uploaded and content hash-verified, the batch rows and attachments were updated, and the three old failed orders were cancelled at the bank. Both files reached Processed and the patients received their refunds. 284 refunds in total. The runbook for the whole procedure was written the same day the procedure was first run.

Then the part that matters more than the fix. Failures that used to be silent now report themselves: a monitor polls the bank portal every ten minutes and alerts Teams on a failed or stuck file, raising a ticket per failure. And the four defect classes that stranded these refunds are now refused at intake, so the request app rejects short phone numbers, non-ASCII characters and wrong-length ZIP codes before a request can be sent for approval. Nothing about how money moves was changed. Eight days after the monitor went live it caught a refused file first, so the client heard about the failure from us with the fix already made, inside two hours.

Honest limits: the cause is an inference, because the bank gave no reason. The corrected files loading and paying is consistent with it and does not prove it. The failure path of one of the fixes has not fired in production yet, so it is verified by read-back and design review rather than by a real failure. No client is named, and no figure beyond record counts appears here. That catch is one incident rather than a track record.