Unit tests tell you the code is correct for one user. A warehouse has dozens of scanners confirming tasks at the same second, so we wrote a simulator: simulated pickers, receivers and supervisors running a full order cycle against a real database.
What we measured
- More than 16,000 API calls in one run, across receiving, putaway, waves, picking, shipping and counting.
- Thirteen data-integrity invariants checked afterwards: stock balances match movements, no negative bins, no orphan handling units.
- Latency percentiles per call, so slow paths are visible.
What broke
Modern MariaDB can run with snapshot isolation. Under it, two scanners locking the same stock row do not simply wait: the second one gets an error saying the record changed. Our locking reads needed to see the latest committed data, and rows had to be locked in a fixed order to avoid deadlocks.
The fix was small. Finding it would have taken weeks of production incidents instead of one afternoon.
The takeaway
Run your system the way your busiest day will, before your busiest day does it for you. The result was zero server errors and all invariants clean, and the simulator now runs after every significant change.