For many years, I’ve been working in capacity planning using a combination of mathematical models and one-off test tools.
I’m keeping the models: they’re what make the work predictive, but it’s time for a reusable tool that’s simple and easy to add options to.
The classic way to do load testing is to record and replay actual load from a real site. For example, the Samba smbtorture tool started out as a way to replay observed customer usage against the program, to reproduce bugs, to discover the hot-spots in the program and to test it a higher than normal loads.
I’m usually doing the latter, and these days I can get good recordings of actual load by grovelling over logs, as I mentioned before.
I’ve also been working in Go for the last year of so, and it allows me to build highly parallel programs using a variation on pipes and filters.
I also have been testing a disk sever recently, so all three parts have come together to start me on a new small project, https://github.com/davecb/Play-it-Again-Sam
This is a graph of what happens when I add more and more load to the disk on my machine “calvin”. At above 250-odd TPS, it starts to slow down.
In this diagram, higher is worse, as it means the response was slower. This is the typical hockey-stick curve of a system under load, “_/”. At some point it becomes overloaded, and the performance promptly becomes horrible.
Why it’s valuable
I now have a tool that will happily overload a system under test, so I can measure how well it does at the loads I actually care about. I can also feed the observed data into a queuing model and be able to model machines with more disk, CPU, bus speed and the like.
And, at the strictly practical level, I can replay tests easily for the developers while they’re chasing bugs they can’t reproduce at low loads.