Play it Again, Sam — a load testing tool

For many years, I’ve been working in capacity planning using a combination of mathematical models and one-off test tools.

I’m keeping the models: they’re what make the work predictive, but it’s time for a reusable tool that’s simple and easy to add options to.

Record-replay testing

The classic way to do load testing is to record and replay actual load from a real site.  For example, the Samba smbtorture tool started out as a way to replay observed customer usage against the program, to reproduce bugs, to discover the hot-spots in the program and  to test it a higher than normal loads.

I’m usually doing the latter, and these days I can get good recordings of actual load by grovelling over logs, as I mentioned before.

I’ve also been working in Go for the last year of so, and it allows me to build highly parallel programs using a variation on pipes and filters.

I also have been testing a disk sever recently, so all three parts have come together to start me on a new small project,

An example

This is a graph of what happens when I add more and more load to the disk on my machine “calvin”. At above 250-odd TPS, it starts to slow down.

Figure 1 from 1

In this diagram, higher is worse, as it means the response was slower. This is the typical hockey-stick curve of a system under load, “_/”. At some point it becomes overloaded, and the performance promptly becomes horrible.

Why it’s valuable

I now have a tool that will happily overload a system under test, so I can measure how well it does at the loads I actually care about.  I can also feed the observed data into a queuing model and be able to model machines with more disk, CPU, bus speed and the like.

And, at the strictly practical level, I can replay tests easily for the developers while they’re chasing bugs they can’t reproduce at low loads.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s