Play it Again, Sam — a load testing tool

For many years, I’ve been working in capacity planning using a combination of mathematical models and one-off test tools.

I’m keeping the models: they’re what make the work predictive, but it’s time for a reusable tool that’s simple and easy to add options to.

Record-replay testing

The classic way to do load testing is to record and replay actual load from a real site.  For example, the Samba smbtorture tool started out as a way to replay observed customer usage against the program, to reproduce bugs, to discover the hot-spots in the program and  to test it a higher than normal loads.

I’m usually doing the latter, and these days I can get good recordings of actual load by grovelling over logs, as I mentioned before.

I’ve also been working in Go for the last year of so, and it allows me to build highly parallel programs using a variation on pipes and filters.

I also have been testing a disk sever recently, so all three parts have come together to start me on a new small project, https://github.com/davecb/Play-it-Again-Sam

An example

This is a graph of what happens when I add more and more load to the disk on my machine “calvin”. At above 250-odd TPS, it starts to slow down.

Figure 1 from 1 https://github.com/davecb/Play-it-Again-Sam/blob/master/Running_Record-Reply_Tests.md

In this diagram, higher is worse, as it means the response was slower. This is the typical hockey-stick curve of a system under load, “_/”. At some point it becomes overloaded, and the performance promptly becomes horrible.

Why it’s valuable

I now have a tool that will happily overload a system under test, so I can measure how well it does at the loads I actually care about.  I can also feed the observed data into a queuing model and be able to model machines with more disk, CPU, bus speed and the like.

And, at the strictly practical level, I can replay tests easily for the developers while they’re chasing bugs they can’t reproduce at low loads.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s