A bit more on Pairwise testing

I tweeted yesterday about how i wished I had known about pairwise testing earlier, so perhaps a little explanation would help.

I attended at Scottish Test Forum event yesterday in Glasgow.
One of the presentations was given by Stephen Allot from Electromind and covered pairwise testing. Presentations will be available at http://www.bcs.org/server.php?show=nav.9979 in near future.

I hadn't come across it before and basically is a technique that assumes that almost all defects are due to one error or a combination of two errors and so we can apply the mathematics of pairs to reduced the number of tests required.

So if you are trying to test a situation with a number of parameters each with different possible values then the total possible combinations multiply up to huge numbers which become impossible or at least impractical testing, but pairwise can reduce that to much smaller and achievable numbers of tests needed.

The mathematics of pairs is well known (well its well known to mathematicians ;-) ) and goes back to Latin squares and orthogonal arrays and all that.

Once you've decided to use pairwise, there are lots of tools available (easy to find try Wikipedia or http://pairwise.org/ or http://www.satisfice.com/ etc.) that can take your parameters and produce the lists of tests. Some are freeware and some cost, the benefits of paying being that you can get better lists (shorter and with illegal combinations excluded).

An example of this is that if you had 10 parameters each with 10 values then the free tools would give you 177 tests to execute, whereas the paid-for tools could reduce that to 140. However if you compare that to the total combinations of 10 billion, then the difference between free and not is minor.

So at this point its a wow, what great reductions, but where did that assumption come from?

We are that assuming that almost all defects are due to one error or a combination of two errors. the answer I got was that it was all based on engineering and medical experience. Wikipedia showed the way to an IEEE paper that brought the research together
http://csrc.nist.gov/groups/SNS/acts/documents/TSE-0172-1003-1.pdf
The conclusion was that testing single errors only covered 68%, covering for pairs of errors covered 93%, triples covered 98% and quadruples 100%.

So a combination of pairwise testing plus defined test cases for known corner point issues should get pretty close to complete coverage with a fraction of the testing.


However the basis is all empirical so you will always have people saying its not applicable, and even the IEEE paper ends with "many more empirical studies of other classes of software are needed to evaluate the applicability of combinatorial testing for other classes of systems" so you must use the technique with your brain switched on.