Decision tables and pairwise testing

← back to the section

A customer writes to support: the promo code did not work. You open the order form and check one field at a time. The promo code applies, the loyal-customer discount is granted, cash on delivery goes through. Every field works — and the bug lives in a combination: the promo code disappears only for a loyal customer who chose cash on delivery.

Equivalence classes and boundary values do not answer that — they are about a single field. Two techniques do: a decision table lays the combinations out when each of them has its own result, and pairwise testing compresses the list when the combinations run into the dozens.

A combination is a test target of its own

It feels like a form is covered once every field is covered. That is exactly what the bug at the top of this article lives on: behind the form there is not one comparison per field but a chain of nested conditions — "if a promo code is entered, 10% off; otherwise, if the customer is loyal, 5% off; if payment is cash on delivery, no discounts". Moving the fields one at a time, you keep walking the upper branches, while the branch that fires only on a combination never runs: there is nothing to switch it on. On top of that the branches were written by different people at different times — the June rule went to the top of the chain, and the March discount below it stopped being reached.

Hence the choice of technique: if every combination has its own result (the discount amount, the action that is allowed), every column matters — that is decision table territory; if the conditions are mere background and the result is the same everywhere, that is pairwise territory.

Decision table: five steps from a requirement to checks

A requirement is written as prose, and prose does not enumerate cases: it looks complete right up to the moment you try to apply it to a specific order.

A promo code gives a 10% discount. Loyal customers get 5%. No discounts apply when paying cash on delivery.

There are five steps. Write out the conditions — questions about the order whose answer changes the result: is a promo code entered, is the customer loyal, is payment by card or cash on delivery. Write out the values of each: all three have two here, "yes" and "no"; they are the same equivalence classes applied to a whole condition. Multiply: 2 × 2 × 2 = 8 rows, and while the list still fits on a screen it is written out in full — the missing case shows up precisely in the written-out form. Fill in the actions, taking them from the requirement rather than from the running program; where the requirement says nothing, leave the cell empty. Collapse the repeats — that is the next section.

Collapsing, and the empty cell in the requirements

Eight rows mean eight checks, and half of them are about the same thing: whenever payment is not by card, the discount is zero regardless of the other answers. Rows with the same result that differ in exactly one condition can be merged — that condition is replaced by a dash meaning "does not matter". The bottom four rows collapse into one, and eight becomes five. A dash means not "no need to check" but "does not affect the result, take any value", and the representative you take is the most contentious one: a customer who has both a promo code and loyal status and pays cash on delivery. Collapsing is a claim about the requirement: change it and the row unfolds back into four, which is why the uncollapsed table is kept next to the short one.

The most valuable thing in the table is not the ready-made checks but the cell you have nothing to fill in. Here it is the first row: a promo code is entered, the customer is loyal, payment is by card. The requirement mentions 10% and it mentions 5%, but it says nothing about both being true. There are at least four answers, and each is implemented somewhere: the discounts stack (15%), the larger one wins (10%), the promo code wins, the status wins (5%). A developer will silently pick whichever came to mind first — so a question mark is not a reason to guess but a finding for the questions about the requirements: a mistake found in the text is the cheapest kind there is. Gaps come in three kinds: the requirement is silent — the product owner has to answer; it is described twice and differently — a contradiction; the combination should not exist at all (instalments with pickup) — it is forbidden in the interface rather than handled.

Where forty-eight comes from

While there are three conditions, everything fits in your head. A real order form is bigger: promo code — 2 values, customer status (new, regular, loyal) — 3, payment method (card, bonus points, cash on delivery, instalments) — 4, delivery (courier or pickup) — 2. That gives 2 × 3 × 4 × 2 = 48 combinations: four conditions instead of three, and forty-eight combinations instead of eight — each new condition does not add cases, it multiplies them. A "gift wrapping" checkbox makes it 96, a choice of three cities makes it 288. If placing one order takes about six minutes, forty-eight checks are nearly five hours on a single form. And most of those combinations travel through the same code branches, differing only in numbers.

Pairwise coverage: from forty-eight down to twelve

You cannot test everything and picking at random is frightening, so you need a criterion for when a set is enough. "We will test the important ones" is not one: a week later nobody remembers what counted as important.

The criterion comes out of defect statistics: a large share of bugs fire on a single value, pairs account for most of the rest, and defects that need five or six conditions to coincide are almost never found — a mistake survives where two branches met and the order between them was never agreed. Hence the criterion: a set is sufficient if every pair of values of any two conditions appears in it at least once.

The lower bound takes a second to work out: take the two widest conditions, customer status (3 values) and payment method (4). They have 3 × 4 = 12 pairs, every one must appear, and a single row holds one such pair — so there cannot be fewer than twelve rows.

The set is verified by hand: "promo code + pickup" is row 4, "loyal + cash on delivery" is row 11. Sets like this are built by generators, the best known being the command-line PICT, and one of their features is mandatory: constraints such as "instalments are not available with pickup". Without them the set will contain rows that physically cannot be entered into the form; you cross them out on the fly and lose the pairs they carried. And last: a pairwise set is a list of input data, not of test cases — the expected result for every row is written from the same decision table.

What pairwise coverage misses, and when you test everything

The compression has a price: twelve rows cover every pair — and only pairs. Combinations of three conditions along the "status + payment + delivery" axes number exactly 3 × 4 × 2 = 24, and twelve rows hold at most twelve of them: half the triples are never tested. "New customer + instalments + courier" does not appear in the table, and if the bug sits precisely there, all twelve checks come back green.

There are three ways to handle that: add three or four risky rows by hand — where a bug was fixed recently, what a customer complained about; raise the strength of coverage to triples, though the lower bound is then 3 × 4 × 2 = 24, twice as expensive, so it is done for one area rather than the whole form; or split the form — the discount calculation is tested exhaustively with a decision table, while the background around it is covered pairwise.

Exhaustive testing comes back where an untested combination costs more than the hours it saves: money — discounts, taxes, bonus points, refunds, where a triple means a wrong number on the receipt; access rights — role, object owner, object state, action, where it means somebody else's order opening for a stranger. And pairwise is not worth it when there are few conditions: three conditions with two values each give eight checks and five after collapsing — a trip to a generator does not pay off. The everyday rule: every combination has its own result — decision table, and you test all the columns; the result is the same everywhere and the conditions are many — pairwise coverage.

In short

• Testing fields one at a time does not find a combination bug: the branch that fires only on the combination never runs.
• A decision table unfolds a requirement into cases: conditions → values → multiply → an action for each combination → collapse the repeats.
• The number of combinations is the product of the value counts, not their sum: 2 × 3 × 4 × 2 = 48.
• A cell you have nothing to fill in is the table's main finding: the combination is clarified, not guessed.
• Pairwise coverage asks for every pair of values of any two conditions at least once; fewer than twelve rows is impossible — that is the product of the two widest conditions, 3 × 4.
• The price of the compression is triples: half of the 24 make it into the set, which is why money and access rights are tested exhaustively.

What to read next

• Test design techniques: equivalence classes and boundary values — how to choose values for a single field; combinations are built out of those.
• Requirements review: techniques and good questions — what to do with a cell you cannot fill in.
• Checklists and test data — where the orders and promo codes for the twelve rows come from.
• Cross-browser and mobile testing — the most common case where conditions run into the dozens and the expected result is one.