Lesson 43 of 45 · Testing as a Discipline
Mutation Testing
95% coverage, still buggy
Your project shows 95% test coverage. CI is green. The team feels good about the suite. Then a bug ships in a piece of code that the coverage report claimed was tested. The lines were executed. Assertions ran. Nothing was checking that the right thing happened.
Coverage tools answer one question: "did this line of code get touched by some test?". They don't answer the deeper question: "if this line of code were wrong, would any test catch it?". Those are very different things, and the gap between them is where bugs hide.
Mutation testing is the tool for the deeper question. This lesson is about what it does, why it matters, and when it earns its place in PHP work.
What mutation testing actually does
The technique is mechanical and clever:
The tool changes a small piece of your code (a "mutation"). For example, it flips
<to<=, or removes areturn, or replaces+with-.It runs your test suite against the mutated code.
If a test fails, the mutation was killed. The suite caught the change.
If all tests still pass, the mutation escaped. The suite didn't notice the code changed.
Repeat for hundreds or thousands of small mutations. The share of generated mutants the suite detects is the Mutation Score Indicator (MSI).
A high MSI means the suite detected most of the changes the tool made. A low MSI means most mutated versions of your code still passed the tests. High coverage paired with a low MSI means most lines were executed, but most variations of those lines wouldn't have been caught.
Setting it up
Infection (infection/infection) is the mutation testing tool for PHP. It needs PHP 8.3 or newer, a code coverage driver (Xdebug, PCOV or phpdbg), and a test suite that is green before it starts, because it treats a failing initial run as an error. Install it as a dev dependency:
composer require --dev infection/infectionInfection reads infection.json5 from the project root. source.directories lists the PHP code to mutate, and minMsi and minCoveredMsi are the two thresholds that fail the run when the score drops below them (the next section defines both metrics):
{
"$schema": "vendor/infection/infection/resources/schema.json",
"source": {
"directories": ["src"]
},
"logs": {
"text": "infection.log"
},
"minMsi": 80,
"minCoveredMsi": 90
}Then run it. The --min-msi and --min-covered-msi options set the same thresholds from the command line, which is how a CI job overrides the file:
vendor/bin/infection --threads=4 --show-mutationsPick the thresholds just below the score you have today, so the gate stops the score from sliding without failing the first run.
A small example
The example reuses StrictCancellation from Lesson 19 of this course, unchanged. It refunds half the price when the guest cancels at least 14 days before check-in, and nothing otherwise. Money comes from Lesson 5 and DateRange from Lesson 6.
<?php
declare(strict_types=1);
namespace App\Reservations\Cancellation;
use App\Billing\Money;
use App\Reservations\DateRange;
use Carbon\CarbonImmutable;
use DateTimeImmutable;
final readonly class StrictCancellation implements CancellationPolicy
{
public function refundFor(Money $totalPrice, DateRange $stayDates, DateTimeImmutable $now): Money
{
return (int) CarbonImmutable::instance($now)->diffInDays($stayDates->checkIn) >= 14
? $totalPrice->applyDiscountPercent(50)
: Money::zero($totalPrice->currency);
}
}And this test:
<?php
declare(strict_types=1);
namespace Tests\Reservations\Cancellation;
use App\Billing\Currency;
use App\Billing\Money;
use App\Reservations\Cancellation\StrictCancellation;
use App\Reservations\DateRange;
use Carbon\CarbonImmutable;
use DateTimeImmutable;
use PHPUnit\Framework\TestCase;
final class StrictCancellationTest extends TestCase
{
public function test_cancelling_a_month_ahead_gets_a_refund(): void
{
$policy = new StrictCancellation();
$totalPrice = Money::fromCents(60000, Currency::usd());
$stayDates = DateRange::ofNights(CarbonImmutable::parse('2026-06-30'), 3);
$now = new DateTimeImmutable('2026-05-31');
$refund = $policy->refundFor($totalPrice, $stayDates, $now);
$this->assertFalse($refund->isZero());
}
}Coverage of StrictCancellation: 100%. Every line is executed. Running Infection against this one file generates five mutants and the suite kills one. --with-uncovered is explained in the next section, and the path argument limits the run to that file:
vendor/bin/infection --threads=4 --with-uncovered src/Reservations/Cancellation/StrictCancellation.php5 mutations were generated:
1 mutants were killed by Test Framework
4 covered mutants were not detected
Metrics:
Mutation Score Indicator (MSI): 20%
Mutation Code Coverage: 100%
Covered Code MSI: 20%The mutant the test kills is the Ternary one, which swaps the two branches. The refund would become zero, and assertFalse($refund->isZero()) fails. Two of the four that escape matter:
GreaterThanOrEqualTochanges>= 14to> 14. A guest cancelling exactly 14 days ahead would lose the refund the policy promises. The test uses a stay 30 days away, so30 > 14is still true.DecrementIntegerandIncrementIntegerchange the refund percentage from50to49and51. The test only checks that the refund is not zero, so any percentage passes.
Each escape points at a missing assertion. Replace the test with one that checks the exact amount and adds the boundary:
<?php
declare(strict_types=1);
namespace Tests\Reservations\Cancellation;
use App\Billing\Currency;
use App\Billing\Money;
use App\Reservations\Cancellation\StrictCancellation;
use App\Reservations\DateRange;
use Carbon\CarbonImmutable;
use DateTimeImmutable;
use PHPUnit\Framework\TestCase;
final class StrictCancellationTest extends TestCase
{
public function test_cancelling_a_month_ahead_refunds_half(): void
{
$policy = new StrictCancellation();
$totalPrice = Money::fromCents(60000, Currency::usd());
$stayDates = DateRange::ofNights(CarbonImmutable::parse('2026-06-30'), 3);
$now = new DateTimeImmutable('2026-05-31');
$refund = $policy->refundFor($totalPrice, $stayDates, $now);
$this->assertTrue($refund->equals(Money::fromCents(30000, Currency::usd())));
}
public function test_cancelling_exactly_fourteen_days_ahead_refunds_half(): void
{
$policy = new StrictCancellation();
$totalPrice = Money::fromCents(60000, Currency::usd());
$stayDates = DateRange::ofNights(CarbonImmutable::parse('2026-06-14'), 3);
$now = new DateTimeImmutable('2026-05-31');
$refund = $policy->refundFor($totalPrice, $stayDates, $now);
$this->assertTrue($refund->equals(Money::fromCents(30000, Currency::usd())));
}
}The exact amount kills the 49 and 51 mutants. The 14-day test kills > 14, because 14 > 14 is false and the refund would be zero. The same command now reports:
5 mutations were generated:
4 mutants were killed by Test Framework
1 covered mutants were not detected
Metrics:
Mutation Score Indicator (MSI): 80%
Mutation Code Coverage: 100%
Covered Code MSI: 80%The one that remains is CastInt, which removes the (int) before CarbonImmutable::instance($now)->diffInDays(...). It is an equivalent mutant: it changes the code but not what the code does. diffInDays returns a float, and the cast drops the fraction. For a whole-number threshold the cast changes nothing, because a float is >= 14 exactly when its truncated integer is. 13.9 fails both ways and 14.0 passes both ways. No test can tell the two versions apart, so no test can kill it. Infection can't prove that automatically, so it reports the mutant anyway.
On this single file the final run also exits with an error, because Covered Code MSI is 80% and the gate from the setup section is 90%. When the only survivor is equivalent, you either ignore that mutator for that method or lower the gate. The first is one entry in infection.json5:
{
"mutators": {
"@default": true,
"CastInt": {
"ignore": ["App\\Reservations\\Cancellation\\StrictCancellation::refundFor"]
}
}
}Add it next to the other keys. With it, the run generates four mutants and reports 100% on all three metrics.
This is what mutation testing does at scale. It probes hundreds of small variations of the code and reports which ones the suite missed. Each escape is either a specific hole in the suite or an equivalent mutant you can ignore.
Reading a mutation report
On a larger codebase the same report looks like this (a sample run with --with-uncovered, which includes code no test executes):
1247 mutations were generated:
1000 mutants were killed by Test Framework
89 mutants were not covered by tests
142 covered mutants were not detected
8 errors were encountered
8 time outs were encountered
Metrics:
Mutation Score Indicator (MSI): 81%
Mutation Code Coverage: 92%
Covered Code MSI: 87%A few useful numbers:
Killed. Mutations the suite caught. Good.
Covered mutants were not detected. The escaped mutants. These are the holes worth investigating.
Errors / time outs. Mutations that crashed or hung. Infection counts them as defeated, the same as killed.
Not covered. Mutations in code that no test executes at all. That means untested, which is a different finding from the code being dead.
The three metrics divide the same counts differently. "All mutations" excludes mutants that were skipped or ignored by configuration, and time outs count as defeated unless you set timeoutsAsEscaped to true in infection.json5. The console rounds down to whole percentages, so the exact values for this sample are slightly higher:
Metric | Formula | This report |
|---|---|---|
Mutation Score Indicator (MSI) | (killed + errors + time outs) / all mutations | 1,016 / 1,247 = 81.5% |
Mutation Code Coverage | (all mutations - not covered) / all mutations | 1,158 / 1,247 = 92.9% |
Covered Code MSI | (killed + errors + time outs) / (all mutations - not covered) | 1,016 / 1,158 = 87.7% |
Mutation Code Coverage is the nearest thing to line coverage, and it still counts mutants, not lines: a line that can be mutated five ways counts five times, and a line with nothing to mutate counts zero. It only says whether a test ran the code a mutant lives in. Covered Code MSI says whether those tests noticed the change, which is why it is the sharper number.
By default Infection only generates mutants in code that tests execute, and then the report has no "not covered" line and no plain MSI line. Pass --with-uncovered to include the rest. The Infection command line reference describes the option as "Allow mutation of code not covered by tests."
The detailed report (often an HTML file or a list) shows each escaped mutation: the file, the line, what was changed, and the failure to detect. Each one is a small puzzle: "why didn't any test fail when this changed?". Often the answer is "the test exercised the line but didn't actually verify the outcome that mutation broke", as with the refund percentage above.
The PHP landscape
Infection runs against PHPUnit test suites and mutates only the PHP files in source.directories. The first time you run it on an existing codebase, expect to find:
Tests that exercise code without asserting anything meaningful.
Boundary conditions that aren't tested (off-by-one bugs hiding).
Branch logic where the suite tests the happy path but not the negation.
Code with no test at all, which shows up as not covered when you pass
--with-uncovered.
When mutation testing earns its place
It earns its place when:
The codebase has critical domain logic. Pricing, billing, accounting, scheduling, security. The places where a missed bug costs real money or trust. A high MSI in these areas tells you the suite would notice a change to the rules that matter.
The team is past the early stage of testing. You already have decent coverage and want to know whether it's meaningful. Reaching for mutation testing on a codebase with 20% coverage is the wrong order; build the suite first, sharpen it second.
The suite is the team's confidence in refactoring. If you're going to trust the green CI to let you refactor freely, you want the green CI to be a reliable signal. Mutation testing tells you whether it is.
It doesn't earn its place when:
The codebase is mostly framework wiring. Infection mutates only the PHP in
source.directories. Templates (Blade or Twig) and configuration files are never mutated, and trivial controllers produce mutants that tell you little.The suite is too slow to mutate. Mutation testing runs the suite many times. If one run takes minutes, mutation testing on it takes hours. Speed up the suite first.
Pitfalls
A few common mistakes:
Chasing the score. "We need 95% MSI" as a metric becomes its own goal, and the suite gets padded with assertions that exist to kill mutations rather than to verify behavior. The point is the quality of the suite. Don't optimize the number; optimize the value.
Equivalent mutants. The removed (int) cast above is one. Tools can't detect every one of these, so don't chase 100% MSI; some escapes are unavoidable.
Treating it as part of every CI run. Mutation testing is slow. Running it over the whole codebase on every PR is expensive. Run the full analysis nightly, on the main branch, or weekly, and use the trend, not the per-commit report. In pull requests, vendor/bin/infection --git-diff-lines --git-diff-base=origin/main mutates only the added and changed lines.
Ignoring the report. The score on its own doesn't help. The escape report is where the value is. If nobody reads the escape list and adds tests for the holes, the score is just a number that gets watched.
The one thing to remember
Coverage tells you which lines were touched. Mutation testing tells you which lines were actually verified, and it builds on coverage: a line has to be executed before a mutation to it can be caught. It doesn't catch design problems or integration bugs. Run it occasionally and fix the holes it surfaces.
Related Links
- InfectionThe official site for PHP's mutation testing tool, covered in this lesson's PHP landscape section.
- Introduction to Mutation TestingInfection's own explanation of the Mutation Score Indicator and Covered Code MSI this lesson's report-reading section walks through.
- PHPUnitThe test framework Infection runs mutations against, referenced throughout this lesson's examples.