mutator

Overview

mutator is an automated mutation testing tool for the R language. It applies mutation testing principles to help developers improve test suite quality by introducing small, systematic changes (mutations) to source code and verifying if tests can detect these changes.

For instance, imagine you have the following function f in your package:

f <- function(x) {
  if (x > 0) {
    return(x + 1)
  } else {
    return(x - 1)
  }
}

mutator will generate several mutants, including:

f <- function(x) {
  if (x < 0) { # Mutated comparison operator
    return(x + 1)
  } else {
    return(x - 1)
  }
}

In this mutant, > has been replaced with <.

If your test suite does not catch this change, it indicates that your tests may not be comprehensive enough.

mutator will compute how many mutants survives, i.e. the test suite does not fail for the mutant, and how many mutants are killed, i.e. the test suite fails for the mutant. The ratio of killed mutants to total mutants is called the mutation score and is a measure of test suite effectiveness.

For a broader introduction to the tool, see the useR! 2026 talk on mutator.

Features

Comprehensive Mutation Testing: Applies various mutation operators to R source code
AST-Based Mutation: Uses Abstract Syntax Tree analysis for intelligent code mutations
Parallel Test Execution: Runs tests in parallel for improved performance
Coverage-guided Test Selection: Runs only the tests that cover mutated lines, also for improved performance
Configurable Test Harness: First-class testthat and tinytest support, plus a generic installed-tests fallback for other layouts
Timeout Management: Automatically calibrates timeouts for mutant test runs to prevent hangs
Annotations and Exclusions: Allows developers to exclude specific files or code sections from mutation testing
Equivalent Mutant Detection: Identify mutants that are functionally equivalent to the original code (currently, using LLMs)
Detailed Reporting: Provides mutation scores and analysis of test suite effectiveness

Installation

# Install the released version from CRAN
install.packages("mutator")

Development version

# Install the development version from GitHub; package dependencies are installed automatically
# install.packages("remotes")
remotes::install_github("PRL-PRG/mutator")

# Or install pre-built binaries of the development version from R-universe
# (https://prl-prg.r-universe.dev)
install.packages("mutator", repos = c(
  "https://prl-prg.r-universe.dev",
  "https://cloud.r-project.org"
))

# Or install from local source, in an R console
# install.packages("devtools")
setwd("path/to/mutator")
devtools::install()

Quick Start Guide

library(mutator)

# Mutate a single file
mutants <- mutate_file("path/to/your/file.R")

# Optional: cap returned mutants by random selection
mutants <- mutate_file("path/to/your/file.R", max_mutants = 20)

# Mutate an entire package and run tests
result <- mutate_package("path/to/your/package")

# Optional: cap tested mutants across the whole package
result <- mutate_package("path/to/your/package", max_mutants = 100)

# Optional: set a fixed timeout (seconds) per mutant test run
result <- mutate_package("path/to/your/package", timeout_seconds = 60)

# Optional: control where mutant files are written
result <- mutate_package("path/to/your/package", mutation_dir = tempdir())

mutator, in addition to show you the number of generated mutants, the surviving ones, and the mutation score, returns an invisible list with four components, a list of the generated mutants, a list of mutant outcomes, a list of phase durations, and a summary of the mutation testing run.

Mutant outcomes are reported as:

SURVIVED: tests passed for the mutant
KILLED: tests failed (or execution error)
HANG: mutant exceeded timeout

See the pkgdown reference for the full argument and return-value documentation.

Continuous integration (GitHub Actions)

mutator ships a reusable workflow so any R-package repository can run mutation testing in CI without copying scripts. Add a caller workflow at .github/workflows/mutation-testing.yaml:

on:
  pull_request:
  push:
    branches: [main, master]

name: mutation-testing

jobs:
  mutation:
    uses: PRL-PRG/mutator/.github/workflows/mutation-testing.yaml@v0.3.0
    with:
      target-margin: "0.10" # sample to +/-10 percentage points
      fail-under: "75" # fail CI below a 75% mutation score

Pin to a released tag such as @v0.3.0; the workflow is versioned with the mutator package, so the tag matches the package version. It installs mutator from CRAN and falls back to the matching GitHub tag when CRAN cannot provide it; mutator-source: r-universe or mutator-source: github picks another source. Set deploy-badge: true (with contents: write permission) to publish a shields.io badge. See the Continuous integration vignette for every input, threshold guidance, and badge setup. From an installed copy, open it with vignette("continuous-integration", package = "mutator").

Configuration

mutate_package() exposes a number of options to control how mutants are run, which tests are selected, and how results are refined. Each is covered in depth in the Configuration vignette (from an installed copy, open it with vignette("configuration", package = "mutator")):

Test strategy selection (strategy): how mutator auto-detects a package’s test harness (testthat, tinytest, or the generic installed-tests fallback) and when to override it with tinytest-installed.
Timeouts and the contended baseline: how the per-mutant HANG timeout is self-calibrated from a parallelism-aware baseline, and timeout_seconds to override it.
CRAN mode (cran): run the tests CRAN would (skipping guarded slow/network tests) or the full suite.
Fail-fast (fail_fast): stop each mutant’s run at the first failing test.
Parallel execution and isolation (isolate, cores): symlink-vs-copy of the package tree and how to handle non-hermetic tests, plus the optional pbmcapply progress bar.
Excluding code from mutation: exclude_files, in-source # mutator:ignore-* directives, covr # nocov annotations, and .covrignore.
Coverage-guided test selection (coverage_guided, coverage_backend): on by default, runs only the tests that cover each mutated line (testthat and tinytest strategies; warns and runs the full suite under the generic installed-tests fallback).
Precise mutant locations: the optional imputesrcref package for narrower operator-mutant source ranges.
Equivalent mutant detection (detectEqMutants): configuring the OpenAI-compatible API used to flag equivalent mutants.

Mutation Operators

mutation_operators() lists the operators. The operators argument of mutate_file() and mutate_package() selects them by id or family, and - removes one, e.g. operators = c("default", "seq_idiom", "-bool_flip").

On by default

Operator	Mutation
`arith_swap`	`+` ↔︎ `-`, `*` ↔︎ `/`
`arith_extra`	`^` → `*`, `%%` ↔︎ `%/%`, `-x` → `x`
`rel_swap`	`<` ↔︎ `>`, `<=` ↔︎ `>=`, `==` ↔︎ `!=`
`rel_boundary`	`<` ↔︎ `<=`, `>` ↔︎ `>=`
`logic_swap`	`&` ↔︎ `\\|`, `&&` ↔︎ `\\|\\|`
`not_remove`	`!x` → `x`
`and_or_operand`	`a && b` → `a` and → `b` (same for `\\|\\|`)
`cond_negate`	`if (c)`, `while (c)` → `if (!(c))`, `while (!(c))`
`loop_ctrl`	`break` ↔︎ `next`
`return_null`	`return(x)` → `return(NULL)`
`na_replace`	Constant → `NA` of the same type, e.g. `1` → `NA_real_`
`const_null`	Constant → `NULL`
`bool_flip`	`TRUE` ↔︎ `FALSE`, including argument defaults: `f(x, na.rm = TRUE)` → `f(x, na.rm = FALSE)`
`super_assign`	`x <<- v` → `x <- v`
`named_arg_drop`	Drops an argument such as `na.rm`, `drop`, `sep`, `collapse` or `fixed`, so that its default applies
`fun_swap`	`any` ↔︎ `all`, `min` ↔︎ `max`, `pmin` ↔︎ `pmax`, `which.min` ↔︎ `which.max`, `head` ↔︎ `tail`, `nrow` ↔︎ `ncol`, `isTRUE` ↔︎ `isFALSE`, `sub` ↔︎ `gsub`
`error_handling`	`tryCatch(e, ...)`, `try(e)`, `withCallingHandlers(e, ...)` → `e`; `stop()`, `stopifnot()` → `invisible(NULL)`
`stmt_delete`	Deletes a statement of a `{ }` block
`line_delete`	Deletes a source line (budget: `max_line_deletions`, `0` by default in `mutate_package()`)

Opt-in

Operators are opt-in when they create too many equivalent mutants (more than 20%) or too many mutants in general.

We ran each operator on seven packages (prettyunits, stringr, forcats, scales, jsonlite, lumberjack, R.methodsS3; about 10,600 source lines), with up to 100 sampled mutants per operator and package: 9,414 mutants, 3,996 of which survived. An LLM (qwen3.8-flash-next) judged whether each survivor is equivalent to the original code. We checked 56 of these verdicts by hand: 45 of the 46 “equivalent” verdicts were correct. The benchmark script is benchmarks/operator_selection.R.

Operator	Mutation	Why it is opt-in
`cond_force`	`if (c)` → `if (TRUE)`, `if (FALSE)`	Overlaps with `cond_negate`; 142 mutants per 1,000 source lines
`const_off_by_one`	Number `n` → `n + 1`, `n - 1`	185 mutants per 1,000 source lines
`value_42`	Numbers, assigned values and call results → `42` (or `0`)	860 mutants per 1,000 source lines, mostly killed by type errors
`string_empty`	Non-empty string → `""`	Mostly changes error and warning messages, which tests rarely check
`na_type_swap`	`NA` ↔︎ `NA_integer_` ↔︎ `NA_real_` ↔︎ `NA_character_`	62% of survivors are equivalent, e.g. any `NA` assigned into a character vector
`scalar_vector_logic`	`&&` ↔︎ `&`, `\\|\\|` ↔︎ `\\|`	67% equivalent: operands are usually scalars
`scalar_vector_fun`	`max(a, b)` ↔︎ `pmax(a, b)`, `min` ↔︎ `pmin`	83% equivalent: arguments are usually scalars
`index_ops`	`x[[i]]` ↔︎ `x[i]`	47% equivalent: same result for one element of an atomic vector
`seq_idiom`	`seq_len(n)` → `1:n`, `seq_along(x)` → `1:length(x)`	35% equivalent (lengths that cannot be zero); the other survivors show untested empty inputs
`drop_idiom`	`m[i, , drop = FALSE]` → `m[i, ]`	Rare; all 3 survivors were equivalent
`fun_swap_extra`	`paste` ↔︎ `paste0`, `floor` ↔︎ `ceiling`, `sapply` → `lapply`, …	30% equivalent, e.g. `paste` vs `paste0` with a single argument
`call_unwrap`	`rev(x)`, `unique(x)`, `as.numeric(x)`, … → `x`	24% equivalent: the wrapper often does not change the value

Dependencies

mutator depends on:

R Packages:
- pkgload: For loading mutated package copies
- testthat: For test execution
- xml2: For running C++ tests through testthat::run_cpp_tests()
- covr: For coverage-guided test selection (coverage_guided = TRUE)
- future and furrr: For parallel execution
- callr: For subprocess test execution and hard timeouts
- R6: For the per_file coverage backend reporter
- httr and jsonlite: For OpenAI API integration
- cli: For progress bars and user feedback
- pkgbuild: For building package copies for mutation testing
- pbmcapply: For progress bars in parallel execution
- imputesrcref: For precise mutant source ranges (optional)
LinkingTo: testthat (for Catch2 C++ test headers)
C++17: For native mutation engine implementation

mutator’s test suite

mutator itself uses testthat for its own R tests and testthat + Catch2 for C++ tests.

C++ tests are located in src/test-*.cpp
The C++ test runner is src/test-runner.cpp
C++ tests are executed from tests/testthat/test-cpp.R via run_cpp_tests("mutator")

Run the full test suite with:

devtools::test()

System tests

mutator includes a set of system tests that run mutation testing on 9 packages from CRAN. It compares the mutants, surviving, killed, hanged mutant detections, and the mutation score against the expected results using snapshot testing, with a fixed seed on 10 mutants or 50 mutants per package depending on the release cycle. It also compares the results across a set of configuration options that ought not to change the results. The system tests are located in tests/system and are automatically run as par of the CI.

Mutation testing

We practice dogfooding and run mutator on mutator test suite itself. This already led to numerous improvements in mutator speed, as waiting for the mutation score to be computed on a package with a long running test suite can be frustrating!