Getting Started with bayespmtools

What is bayespmtools?

The bayespmtools package helps you determine sample sizes for external validation studies of risk prediction models using a Bayesian approach. Unlike traditional methods that require fixed performance values, this package allows you to incorporate uncertainty about model performance into your calculations.

Why use a Bayesian approach?

Traditional sample size calculations require you to specify exact values for metrics like the c-statistic or calibration slope. But in reality, we’re uncertain about these values. The Bayesian approach in bayespmtools lets you:

Express uncertainty about model performance using probability distributions
Calculate sample sizes based on expected precision OR assurance levels
Incorporate Value of Information (VoI) analysis to assess clinical utility

Quick Example

Let’s walk through a simple example. Suppose you’re planning to externally validate a risk prediction model and you have some prior information about its likely performance.

library(bayespmtools)
set.seed(123) # Set seed for reproducibility

Step 1: Specify Your Evidence

First, define what you know (or believe) about the model’s performance using probability distributions:

evidence <- list(
  prev ~ beta(116, 155),           # Outcome prevalence
  cstat ~ beta(3628, 1139),        # C-statistic
  cal_mean ~ norm(-0.009, 0.125),  # Mean calibration error
  cal_slp ~ norm(0.995, 0.024)     # Calibration slope
)

What this means:

prev: Outcome prevalence
cstat: c-statistic (discrimination)
cal_mean: Mean calibration error (differences between average observed and expected risks)
cal_slp: Expected calibration slope (from a logistic model regressing observed outcome on logit-transformed predicted risks)

You can parameterize distributions flexibly using means and SDs, confidence interval bounds, or natural parameters.

Step 2: Define Your Targets

Next, specify what precision you want to achieve. We want to evaluate sample size size on three rules:

Targeting expected 95% CI width for c-statistic, observed-to-expected outcome ratio (cal_oe), and calibration slope (cal_slp).
Targeting 90% assurance on calibration slope. In particular, we want to be 90% confident that the calibration slope’s CI width will be no greater a maximum tolerable value.
Using Value of Information (voI) criterion for net benefit: we want this validation study to reduce net benefit loss due to uncertainty by 90%.

targets <- list(
  eciw.cstat = 0.1,             # Expected CI width for c-statistic
  eciw.cal_oe = 0.22,           # Expected CI width for O/E ratio
  eciw.cal_slp = 0.30,          # Expected CI width for calibration slope
  qciw.cal_slp = c(0.9, 0.35),  # 90% assurance that CI width ≤ 0.35
  voi.nb = 0.90
)

Step 3: Calculate Sample Size

Now run the main calculation:

results <- bpm_valsamp(
  evidence = evidence,
  targets = targets,
  n_sim = 1000,           # Number of Monte Carlo simulations
  threshold = 0.2         # Risk threshold for net benefit calculations
)

NOTE: the number of simulations (n_sim) of 1,000 is low and is used here for convenience. In practice, consider simulation sizes of at least 10,000 and consider stability of results using different random seed.

Step 4: View Results

print(results$results)
#>   eciw.cstat  eciw.cal_oe eciw.cal_slp qciw.cal_slp       voi.nb 
#>          347          430         1037          896          717

The output shows the required sample size for each criterion. The largest sample size (1037) ensures all targets are met. However, the VoI criterion indicates that a sample size of 717 is expected to reduce uncertainty-related clinical utility loss by 90%. This might be used to reason that the criteria on calibration slope can potentially be relaxed.

Next Steps

For more advanced usage, see the full tutorial vignette:

vignette("bayespmtools_tutorial")

This covers: - Working with different distribution types - Net benefit and Value of Information analysis - Precision calculations for fixed sample sizes - A real-world case study

Key Functions

bpm_valsamp(): Calculate required sample size given targets
bpm_valprec(): Calculate precision/VoI given a fixed sample size

Getting Help

For detailed documentation on any function:

?bpm_valsamp
?bpm_valprec

Visit the package repository: https://github.com/resplab/bayespmtools

References

For methodological details, see:

Sadatsafavi M, et al. (2025). Bayesian sample size considerations for external validation of risk prediction models. Statistics in Medicine. doi:10.1002/sim.70389 ```