Paper reproduction status: what irtsim can and cannot reproduce

Scope

One of irtsim’s two stated goals is paper reproducibility: users should be able to run the Monte Carlo examples from Schroeders and Gnambs (2025), “Sample Size Planning for Item Response Models: A Tutorial for the Quantitative Researcher” (https://ulrich-schroeders.github.io/IRT-sample-size/), and compare irtsim output against the published reference.

This vignette is the honest scorecard for that goal as of the current release. It documents which of the paper’s three examples irtsim can reproduce end-to-end today, which it cannot, and what architectural changes would close the remaining gaps.

Summary table

Example	Paper scenario	irtsim status	Where to find it
Example 1	1PL linked two-form design, 30 items, 438 MC iterations, MSE criterion	Reproducible	`vignette("paper-example-1-linked-design")`
Example 2	2PL with bivariate θ + external criterion, MCAR, SE of cor(θ, criterion) via TAM latent-regression β	Not reproducible — architectural gaps	See “Example 2 gap” below
Example 3	GRM with leave-one-measure-out, RMSE of `testinfo(mod, θ = 2.0) / (1 + testinfo(mod, θ = 2.0))`	Not reproducible — fitted-model access gap	See “Example 3 gap” below

Example 2 gap (personality test / external criterion)

The paper’s Example 2 constructs a bivariate latent trait (θ, external criterion ξ) with population correlation ρ = 0.5, generates responses to 30 2PL items, applies MCAR missingness at 0%/33%/67%, fits the model with TAM::tam.mml() (or tam.mml.2pl()) using the external criterion as a latent regressor, and extracts SE(cor(θ, ξ)) = tam.se(mod)$beta[2, 2].

What irtsim is missing:

Bivariate θ generation. irt_design() accepts a single theta_dist (a distribution name or a function producing a univariate θ vector). There is no way to declare a bivariate (θ, external criterion) generating distribution.
External-covariate plumbing. fit_model() wraps mirt::mirt() directly and has no provision for passing an external criterion column through to a latent-regression estimator.
Latent-regression backend. The paper uses TAM, which irtsim does not depend on. mirt supports latent regression through mirt(..., covdata = ...), but irtsim does not expose that surface.
Criterion-callback signature. criterion_fn in summary.irt_results() receives only (estimates, true_value, ci_lower, ci_upper, converged) — all item- scoped. There is no way to hand the callback a fitted model, a θ estimate vector, or an external covariate vector, so even an ad hoc post-hoc reconstruction of the paper’s β SE is not expressible today.

Obj 30 in the project backlog tracks closing this gap.

Example 3 gap (GRM conditional reliability at θ = 2.0)

The paper’s Example 3 calibrates a 50-item GRM composed of three clinical symptom scales and computes, per iteration, testinfo(mod, Theta = 2.0) / (testinfo(mod, Theta = 2.0) + 1) — the conditional reliability at a target θ value. The criterion reported is the RMSE of that quantity across iterations.

What irtsim is missing:

Access to the fitted model. run_one_iteration() in irt_simulate.R calls extract_params() immediately after fit_model() returns and then discards the fitted mirt object. Nothing downstream has access to mod for test-info queries.
Custom quantity extraction per iteration. There is no hook analogous to extract_fn(mod, data) that would let a user pull arbitrary quantities out of the fitted model (test info at θ, reliability at θ, discriminant validity, etc.) and attach them to the per-iteration result store.
Post-hoc criterion on a non-parameter quantity. Even with criterion_fn in summary.irt_results(), the callback sees only item-level parameter estimates — not per-iteration scalar extracts that depend on the fitted model.

Obj 31 in the project backlog tracks closing this gap.

Why the gaps exist

irtsim’s pipeline was originally scoped tightly around item parameter recovery — estimands like bias, MSE, RMSE, coverage, and empirical SE on a, b, b1..bk. That scope is well served by the current (estimates, true_value, ci_lower, ci_upper) interface. The paper’s Examples 2 and 3, however, target estimands that are not item-scoped parameters at all:

Example 2’s estimand is a latent-regression β standard error — a scalar that lives in a fitted-model object, not in the item parameter vector.
Example 3’s estimand is a test-information-derived conditional reliability — a function of the fitted model evaluated at a specific θ, not an item parameter.

Closing both gaps with a narrow, one-off hook for each estimand would entangle irtsim with TAM and mirt::testinfo and would keep adding scope for every new paper example that showed up.

Preferred architectural direction

The project is considering a single pluggable-hook addition that subsumes both gaps without hard-coding any specific backend:

irt_simulate(
  study,
  iterations = 438,
  seed       = 2024,
  parallel   = TRUE,
  fit_fn     = my_fit_fn,     # user-supplied: fits a model their way
  extract_fn = my_extract_fn  # user-supplied: returns a named list of
                              # per-iteration scalars / vectors
)

The returned irt_results object would gain a third store alongside item_results and theta_results — call it extracted_results — in which each named output of extract_fn becomes a column. Users would then write their own summary/recommended_n logic on that slot, or irtsim would provide a thin convenience that computes Morris-style criteria on each extracted column.

This approach:

keeps irtsim’s built-in pipeline and its test coverage intact;
lets users reproduce Example 2 by plugging in a TAM fit closure plus an extract function that pulls tam.se(mod)$beta[2, 2];
lets users reproduce Example 3 by plugging in a mirt fit closure plus an extract function that pulls testinfo(mod, Theta = 2.0) / (1 + testinfo(...));
does not hard-code dependencies on TAM or on any specific test-info query;
leaves the door open for novel estimands no one has thought of yet.

The tradeoff is that the fit/extract contract becomes a new public surface that has to be documented, versioned, and tested.

Alternative: compose irtsim with external tools

Even without the pluggable hook, the paper’s examples can be partially reproduced by using irtsim for the pieces it handles well and stitching in external code for the rest:

irtsim handles: Monte Carlo orchestration, seed management (L’Ecuyer-CMRG substreams in parallel mode), sample-size grids, structured missingness (booklet, linking), summary criteria on item parameters.
Users handle outside irtsim: bivariate θ generation, external covariate construction, latent regression fitting (TAM), fitted-model queries (mirt::testinfo), and collection of per-iteration scalar extracts.

This is a reasonable pattern while Obj 30 / Obj 31 are deferred. It also argues against inflating irtsim’s dependency surface.

What Example 1 buys users today

If your goal is a published-reference comparison for irtsim’s core workflow — linked-test design, Rasch fitting, MSE of item difficulty vs. sample size — Example 1 gives you that comparison as a rendered vignette today. See vignette("paper-example-1-linked-design") for the faithful reproduction and a table of the design decisions mapped from the paper onto irtsim’s API.