DecisionDrift: Auditing Temporal Drift in Repeated Decision Systems

Subir Hait

2026-04-12

Overview

DecisionDrift provides tools for detecting, decomposing, and stress-testing temporal drift in repeated binary decision systems. It is designed for settings in which an institutional or algorithmic system issues a binary decision — approve or deny, flag or pass, intervene or not — repeatedly over time for the same or overlapping set of units.

The package does not estimate causal effects and does not require full specification of the underlying decision rule. Instead, it treats the decision system as a time-indexed stochastic process and asks whether that process changed over time — and if so, how, when, and for whom.

Formal framework

1. The decision system as a stochastic process

Let:

The decision system is defined as the stochastic process:

\[S = \{ D_{it} : i = 1, \ldots, N;\ t = 1, \ldots, T \}\]

Critically, \(S\) is treated as a generator of decisions over time, not as a parametric model. This makes the framework applicable to hybrid human–AI systems where the decision rule \(f_t\) is only partially observed.

2. The human–AI decision mechanism

In many applied settings, decisions are produced by a joint human–AI process:

\[D_{it} = f_t(X_{it},\ H_{it})\]

where \(X_{it}\) are inputs (features, signals) available to the system, \(H_{it}\) represents human influence (override, discretion, intervention), and \(f_t\) is the time-varying decision mechanism emerging from the interaction of algorithmic and human components.

Drift in this system means \(f_t \neq f_{t'}\), or equivalently:

\[\Pr(D_{it} = 1 \mid X_{it}, H_{it}) \text{ changes over time}\]

DecisionDrift audits changes in \(f_t\) without requiring its full specification. This is the core methodological contribution.

3. Definition of temporal drift

Marginal drift (level shift): a change in the marginal distribution of decisions across waves.

\[\text{Drift exists if } \exists\ t_1 \neq t_2 \text{ such that } P_{t_1} \neq P_{t_2}\]

where \(P_t(d) = \Pr(D_{it} = d)\).

Transition drift (dynamic instability): a change in the conditional distribution given the prior decision.

\[\text{Transition drift exists if } P_t(d_t \mid d_{t-1}) \text{ varies with } t\]

The distinction matters. A system can have stable prevalence but unstable dynamics (transition drift without marginal drift), or shifting prevalence with stable dynamics (marginal drift without transition drift). DecisionDrift separates these two phenomena.

The four drift indices

Decision Drift Index (DDI)

The DDI estimates the linear trend in wave-level decision prevalence:

\[p_t = \frac{1}{N} \sum_{i=1}^{N} D_{it}\]

\[\text{DDI} = \frac{\hat{\beta}}{\text{SD}(p_t)}\]

where \(\hat{\beta}\) is the OLS slope of \(p_t\) on \(t\). Standardising by \(\text{SD}(p_t)\) makes the DDI scale-invariant and comparable across datasets.

Properties:

Transition Drift Index (TDI)

Define the time-indexed transition probabilities:

\[\pi_t^{ab} = \Pr(D_{it} = b \mid D_{i,t-1} = a),\quad a, b \in \{0, 1\}\]

The TDI measures average absolute change in persistence and activation probabilities:

\[\text{TDI} = \frac{1}{T-1} \sum_{t=2}^{T} \frac{|\pi_t^{11} - \pi_{t-1}^{11}| + |\pi_t^{01} - \pi_{t-1}^{01}|}{2}\]

Properties:

Group Differential Drift (GDD)

Let \(\text{DDI}^{(g)}\) denote the DDI estimated within group \(g\). For groups \(A\) and \(B\):

\[\text{GDD}_{AB} = \text{DDI}^{(A)} - \text{DDI}^{(B)}\]

GDD measures inequality in system evolution, not static disparity. A positive \(\text{GDD}_{AB}\) indicates the system became more permissive for group \(A\) faster than for group \(B\). A value near zero indicates parallel drift regardless of any baseline disparity.

Cumulative Drift Burden (CDB)

\[\text{CDB} = \sum_{t=2}^{T} |p_t - p_{t-1}|\]

CDB captures total accumulated volatility in the decision rate, regardless of direction. Unlike the DDI, CDB is non-zero even when drift oscillates (e.g., the system alternates between permissive and restrictive periods). A high CDB with a near-zero DDI indicates instability without a directional trend.

Modules

The five core modules each estimate one aspect of drift:

Module Stochastic object estimated Index
dd_prevalence() \(\Pr(D_t = 1)\) over time DDI
dd_transition() \(\Pr(D_t \mid D_{t-1})\) over time TDI
dd_entropy_trend() Complexity of path distribution —
dd_group_drift() Heterogeneity in \(\Pr(D_t)\) across \(G\) GDD
dd_changepoint() Structural break in process —

Together they answer: Did the system change? The robustness and sensitivity modules then ask: Is that conclusion credible?

Worked example

# Simulate a system that became more permissive after wave 4.
# Group A drifts faster than Group B.

n_units <- 80
n_waves <- 8

probs_A <- c(rep(0.20, 4), rep(0.60, 4))  # sharp increase at wave 5
probs_B <- c(rep(0.25, 4), rep(0.40, 4))  # modest increase at wave 5

dat <- data.frame(
  id       = rep(seq_len(n_units), each = n_waves),
  time     = rep(seq_len(n_waves), times = n_units),
  decision = c(
    rbinom(40 * n_waves, 1, rep(probs_A, 40)),
    rbinom(40 * n_waves, 1, rep(probs_B, 40))
  ),
  group    = rep(c("A", "B"), each = 40 * n_waves)
)

dp <- dd_build(dat, id, time, decision,
               group      = group,
               event_time = 5L)
print(dp)
#> 
#> ── DecisionDrift Panel Object ──────────────────────────────────────────────────
#> • Units: 80
#> • Waves: 8 (1–8)
#> • Balanced: TRUE
#> • Group var: group
#> • Event time: 5
#> • Units dropped (min_waves): 0

Module 1 — Prevalence drift

prev <- dd_prevalence(dp)
print(prev)
#> 
#> ── Prevalence Drift (Module 1) ─────────────────────────────────────────────────
#> • Mean decision rate: 0.359
#> • Trend slope: 0.0557 (p = 0.0191)
#> • R²: 0.627
#> • Cumulative change: 0.4
#> • DDI: 0.323
plot(prev)

The DDI is positive and the slope is statistically significant, confirming that the overall decision rate increased over time.

Module 2 — Transition drift

tr <- dd_transition(dp)
print(tr)
#> 
#> ── Transition Drift (Module 2) ─────────────────────────────────────────────────
#> • TDI (Transition Drift Index): 0.1416
#> • Persistence drift |Delta P(1|1)|: 0.1319
#> • Reversal drift |Delta P(1|0)|: 0.1513
#> 
#> ── Linear trends in transition probabilities ──
#> 
#>   P(p11): slope = 0.0566, p = 0.1089
#>   P(p10): slope = -0.0566, p = 0.1089
#>   P(p01): slope = 0.0465, p = 0.1163
#>   P(p00): slope = -0.0465, p = 0.1163
plot(tr)

The TDI captures the instability in switching and persistence dynamics that accompanies the structural break at wave 5.

Module 3 — Entropy and stability trend

et <- dd_entropy_trend(dp, window = 3L)
print(et)
#> 
#> ── Entropy & Stability Trend (Module 3) ────────────────────────────────────────
#> • Method: binary
#> • Window: 3 waves
#> • Entropy trend slope: 0.0618 (p = 0.0107)
#> • Switching trend slope: 0.0296 (p = 0.0402)
plot(et)

Module 4 — Group-differential drift

gd <- dd_group_drift(dp)
print(gd)
#> 
#> ── Group-Differential Drift (Module 4) ─────────────────────────────────────────
#> 
#> ── Per-group trend slopes ──
#> 
#> # A tibble: 2 × 5
#>   group  slope     se p_value r_squared
#>   <chr>  <dbl>  <dbl>   <dbl>     <dbl>
#> 1 A     0.0729 0.0282  0.0412     0.528
#> 2 B     0.0384 0.0135  0.0297     0.573
#> ── GDD table ──
#> # A tibble: 1 × 4
#>   group_a group_b    GDD direction
#>   <chr>   <chr>    <dbl> <chr>    
#> 1 A       B       0.0345 A faster
#> • Gap trajectory: diverging (slope = 0.0345, p = 0.2468)
plot(gd)

Group A experienced faster drift than Group B (positive GDD). The gap trajectory shows divergence over time.

Module 5 — Change-point detection

cp <- dd_changepoint(dp)
print(cp)
#> 
#> ── Change-Point & Regime Detection (Module 5) ──────────────────────────────────
#> 
#> ── CUSUM ──
#> 
#> • Detected break wave: 4
#> • Max |CUSUM|: -3.631
#> 
#> ── Segmented regression ──
#> 
#> • Best breakpoint: 6
#> • delta_AIC (null - seg): 1.58
#> 
#> ── Event alignment ──
#> 
#> • Event wave: 5
#> • Pre-mean: 0.203
#> • Post-mean: 0.516
#> • Change: 0.312
#> • p-value: 5e-04
#> • Evidence: strong
plot(cp)

Both the CUSUM and segmented regression methods identify a structural break near wave 5, consistent with the known event time.

All four indices in one call

idx <- dd_indices(dp)
print(idx)
#> 
#> ── DecisionDrift Summary Indices ───────────────────────────────────────────────
#> • DDI (Decision Drift Index): 0.3233
#> • TDI (Transition Drift Index): 0.1416
#> • GDD (Group Differential Drift): 0.0345
#> • CDB (Cumulative Drift Burden): 0.6125

Full audit

aud <- dd_audit(dp, include_robustness = TRUE, verbose = FALSE)
print(aud)
#> 
#> ── DecisionDrift Audit Report ──────────────────────────────────────────────────
#> • Units: 80 | Waves: 8 | Balanced: TRUE
#> • Group var: group
#> • Event time: 5
#> 
#> ── Summary Indices ──
#> 
#> ── DecisionDrift Summary Indices ───────────────────────────────────────────────
#> • DDI (Decision Drift Index): 0.3233
#> • TDI (Transition Drift Index): 0.1416
#> • GDD (Group Differential Drift): 0.0345
#> • CDB (Cumulative Drift Burden): 0.6125
#> 
#> ── Prevalence Drift ──
#> 
#> • DDI = 0.323 | slope = 0.0557 | p = 0.0191
#> 
#> ── Transition Drift ──
#> 
#> • TDI = 0.1416 | persistence = 0.1319 | reversal = 0.1513
#> 
#> ── Change-point Detection ──
#> 
#> • CUSUM break: wave 4
#> • Segmented break: wave 6 (delta_AIC = 1.58)
#> 
#> ── Group-Differential Drift ──
#> 
#> # A tibble: 1 × 4
#>   group_a group_b    GDD direction
#>   <chr>   <chr>    <dbl> <chr>    
#> 1 A       B       0.0345 A faster
#> ── Robustness ──
#> • Fragility index: 0
#> ────────────────────────────────────────────────────────────────────────────────
#> ℹ Verdict: MODERATE DRIFT
#> ────────────────────────────────────────────────────────────────────────────────

Robustness and sensitivity

A key claim of the DecisionDrift framework is that observed drift is not an artefact of analytic choices or data problems. The robustness and sensitivity modules test this claim directly.

Robustness (dd_robustness()) tests whether the DDI sign and magnitude are stable across:

A fragility index summarises the proportion of analytic variants in which the slope changes sign. Values near zero indicate robust conclusions.

Sensitivity (dd_sensitivity()) probes vulnerability to plausible data problems:

A tipping-point estimate identifies the smallest perturbation level at which more than 50 per cent of resampled conclusions flip sign.

rob <- dd_robustness(dp, variants = c("lopo", "min_waves"))
print(rob)
#> 
#> ── Robustness Analysis (Module 6) ──────────────────────────────────────────────
#> • Baseline DDI: 0.323
#> • Fragility index: 0
#> 
#> Robustness table:
#> # A tibble: 12 × 6
#>    variant            DDI  slope p_value n_waves n_units
#>  * <chr>            <dbl>  <dbl>   <dbl>   <int>   <int>
#>  1 Baseline         0.323 0.0557  0.0191       8      80
#>  2 Leave out wave 1 0.341 0.0571  0.0591       7      80
#>  3 Leave out wave 2 0.313 0.0540  0.0499       7      80
#>  4 Leave out wave 3 0.303 0.0540  0.0399       7      80
#>  5 Leave out wave 4 0.312 0.0539  0.0228       7      80
#>  6 Leave out wave 5 0.331 0.0531  0.0103       7      80
#>  7 Leave out wave 6 0.305 0.0515  0.0376       7      80
#>  8 Leave out wave 7 0.314 0.0545  0.0487       7      80
#>  9 Leave out wave 8 0.396 0.0723  0.0143       7      80
#> 10 Min waves >= 2   0.323 0.0557  0.0191       8      80
#> 11 Min waves >= 3   0.323 0.0557  0.0191       8      80
#> 12 Min waves >= 4   0.323 0.0557  0.0191       8      80
plot(rob)

Conceptual positioning

DecisionDrift is designed to occupy a specific and currently underserved position in the audit toolbox:

We define temporal drift as a change in the decision-generating process over time and introduce a set of summary indices that quantify level shifts, dynamic instability, subgroup divergence, and cumulative change. By implicitly treating decision systems as evolving stochastic processes, the framework enables longitudinal auditing of hybrid human–AI systems without requiring full specification of the underlying decision rule.

This positions the package as semi-parametric auditing of dynamic systems — distinct from full Markov modelling, time-series estimation, and cross-sectional fairness testing.

Relationship to decisionpaths

decisionpaths constructs and summarises longitudinal decision paths, answering what happened. DecisionDrift tests whether the system that generated those paths changed over time, answering did the system change. A third package in this ecosystem, AIBias, will ask whether inequality accumulated or amplified over repeated decisions.

References

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.

Hait, S. (2025). Artificial intelligence as decision infrastructure: Rethinking institutional decision processes. Preprint.

Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423.