Package {scindex}

Title:	Strategic Convergence Index for Inter-Rater Reliability
Version:	0.1.0
Description:	Inter-rater reliability analysis for binary classification tasks involving two or more raters within a signal detection-theoretic framework. User-supplied rating data are standardised into a common long-format structure. The package automatically computes Cohen's kappa for two raters and Fleiss' kappa for multiple raters. When ground-truth labels are available, rater-specific hit rates, false-alarm rates, sensitivity, specificity, and decision thresholds are estimated from observed classification responses using standard signal detection-theoretic transformations (DeCarlo, 1998) <doi:10.1037/1082-989X.3.2.186>. The package implements the Strategic Convergence Index (SCI; Gianeselli, 2026) <doi:10.1177/00131644261417643>, defined as SCI = 1 - [Var(t_i) / Var_max], where Var(t_i) denotes the variance of rater-specific decision thresholds and Var_max denotes the reference variance under maximal threshold dispersion. SCI quantifies convergence in rater decision criteria beyond observed agreement alone and complements classical agreement coefficients by distinguishing agreement in observed categorical outcomes from convergence in latent decision thresholds under an explicit signal detection-theoretic model of categorical judgment. The package provides structured summaries and threshold-based diagnostics for applications in which similar agreement coefficients may reflect substantively different underlying decision criteria across raters.
License:	GPL-3
Encoding:	UTF-8
RoxygenNote:	7.3.3
Depends:	R (≥ 4.1.0)
Imports:	stats
Suggests:	testthat (≥ 3.0.0), knitr, rmarkdown
Config/testthat/edition:	3
LazyData:	true
NeedsCompilation:	no
Packaged:	2026-05-18 13:05:50 UTC; irene
Author:	Irene Gianeselli [aut, cre]
Maintainer:	Irene Gianeselli <irene.gianeselli@unibz.it>
Repository:	CRAN
Date/Publication:	2026-05-21 13:40:02 UTC

Estimate SDT Decision Thresholds

Description

Internal utility for estimating signal-detection-theoretic decision thresholds from binary ratings and observed truth labels.

Usage

estimate_sdt_thresholds(
  data,
  item_col,
  rater_col,
  rating_col,
  truth_col,
  positive = 1
)

Arguments

Details

For each rater, the function computes:

• hit rate;

• false-alarm rate;

• SDT decision threshold.

Thresholds are estimated as:

t_i = -\frac{1}{2}[z(H_i) + z(F_i)]

where:

• H_i is the hit rate;

• F_i is the false-alarm rate;

• z(\cdot) is the inverse standard normal CDF.

A log-linear correction is applied to avoid infinite z-values.

Value

A list containing:

thresholds

Data frame with SDT quantities for each rater.

mean_threshold

Mean threshold across raters.

threshold_variance

Variance of thresholds.

References

DeCarlo, L. T. (1998). Signal detection theory and generalized linear models. Psychological Methods, 3(2), 186–205. https://doi.org/10.1037/1082-989X.3.2.186

Macmillan, N. A., & Creelman, C. D. (2004). Detection theory: A user's guide (2nd ed.). New York, NY: Psychology Press. https://doi.org/10.4324/9781410611147

Print a scindex Object

Description

Print method for objects of class "scindex_fit".

Usage

## S3 method for class 'scindex_fit'
print(x, digits = 3, ...)

Arguments

Details

The method displays:

• agreement coefficient;

• agreement type;

• SCI when available;

• basic sample information.

Value

Invisibly returns x.

Compute Agreement and the Strategic Convergence Index

Description

scindex() computes classical inter-rater agreement and, when a ground-truth variable is available, the Strategic Convergence Index (SCI). The function automatically returns Cohen's kappa for two raters and Fleiss' kappa for more than two raters.

Usage

scindex(
  data,
  item,
  rater,
  rating,
  truth = NULL,
  positive = 1,
  variance_bounds = NULL
)

Arguments

Details

SCI is computed from signal-detection-theoretic decision thresholds and should be interpreted as a model-based index of convergence in rater decision criteria, not as a replacement for agreement coefficients.

Value

An object of class "scindex_fit" containing the agreement coefficient, its type, SDT thresholds when available, SCI when available, and diagnostic information.

Examples

ratings <- data.frame(
  item = rep(1:6, each = 2),
  rater = rep(c("A", "B"), times = 6),
  rating = c(1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1),
  truth = rep(c(1, 0, 1, 1, 0, 1), each = 2)
)

fit <- scindex(
  data = ratings,
  item = item,
  rater = rater,
  rating = rating,
  truth = truth
)

fit

Example Binary Inter-Rater Dataset

Description

Simulated binary classification ratings from two raters across eight items, including ground-truth labels.

Usage

data(scindex_example)

Format

A data frame with 16 rows and 4 variables:

item

Item identifier.

rater

Rater identifier.

rating

Binary rating assigned by the rater.

truth

Ground-truth binary classification.

Details

The dataset illustrates the computation of Cohen's kappa and the Strategic Convergence Index (SCI).

Source

Simulated example data.

Examples

data(scindex_example)

fit <- scindex(
  data = scindex_example,
  item = item,
  rater = rater,
  rating = rating,
  truth = truth
)

fit
summary(fit)

Example Binary Inter-Rater Dataset with Multiple Raters

Description

Simulated binary classification ratings from three raters across eight items, including ground-truth labels.

Usage

data(scindex_example_multi)

Format

A data frame with 24 rows and 4 variables:

item

Item identifier.

rater

Rater identifier.

rating

Binary rating assigned by the rater.

truth

Ground-truth binary classification.

Details

The dataset illustrates the computation of Fleiss' kappa and the Strategic Convergence Index (SCI) in a multi-rater setting.

Source

Simulated example data.

Examples

data(scindex_example_multi)

fit <- scindex(
  data = scindex_example_multi,
  item = item,
  rater = rater,
  rating = rating,
  truth = truth
)

fit
summary(fit)

Summarize a scindex Object

Description

Summary method for objects of class "scindex_fit".

Usage

## S3 method for class 'scindex_fit'
summary(object, digits = 3, ...)

Arguments

Details

The method returns a structured summary including the agreement coefficient, SCI when available, SDT threshold estimates, and basic diagnostic information.

Value

Invisibly returns object.