sort_dat <- data.frame(
item = rep(c("A1", "A2", "A3", "B1", "B2", "B3"), each = 20),
rater = rep(1:20, 6),
target_construct = rep(c("A", "A", "A", "B", "B", "B"), each = 20),
assigned_construct = c(
rep("A", 18), rep("B", 2),
rep("A", 16), rep("B", 4),
rep("A", 13), rep("B", 7),
rep("B", 18), rep("A", 2),
rep("B", 17), rep("A", 3),
rep("B", 14), rep("A", 6)
)
)
fit <- sort_validity(sort_dat)
fit
#> contentvalidR item-sort analysis
#> --------------------------------
#> Items: 6 | Raters: 20 | Target scales: 2
#> Item inference: Howard-Melloy exact target-count test (p0 = 0.50, alpha = 0.050)
#> Judges: naive
#>
#> 4 item(s) meet the exact target-assignment criterion; 2 item(s) are flagged for review.
#> Review: A3, B3
#>
#> Item-level evidence:
#> item target n n_target competitor psa psa_low psa_high csv p_value
#> A1 A 20 18 B 0.90 0.699 0.972 0.8 0.000
#> A2 A 20 16 B 0.80 0.584 0.919 0.6 0.006
#> A3 A 20 13 B 0.65 0.433 0.819 0.3 0.132
#> B1 B 20 18 A 0.90 0.699 0.972 0.8 0.000
#> B2 B 20 17 A 0.85 0.640 0.948 0.7 0.001
#> B3 B 20 14 A 0.70 0.481 0.855 0.4 0.058
#> recommendation
#> Retain
#> Retain
#> Review
#> Retain
#> Retain
#> Review
#>
#> 95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#> compared seven methods and recommends score intervals over the Wald
#> interval. An interval reflects how few ratings an item received, not
#> whether the right judges were chosen.
#>
#> Scale-level Colquitt benchmark summary:
#> target n_items mean_psa psa_strength mean_csv csv_strength
#> A 3 0.783 Moderate 0.567 Moderate
#> B 3 0.817 Moderate 0.633 Strong
#> benchmark_set
#> Overall (not correlation-normed)
#> Overall (not correlation-normed)
#>
#> Colquitt labels are empirical percentile norms derived from scale-level averages,
#> not universal cutoffs or automatic scale-retention rules. They place a scale
#> against published scales; Psa and Csv sit on different scales, so their labels
#> are not comparable with each other.
#>
#> What these columns mean
#> psa -- Proportion of Substantive Agreement. Share of judges who assigned
#> the item to the construct it was written for. Higher means judges
#> recognized the item as belonging where you intended. (0 to 1; higher
#> is stronger)
#> psa_low/psa_high -- Interval for Psa. Lower and upper limits of an
#> interval around Psa. A wide interval means few judges sorted the
#> item, so a different sample of judges could plausibly give a quite
#> different Psa. (between 0 and 1; the method and level are named in
#> the output)
#> csv -- Coefficient of Substantive Validity. How much more often the item
#> went to its intended construct than to the alternative construct
#> judges chose most. It rewards being distinctly right, not merely
#> often right. (-1 to 1; 0 means the intended construct and its closest
#> rival were chosen equally often)
#> competitor -- Strongest competing construct. The construct, other than
#> the intended one, that judges chose most often for this item.
#> p_value -- Howard-Melloy exact test. Probability of seeing at least this
#> many target assignments if judges were assigning at the chance rate
#> p0. Small values mean the item's assignment pattern is unlikely to be
#> chance. (0 to 1; compared against alpha)
#>
#> What the status labels mean
#> Supported -- The evidence met the criteria set for this analysis.
#> Review -- Something here needs a closer look. This is not an instruction
#> to delete anything.
#> Insufficient data -- Too little usable data to reach a judgment.
#> Descriptive only -- Reported for description only; no decision rule was
#> applied.
#> Each workflow also uses its own wording in the recommendation column
#> (Retain, Strong support, Typical, Covered, and so on). Those words map
#> onto the shared statuses above.
#>
#> See `contentvalid_glossary()` for all terms, or set
#> `options(contentvalidR.show_key = FALSE)` to hide this key.
#>
#> 'Review' is not an automatic deletion decision. Use theory, construct-domain coverage,
#> item wording, and qualitative judge feedback alongside these statistics.
The item table is intentionally diagnostic rather than merely
numeric. A Review flag is not a command to delete an item.
The output reports the strongest competing construct so that researchers
can distinguish weak target correspondence from specific construct
overlap.
summary(fit)
#> Summary of item-sort content-validity evidence
#> -------------------------------------------
#> Retain: 4 of 6 item(s)
#> Review: 2 of 6 item(s)
#>
#> Target-scale evidence:
#> target n_items n_retain n_review mean_psa psa_strength mean_csv csv_strength
#> A 3 2 1 0.783 Moderate 0.567 Moderate
#> B 3 2 1 0.817 Moderate 0.633 Strong
#> overall_strength
#> Moderate
#> Moderate
#>
#> A: Generally supportive normative standing, with at least one dimension in the moderate range; review weaker items before finalizing.
#> B: Generally supportive normative standing, with at least one dimension in the moderate range; review weaker items before finalizing.
#>
#> Items needing attention:
#> item target competitor psa csv p_value
#> A3 A B 0.65 0.3 0.132
#> B3 B A 0.70 0.4 0.058
#> issue recommendation
#> Target favored, exact criterion not met Review
#> Target favored, exact criterion not met Review
#>
#> Interpret scale norms and item flags alongside theory, domain coverage, and qualitative feedback.
#> This analysis does not by itself establish comprehensiveness or the full content-validity argument.
