--- title: "A real cohort: subarachnoid haemorrhage" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{A real cohort: subarachnoid haemorrhage} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4, dpi = 110 ) ``` ## Why this vignette Every other example in this package uses a cohort that `data-raw/` simulates. That makes the code reproducible, but it says nothing about how the workflow behaves on real data, where probabilities are lumpy, groups can be empty, and a hundred and thirteen patients is all you have. This vignette runs the same workflow on a published clinical data set that ships with `pROC`, so the numbers below are real and you can check them against the source paper. ## The data `pROC::aSAH` contains 113 patients with aneurysmal subarachnoid haemorrhage. The outcome is the dichotomised Glasgow Outcome Scale at six months, and the candidate predictors are two blood markers (`s100b`, `ndka`) plus age, sex, and the World Federation of Neurosurgical Societies grade. ```{r data} asah <- pROC::aSAH str(asah[, c("outcome", "gos6", "gender", "age", "wfns", "s100b", "ndka")]) table(asah$outcome) ``` Note `gos6`: it is the five-level version of the outcome, and `outcome` is exactly `gos6` collapsed to two groups. ```{r leakage} table(asah$gos6, asah$outcome) ``` Leaving `gos6` in the predictor set would let the model read the answer off the answer sheet. It goes in `exclude`, and the specification refuses to use it: ```{r spec} library(uniml) spec <- uniml_spec( task_type = "binary", outcome = "outcome", positive = "Poor", exclude = "gos6" ) spec <- set_preprocess( spec, impute = "median_mode", scale = "zscore", encode = "onehot" ) spec <- set_feature_selection( spec, methods = "univariate", p_threshold = 0.2 ) spec <- set_learners(spec, c("lr", "cart"), tune = FALSE) spec <- set_tuning(spec, folds = 5, repeats = 5, evals = 4, metric = "auc", seed = 123) spec <- set_evaluation( spec, ci = "bootstrap", calibration = "platt", threshold_method = "youden", select_on = "validation" ) spec <- set_leakage_policy(spec, mode = "strict_in") spec <- set_explain(spec, model = "none", plots = character()) spec ``` `strict_in` re-runs the univariate selection inside every fold, so the cross-validated numbers do not profit from having seen the whole training set. With five predictors that costs almost nothing. ## Split and run Seventy per cent of the cohort develops the models, thirty per cent is held out for the whole workflow. ```{r split} split <- make_split( as_uniml_data(asah, spec), prop = c(0.7, 0.3), stratify = TRUE, seed = 123 ) data.frame( set = c("train", "validation"), n = c(nrow(split$train$data), nrow(split$validation$data)), poor_outcome = c( sum(split$train$data$outcome == "Poor"), sum(split$validation$data$outcome == "Poor") ) ) ``` ```{r run} result <- suppressWarnings(run_workflow( spec, train = split$train$data, validation = split$validation$data, ci_R = 40 )) result ``` Nothing was dropped, downgraded, or silently skipped: ```{r failures} result$meta$failures ``` The chunks above suppress warnings. The ones that fired are expected on data this small: bootstrap resamples in which the positive likelihood ratio and the diagnostic odds ratio are infinite, and a Platt fit on tree probabilities that are exactly 0 or 1. Neither is an error, and both are visible in the output below — the affected intervals come back as `NA` rather than as a number invented by the package. ## What the cross-validation says Repeated cross-validation on the 80 development patients is the most stable estimate available here. ```{r cv} cv_auc <- subset(result$cv$summary, metric == "auc") cv_auc[, c("model", "mean", "sd", "lower", "upper", "n")] ``` `n = 5` is the number of repeats: each repeat averages five folds, and the interval summarises spread across those five repeat means. It is narrower than the sampling uncertainty of the AUC itself, so do not read it as a confidence interval for new patients — compare it with the bootstrap interval on the validation patients further down, which is much wider. ## Validation performance The held-out set has 33 patients and 12 events. ```{r metrics} validation <- subset( result$metrics, set == "validation", select = c("model", "probability", "auc", "brier", "sensitivity", "specificity", "ppv", "npv") ) validation ``` Bootstrap intervals around the same numbers: ```{r ci} ci <- subset( result$ci, set == "validation" & metric %in% c("auc", "brier"), select = c("model", "probability", "metric", "mean", "lower", "upper") ) ci ``` The logistic model calibrates better than the tree: Platt scaling cuts its validation Brier score by about 0.019, and `cart`'s by about 0.014. Reads of a single 33-patient split should be treated as illustrative; the cross-validated table above is what to quote. Two of the ratios cannot be given an interval on this data, and the package says so rather than inventing bounds: ```{r ci-undefined} subset( result$ci, set == "validation" & metric %in% c("plr", "dor"), select = c("model", "metric", "mean", "lower", "upper", "n_effective") ) ``` `lr` calls every validation patient at low risk, so its specificity is exactly 1 and the positive likelihood ratio is infinite in all 200 resamples. The interval comes back as `NA` with `n_effective = 0`. `cart` keeps a finite ratio and therefore a finite interval. ## Which model does the package rank first? `rank_models()` scores each model on the comparison set (validation by default) using the calibrated probabilities, averaging AUC, sensitivity, specificity, F2, balanced accuracy, and Youden's index. ```{r ranking} result$ranking[, c("model", "score", "rank", "metric", "set")] ``` `cart` comes first. It has the lower AUC but the much higher sensitivity at the Youden threshold, and the composite weights the two equally. That is a choice, not a law: pass `best_model_metric = "auc"` to `rank_models()` if discrimination is all you care about, and pick deliberately rather than by default. ## Figures ```{r roc, fig.cap = "ROC curves on the held-out validation patients."} plot_roc(result, set = "validation") ``` ```{r calibration, fig.cap = "Calibration on the validation set, before and after Platt scaling."} plot_calibration(result, set = "validation") ``` ```{r dca, fig.cap = "Decision curve analysis."} plot_dca(result) ``` ```{r confusion, fig.cap = "Confusion matrix at the out-of-fold Youden threshold."} plot_confusion(result, models = "best") ``` ## Thresholds The classification threshold comes from out-of-fold training predictions, never from the validation patients. ```{r thresholds} as.data.frame(result$meta$thresholds) ``` ## Do the two models differ? ```{r delong} result$tests$delong$summary result$tests$delong$formatted ``` The DeLong test compares AUCs on the same patients and accounts for the correlation between the two prediction vectors. With 12 events it has very little power, so a non-significant p value means "not shown to differ", not "equivalent". ## Calibration, for the record ```{r hl} for (track in c("raw", "calibrated")) { for (model in c("lr", "cart")) { item <- result$tests$hl[[track]][[model]] cat(sprintf( "%-10s %-5s statistic = %s, df = %s, p = %s (%s)\n", track, model, format(round(item$statistic, 3)), format(item$df), format(round(item$p_value, 3)), item$method )) } } ``` `cart` reports `not_estimable` on the raw track: the tree puts a group of patients at a probability of exactly 1, which leaves the expected non-event count at zero and the Hosmer-Lemeshow statistic undefined. The package says so with a message instead of returning `NaN`; the fix in practice is fewer groups or more patients. After Platt scaling the same model has spread-out probabilities and the statistic becomes estimable, which is a fair summary of what calibration does. ## Reclassification ```{r nri} subset( result$tests$nri_idi, set == "validation", select = c("model", "reference", "nri", "nri_events", "nri_nonevents", "idi") ) ``` NRI and IDI are computed against the reference model (the best training AUC, `cart` here). The result is worth reading carefully: `lr` reclassifies none of the events and 71% of the non-events downward, so its net reclassification is negative even though its AUC on the same patients is higher. With twelve events, neither number is evidence of anything — but the disagreement is exactly why AUC alone is a poor basis for choosing a risk model. ## Reporting checklist ```{r tripod} checklist <- tripod_ai_check(result) table(checklist$status) subset(checklist, status != "reported", select = c("domain", "status")) ``` ## Export ```{r export} out <- tempfile("uniml-asah-") exported <- export_results(result, dir = out, include_plots = FALSE) basename(exported$files) ``` ## What this vignette does and does not show It shows that the workflow runs end to end on real, complete clinical data, that the leakage guard is not hypothetical (`gos6` would have made every model look brilliant), and that the reporting layer degrades honestly when the data is too small for a statistic to be defined. It does not show a validated model. A hundred and thirteen patients cannot support one, and the package is not a substitute for a prospective validation study. If you report from a cohort this size, report the cross-validated estimates with their intervals, say how wide they are, and do not select between models on thirty-three patients. ## References Sabbagh C, et al. (2005). Protein S100B and neuron-specific enolase in subarachnoid haemorrhage. The `aSAH` data set is distributed with the `pROC` package and used here under its licence. See `system.file("REFERENCES.md", package = "uniml")` for the methods that each layer implements.