--- title: "Running a complete workflow" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Running a complete workflow} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4, dpi = 110 ) ``` ## Overview `run_workflow()` chains the data, preprocessing, selection, modelling, evaluation, and result layers into one object. The held-out validation and test sets are transformed with parameters learned from the training set; they are never used to fit preprocessing, selection, calibration, or thresholds. The vignette uses the bundled cardiovascular example cohort. It is deliberately trimmed here so the vignette builds quickly; the full cohort has 2000 rows per data set. ## Build and configure the specification ```{r specification} library(uniml) development <- read.csv( system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml") ) new_admissions <- read.csv( system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml") ) set.seed(123) development <- development[1:800, ] new_admissions <- new_admissions[1:400, ] spec <- uniml_spec( task_type = "binary", outcome = "died", positive = "yes", id = "id" ) spec <- set_preprocess( spec, impute = "median_mode", scale = "zscore", encode = "onehot" ) spec <- set_feature_selection( spec, methods = c("univariate", "lasso"), combine = "intersect" ) spec <- set_learners(spec, c("lr", "cart"), tune = FALSE) spec <- set_evaluation( spec, ci = "bootstrap", threshold_method = "youden", calibration = "platt", select_on = "validation" ) # Kernel SHAP is the most expensive step of a workflow, so its two size knobs are # part of the specification. They are small here to keep the vignette quick; a # paper-quality explanation wants a few hundred rows on both sides. spec <- set_explain(spec, n_explain = 20, bg_n = 20) spec ``` The built-in learner keys are `lr`, `rf`, `xgb`, `nnet`, `knn`, `cart`, and `nb`, plus the optional `log_reg`, `svm`, `lda`, and `qda`. Any installed mlr3 classification learner can also be passed directly, such as `classif.log_reg`, `classif.svm`, or `classif.lda`. Learners from packages that are not on CRAN (for example `classif.lightgbm` from `mlr3extralearners`) work the same way once that package is installed and loaded; external learners use their mlr3 defaults unless a custom search space is registered. ## Create leakage-safe splits `make_split()` returns a `uniml_split` whose elements are `uniml_data` objects. Pass the `$data` component to `run_workflow()`. The independent new-admission cohort stays outside the development split. ```{r split} split <- make_split( as_uniml_data(development, spec), prop = c(0.8, 0.2), stratify = TRUE, seed = 123 ) data.frame( set = c("train", "validation"), n = c(nrow(split$train$data), nrow(split$validation$data)), events = c( sum(split$train$data$died == "yes"), sum(split$validation$data$died == "yes") ) ) ``` ## Run the workflow The first 200 new admissions play the role of an independent test set, the next 200 the role of an external cohort. Both are reporting-only. ```{r workflow} result <- run_workflow( spec, train = split$train$data, validation = split$validation$data, test = new_admissions[1:200, ], external = new_admissions[201:400, ], ci_R = 25 ) result ``` ## Inspect the results `result$metrics` is a wide table with one row per model, set, and probability track (raw or calibrated): ```{r metrics} metrics <- result$metrics metrics[, c("model", "set", "probability", "auc", "brier", "accuracy")] ``` Bootstrap confidence intervals are stored separately so that they can be exported on their own: ```{r ci} head(result$ci[result$ci$set == "validation", ], 4) ``` The ranking selects on the comparison set (validation by default) and never on the test or external cohorts: ```{r ranking} result$ranking[, c("model", "score", "rank", "metric", "set")] ``` Cross-validation output is available per model and metric: ```{r cv} cv_auc <- subset(result$cv$summary, metric == "auc") cv_auc[, c("model", "mean", "sd", "n")] ``` ## Performance figures All plotting helpers accept `palette`, `theme`, `title`, `subtitle`, `caption`, and `set`, so figures can be styled for a manuscript without touching the result. ```{r roc, fig.cap = "ROC curves on the held-out validation set."} plot_roc(result, set = "validation") ``` ```{r calibration, fig.cap = "Calibration on the validation set, raw and calibrated probabilities."} plot_calibration(result, set = "validation") ``` ```{r dca, fig.cap = "Decision curve analysis."} plot_dca(result) ``` ```{r confusion, fig.cap = "Confusion matrix at the Youden threshold."} plot_confusion(result, models = "best") ``` ## Validation, test, and external performance The three held-out sets answer different questions, so they are reported separately and never pooled. Model selection uses the validation set only; the test set and the external cohort are reporting-only. ```{r role-table} role_table <- subset( result$metrics, set %in% c("validation", "test", "external"), select = c("model", "set", "probability", "auc", "brier", "sensitivity", "specificity") ) role_table[role_table$probability == "calibrated", ] ``` Discrimination on the external cohort, which took no part in training, feature selection, or calibration: ```{r external-roc, fig.cap = "ROC curves on the external cohort."} plot_roc(result, set = "external") ``` Comparing the calibration curves of the validation and external cohorts shows whether the probabilities travel; a model that only looks good on the development split is usually overfitted to it. ```{r external-calibration, fig.cap = "Calibration on the external cohort."} plot_calibration(result, set = "external") ``` ## Thresholds, reclassification, and explainability The classification threshold is derived from out-of-fold training predictions, so it is not tuned on the validation data: ```{r thresholds} as.data.frame(result$meta$thresholds) ``` When two or more models are compared, NRI and IDI are computed against the reference model (the best training AUC by default): ```{r nri} if (!is.null(result$tests$nri_idi)) { result$tests$nri_idi$summary } ``` SHAP explanations use `kernelshap`; the number of explained rows and the background sample are controlled by `n_explain` and `bg_n`: ```{r shap, fig.cap = "Global SHAP importance of the best model."} explained <- explain_shap(result, model = "best", n_explain = 30, bg_n = 20) plot_shap_importance(explained) ``` ## Predict new data `predict_uniml()` reuses the stored preprocessing, selected feature mapping, calibration, threshold, and fitted learner. It does not refit anything. ```{r predict} predicted <- predict_uniml( result, newdata = new_admissions[1:5, ], model = "best" ) predicted[, c("model", "prob", "calibrated_prob", "probability", "predicted")] ``` The `newdata` argument may omit the outcome column. In that case the returned data frame contains predicted probabilities and classes without a truth column. ## Reporting checklist `tripod_ai_check()` audits the result against the TRIPOD+AI domains and states where each piece of information lives. It never claims compliance: items such as study design, subgroup fairness, and code availability must be written by the author. ```{r tripod} checklist <- tripod_ai_check(result) table(checklist$status) subset(checklist, status != "reported", select = c("domain", "status")) ``` ## Export results `export_results()` writes the tables, figures, and fitted models, plus the reporting checklist, to a directory that can be used outside R: ```{r export} out <- tempfile("uniml-vignette-") exported <- export_results(result, dir = out, include_plots = FALSE) basename(exported$files) ``` ## Leakage policies The default is a transparent custom policy. It can be changed explicitly: ```{r leakage} spec_in <- set_leakage_policy( spec, mode = "strict_in", in_cv = c("selection", "balance", "tuning") ) spec_in$leakage_policy spec_in$in_cv ``` `strict_in` repeats configured steps inside each cross-validation fold. `strict_out` keeps those steps outside the fold loop. The validation and test sets remain held out in both cases. ## References See `system.file("REFERENCES.md", package = "uniml")` for the methods and reporting guidelines that this workflow implements.