--- title: "Multiclass outcomes" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Multiclass outcomes} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4, dpi = 110 ) ``` ## What "multiclass" means here A multiclass task predicts one of three or more classes that carry no order. `uniml` treats it as a separate task type rather than as a binary problem with a positive class bolted on, because almost every binary report is built around a single event: a threshold, a positive class, a calibration curve, a decision curve. None of those survives contact with four unordered classes without a statement of convention. What the package fixes for you is the convention: | Quantity | Definition | | --- | --- | | `auc` | Hand and Till multiclass AUC: the mean over ordered class pairs of the AUC on class `i`'s own score column (`mlr3measures::mauc_au1u`) | | `auc_ovr` | Macro one-vs-rest AUC: each class against the rest, then the unweighted mean; equals `mean(per_class$auc)` | | `micro_auc` | Micro-average AUC: all (patient, class) pairs pooled into one binary problem | | `brier` | Multi-class Brier score: mean over rows of the summed squared class errors | | `logloss` | Mean negative log probability of the observed class | | `sensitivity`, `specificity`, `ppv`, `npv`, `f1` | Macro averages of the one-vs-rest values | | `micro_specificity` | Specificity pooled over the one-vs-rest cells | | `balanced_accuracy` | Mean per-class recall, which equals the macro sensitivity | | `kappa` | Cohen's kappa on the k-by-k table | There is no micro precision, micro recall, or micro F1. For a single-label multiclass outcome all three are identically the accuracy — the same number three times — so the package reports accuracy once and says why. Micro AUC and micro specificity are the two micro numbers that carry information of their own. Every number is reproducible from the confusion matrix that comes with it, and `tests/testthat/test-multiclass.R` pins them to `mlr3measures` and to a direct recomputation. ## A worked example `datasets::iris` is used here because it ships with R and has three classes; the mechanics are the same for a clinical outcome with three risk groups. ```{r data} library(uniml) data <- iris data$Species <- factor(data$Species) table(data$Species) spec <- uniml_spec( task_type = "multiclass", outcome = "Species" ) spec <- set_learners(spec, c("cart", "rf"), tune = TRUE) spec ``` Note there is no `positive`: a positive class only exists when there are two. ```{r task} prepared <- as_uniml_data(data, spec) levels(prepared$outcome) task <- make_task(data, spec) task$class_names task$positive ``` ## Train and score ```{r train} learner <- train_model( task, mlr3::lrn("classif.rpart", predict_type = "prob") ) prediction <- learner$predict(task) head(prediction$prob, 3) ``` `calc_metrics()` recognises a prediction with more than two probability columns and switches to the multiclass metric set: ```{r metrics} metrics <- calc_metrics(prediction) metrics[setdiff(names(metrics), c("confusion", "per_class"))] ``` The confusion matrix has the reference classes in rows and the predicted classes in columns, and the per-class table turns it into one-vs-rest rates: ```{r confusion} metrics$confusion metrics$per_class ``` The macro sensitivity and the balanced accuracy are the same quantity computed two ways, which is a useful check rather than a duplication: ```{r macro} data.frame( sensitivity = metrics$sensitivity, balanced_accuracy = metrics$balanced_accuracy, accuracy = metrics$accuracy, kappa = metrics$kappa ) ``` ## Does a model survive resampling? Nothing here is special-cased for the workflow, so ordinary mlr3 resampling applies; `calc_metrics()` scores the pooled out-of-fold predictions. ```{r cv} resampling <- mlr3::rsmp("cv", folds = 5L) resampling$instantiate(task) cv <- mlr3::resample(task, learner, resampling, store_models = FALSE) pooled <- as_uniml_multiclass(cv$prediction(), truth = cv$prediction()$truth) cv_metrics <- calc_metrics(pooled) round(cv_metrics$auc, 3) round(cv_metrics$accuracy, 3) ``` Tuning works the same way as for a binary task, because the search space and the resampling are mlr3 objects. `resolve_learners()` returns the package's learner specifications, which carry their search spaces: ```{r tune} learners <- resolve_learners(spec, task = task) tuned <- tune_model( task, learners[["cart"]], resampling = mlr3::rsmp("cv", folds = 3L), tuner = "random_search", terminator = mlr3tuning::trm("evals", n_evals = 4L), seed = 123 ) tuned$status tuned$best_params ``` ## Comparing models `rank_models()` averages a composite score whose metric set is declared by the task type, so a multiclass comparison is not ranked on two-class numbers. The multiclass composite is AUC, accuracy, balanced accuracy, and macro F1. ```{r rank} score_row <- function(model, prediction) { values <- calc_metrics(prediction) data.frame( model = model, set = "validation", auc = values$auc, accuracy = values$accuracy, balanced_accuracy = values$balanced_accuracy, f1 = values$f1, stringsAsFactors = FALSE ) } forest <- train_model(task, mlr3::lrn("classif.ranger", predict_type = "prob")) comparison <- rbind( score_row("cart", prediction), score_row("rf", forest$predict(task)) ) comparison rank_models(comparison, task_type = "multiclass") ``` A plain metric data frame carries no task type, so `task_type` has to be passed; a `uniml_result` remembers it for you. Ranking a binary result is unchanged. ## Figures Discrimination is shown as one-vs-rest ROC curves, one per class. There is no single ROC curve for a multiclass problem, and averaging them into one line hides exactly the class you are worried about. ```{r roc, fig.cap = "One-vs-rest ROC curves."} binary_prediction <- as_uniml_multiclass(prediction) plot_roc(binary_prediction) ``` ```{r confusion-plot, fig.cap = "Confusion matrix: reference classes in rows, predicted in columns."} plot_confusion(binary_prediction) ``` ## Calibration A multiclass model outputs a whole vector of probabilities for every patient, so "is the number right" has to be asked in two ways. Both are reported, because a model can pass one and fail the other. ```{r calibration} calibration <- multiclass_calibration(prediction$prob, prediction$truth) calibration$summary calibration$per_class ``` * **Top-label** calibration keeps the largest probability of each row as the confidence and asks whether patients with 0.8 confidence really belong to the predicted class 80% of the time. Its numbers are `ece` (expected calibration error) and `mce` (the worst bin). * **Class-wise** calibration repeats a one-vs-rest comparison for every class and averages the per-class ECEs without weighting, so a rare class that is badly wrong cannot hide behind the common ones. That is `ece_classwise`. * `level_bias` is `mean confidence - accuracy`. It is the closest thing to the binary intercept: the multinomial model has no single slope to report, and its analogue (temperature scaling) is a recalibration method rather than a diagnostic. ECE and MCE are bin-dependent summaries, not proper scoring rules; rank models on the Brier score or the log loss, and read ECE as "how wrong is the stated probability when the model is wrong". ```{r calibration-plot, fig.cap = "Reliability of the multiclass predictions: top-label confidence (left) and one-vs-rest per class (right)."} plot_calibration(binary_prediction, scope = "both") ``` The same numbers ride along with every other metric, so they can be tabulated and bootstrapped without a second call: ```{r calibration-metrics} metrics <- calc_metrics(prediction) round(metrics$ece, 3) round(metrics$ece_classwise, 3) metrics$per_class[, c("class", "ece", "mce")] ``` ## Recalibration Measuring the miscalibration and fixing it are two different jobs. The method for a multinomial model is temperature scaling: divide every logit by one fitted scalar, `softmax(log(p) / T)`. `T > 1` softens an over-confident model, `T < 1` sharpens an under-confident one. ```{r recalibrate} recalibrated <- calibrate_probabilities(binary_prediction, folds = 5L) recalibrated$parameters$fitted recalibrated$metrics_before[, c("accuracy", "ece", "logloss")] recalibrated$metrics_after[, c("accuracy", "ece", "logloss")] ``` Two properties are worth knowing before reading those numbers. The division is monotone and is applied to every class in the same way, so **temperature scaling cannot change which class the model ranks first**. The confusion matrix, the accuracy, and every threshold-based metric come out identical; only the probabilities attached to them move. A method that can only make the model better at *reporting* and cannot make it worse at *deciding* is worth having even though it has one parameter. The fitted scalar is chosen to be no worse than leaving the model alone, and the `calibrated` matrix is cross-fitted, so the before/after table is measured on probabilities that were not used to fit the scalar. That honesty has a cost: on a model that was already calibrated the cross-fitted log loss can come out a hair worse, because each fold's scalar is a noisy estimate of 1. ```{r recalibrate-plot, fig.cap = "The same predictions after temperature scaling."} plot_calibration( as_uniml_multiclass(recalibrated$calibrated, truth = binary_prediction$truth), scope = "top_label" ) ``` `calibrate_probabilities()` takes `new_probs` (and optionally `new_labels`) so a scalar fitted on the validation set can be applied to the test set. The job never crosses sets by accident: the scalar is a property of the model, and the before/ after table is reported on whichever set you hand it. ## What is deliberately missing These are not gaps to be filled in later without a decision; each needs a convention that a package should not choose silently. * **Dirichlet calibration.** The stronger multiclass recalibration (Kull et al. 2019) fits `softmax(W log p + b)`, which is `K^2 + K` parameters, can change the predicted class, and is not identifiable on the few hundred validation patients a clinical model usually has. Temperature scaling is the part of the idea that such a sample can support, and it is what `calibrate_probabilities()` does. * **Decision-curve analysis.** Net benefit compares one event against no event. Run one class against the rest with `as_uniml_binary()` and `dca()` if that is the question you are asking. * **NRI / IDI, DeLong, Hosmer-Lemeshow.** All three are defined for a single positive class. They are not implemented for `uniml_multiclass`. * **`run_workflow()`.** The full workflow is binary-only for now: its prediction table, thresholds, calibrators, and post-model tests all assume one probability column. Calling it on a multiclass specification stops with a message pointing back at this vignette. The layers underneath it are not binary-only any more — the task, the metrics, the calibration, and the composite ranking are all declared by the task type — which is why the pieces above can be combined by hand. ## References Hand DJ, Till RJ (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. *Machine Learning*, 45, 171-186. doi:10.1023/A:1010920819831 Sokolova M, Lapalme G (2009). A systematic analysis of performance measures for classification tasks. *Information Processing & Management*, 45(4), 427-437. doi:10.1016/j.ipm.2009.03.002 Cohen J (1960). A coefficient of agreement for nominal scales. *Educational and Psychological Measurement*, 20(1), 37-46. doi:10.1177/001316446002000104 Brier GW (1950). Verification of forecasts expressed in terms of probability. *Monthly Weather Review*, 78(1), 1-3. Guo C, Pleiss G, Sun Y, Weinberger KQ (2017). On calibration of modern neural networks. *Proceedings of the 34th International Conference on Machine Learning*, 70, 1321-1330. Kull M, Perello-Nieto M, Kängsepp M, Silva Filho T, Song H, Flach P (2019). Beyond temperature scaling: obtaining well-calibrated multi-class probabilities with Dirichlet calibration. *Advances in Neural Information Processing Systems*, 32. See `system.file("REFERENCES.md", package = "uniml")` for the full list.