Help for package quallmer.app

Title:

Interactive Validation App for 'quallmer'

Version:

0.1.0

Description:

Companion package to 'quallmer' providing an interactive 'shiny' application for manual coding, reviewing large language model (LLM) generated annotations, and computing inter-rater reliability metrics. Supports three modes: blind manual coding, LLM output validation, and agreement calculation. Computes standard reliability metrics including Krippendorff's alpha (Krippendorff 2019 <doi:10.4135/9781071878781>), Cohen's kappa, Fleiss' kappa (Fleiss 1971 <doi:10.1037/h0031619>), intraclass correlation coefficient (ICC), and percent agreement for nominal, ordinal, interval, and ratio data. Also computes gold-standard validation metrics including accuracy, precision, recall, and F1 scores following Sokolova and Lapalme (2009 <doi:10.1016/j.ipm.2009.03.002>).

License:

MIT + file LICENSE

Encoding:

UTF-8

RoxygenNote:

7.3.3

Imports:

quallmer (≥ 0.3.0), shiny, bslib, dplyr, tidyr, irr, htmltools, cli, stats, utils

Suggests:

testthat (≥ 3.0.0)

Config/testthat/edition:

URL:

https://github.com/quallmer/quallmer.app

BugReports:

https://github.com/quallmer/quallmer.app/issues

Depends:

R (≥ 3.5)

LazyData:

true

NeedsCompilation:

Packaged:

2026-03-04 07:54:41 UTC; smaerz

Author:

Seraphine F. Maerz

[aut, cre], Kenneth Benoit

[aut]

Maintainer:

Seraphine F. Maerz <seraphine.maerz@unimelb.edu.au>

Repository:

CRAN

Date/Publication:

2026-03-08 10:40:02 UTC

Launch the Quallmer Interactive App

Description

Starts the Shiny app for manual coding, LLM checking, and validation / agreement calculation.

Usage

qlm_app(base_dir = getwd())

Arguments

base_dir

Base directory for saving uploaded files and progress. Defaults to current working directory. Use tempdir() for temporary storage (e.g., in examples or tests), but note that data will be lost when the R session ends.

Details

In LLM mode, you can also select metadata columns.
In Validation mode, select unit ID and coder columns (no text column), and optionally specify a gold-standard coder.

Value

A shiny.appobj

Examples

if (interactive()) {
  # Launch the app
  qlm_app()

  # Use a temporary directory (useful for testing)
  qlm_app(base_dir = tempdir())
}

Sample Dataset for quallmer app

Description

A sample dataset for demonstrating the quallmer app's validation functionality. Contains movie review texts coded by multiple coders with a gold standard.

Usage

sample_data

Format

A data frame with 20 rows and 6 variables:

.row_id: Unique identifier for each text
text: Movie review text
gold_sentiment: Gold standard sentiment label (positive, negative, neutral)
coder1_sentiment: Human coder's sentiment classification
coder2_sentiment: LLM coder's sentiment classification (moderate accuracy)
coder3_sentiment: Another LLM coder's sentiment classification (lower accuracy)

Details

This dataset is useful for:

Testing inter-rater reliability calculations (using coder1, coder2, coder3)
Testing gold-standard validation (using gold_sentiment as reference)
Learning how to use the quallmer app
Demonstrating nominal measurement level metrics

Examples

if (interactive()) {
  # Option 1: Use the pre-made sample file from the package
  # Get the path to the sample data file
  sample_file <- system.file("extdata", "sample_data.rds", package = "quallmer.app")

  # Launch the app and upload this file through the UI
  qlm_app()

  # Option 2: Load the data and save your own copy
  data(sample_data)
  saveRDS(sample_data, "my_sample.rds")
  # Then load my_sample.rds in qlm_app()
}