---
title: "Getting started with coreval"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with coreval}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

If you build SDTM datasets, you know the loop: write code, export, upload to a
validation tool, wait, read the report, work out which line of code caused each
finding, fix, repeat.

Hardly any of that time is spent *fixing* things. It goes on **finding out
what's broken**.

coreval does the finding part on your machine, in seconds. You still run your
qualified tool before you submit — you just arrive with a lot less for it to
find.

> coreval is a personal open-source project. It's not a CDISC product, isn't
> affiliated with or endorsed by CDISC, and isn't qualified or validated
> software. Treat every result as a hint, not a verdict. Your qualified tool
> and your own review are still what decide whether data is good to go.

```{r setup}
library(coreval)
```

## Start with one dataset

This is the one you'll use most while writing code. You have a data frame;
check it.

Note row 2: `2024-02-30`. February never has 30 days.

```{r one-dataset}
ae <- data.frame(
  STUDYID = "DEMO", DOMAIN = "AE", USUBJID = c("S1", "S1", "S2"),
  AESEQ = c(1, 2, 1), AETERM = c("Headache", "Nausea", "Rash"),
  AESTDTC = c("2024-01-10", "2024-02-30", "2024-01-12"),
  AEENDTC = c("2024-01-12", "2024-02-01", "")
)

result <- check_dataset(ae)
result
```

Each problem is described in words, with the rows and values that caused it, and
the rule number at the end in case you want to look it up.

The tag on each problem is worth understanding. CDISC Open Rules carry no
severity field - Pinnacle 21's Notes/Minor/Major/Critical is P21's own layer,
not CDISC's - so coreval does not report one and will not invent one. What it
does instead is separate the findings that are definitely wrong from the ones
that may be fine:

* `wrong value` - the data contains something that breaks the rule, like a
  month of 13. Nothing about your study explains it away. Start here.
* `missing required` - something the standard marks Required is absent.
* `missing optional` - something Expected is absent, or a value is blank. Often
  legitimate: a screen-failure subject with no reference dates, a variable your
  raw data does not carry yet.

Problems are ordered by that first and by how many records they touch second,
and within a problem the record holding a real offending value is shown before
one that is merely empty. It is also a `triage` column on every finding, so you
can sort a spreadsheet by it.

When you want the rows themselves — to filter or count — they are in
`result$findings`, with the same description in an `issue` column:

```{r one-dataset-date}
result$findings[result$findings$Value == "2024-02-30", ]
```

You can pass a file instead of a data frame — `.xpt`, `.sas7bdat` or `.csv`:

```{r one-dataset-file, eval = FALSE}
result <- check_dataset("ae.xpt")
```

coreval works out the domain from your `DOMAIN` column, and falls back to the
file name only when the data has no `DOMAIN` column at all. That order matters
for a split dataset: `ae1.xpt` is checked as `AE` because its `DOMAIN` column
says `AE` — on a file with no `DOMAIN` column the name `ae1` is taken at face
value. If it guesses wrong, just say so: `check_dataset(ae, domain = "AE")`.

### What it couldn't check, and why that matters

Lots of CDISC rules compare **one dataset against another** — an adverse event
date against the subject's reference dates in `DM`, a visit against the trial
design. Give coreval a single dataset and those questions simply can't be
answered.

coreval won't guess. It skips them and tells you what it wanted:

```{r one-dataset-skips}
cross <- result$skipped[grepl("was not supplied", result$skipped$reason), ]
nrow(cross)
head(unique(cross$reason), 3)
```

If it ran those anyway, it would be comparing your data against columns that
aren't there, and reporting problems that don't exist. Saying nothing is better
than making something up.

The same honesty applies to a limitation you should know about up front: 9 rules
need CDISC's controlled terminology — the codelists saying which values are
legal for `SEX`, `AEOUT` and the like. Those lists run to roughly 438 MB, far
too much to bundle, so this release does not ship them. Rules that need them are
reported as skipped, by name, with that reason, and are never counted as passing.

Most rules do still run — across AE, DM, LB and VS, 76–84% of the applicable
ones work on a single dataset. But the ones that can't are the cross-dataset
checks, and those are often the ones you care about.

**So a short findings list here doesn't mean your data is clean.** It's a quick
first pass, not a verdict.

## Then check the whole study

Once the datasets exist as files, point coreval at the folder. Here's a small
one, built on the fly so this vignette runs without any data of your own:

```{r build-study}
dir <- tempfile("coreval_demo_")
dir.create(dir)

dm <- data.frame(
  STUDYID = "DEMO", DOMAIN = "DM", USUBJID = c("S1", "S2", "S3"),
  RFSTDTC = c("2024-01-05", "2024-01-06", ""),
  AGE = c(34, 61, 47), AGEU = c("YEARS", "YEARS", ""),
  SEX = c("M", "F", "F")
)

haven::write_xpt(dm, file.path(dir, "dm.xpt"))
haven::write_xpt(ae, file.path(dir, "ae.xpt"))
```

Point it at the **folder**, not a file:

```{r check}
study_result <- check_study(dir)
```

coreval reads everything in there, and reading it all at once is the point —
now the cross-dataset rules have both halves to work with. If there's a
Define-XML in the folder it finds it and uses it (that needs the `xml2`
package).

If you want to look at what was parsed, or check the same large study more than
once without re-reading it, do the read yourself and pass the object instead:

```{r read}
study <- read_study(dir)
names(study$datasets)
```

## Reading the results

You get two tables back, and you want to look at both.

### What's wrong

```{r findings-head}
head(study_result$findings)
```

One row per problem, pointing straight at it:

| Column | What it tells you |
|---|---|
| `Dataset` | which dataset, or `STUDY` for whole-study checks |
| `Record` | row number, counting from 1 |
| `Variable` | the variable being complained about |
| `Value` | what was actually in there |
| `issue` | what's wrong, in words |
| `triage` | `wrong value`, `missing required` or `missing optional` |
| `rule_id` | the CDISC rule, if you want to look it up |

One thing that surprises people: `Not in dataset` under `Value` means the rule
wanted a variable you don't have — which is usually exactly the finding.

It's a plain data frame, so slice it however you like:

```{r findings-filter}
f <- study_result$findings
head(f[f$Dataset == "DM", ])
sort(table(f$rule_id), decreasing = TRUE)[1:3]
```

### What couldn't be checked

```{r skipped}
head(study_result$skipped)
```

**This is the table people skip, and it's the one that bites.** An empty
findings table means one of two things: your data is clean, or a lot of rules
never ran. Those look identical if you only read the findings. coreval always
shows you both, with a reason for every rule it couldn't run.

### Saving it, and tracking what you didn't fix

```{r export, eval = FALSE}
write_findings(study_result, "issues.xlsx")  # one workbook, a sheet per table
write_findings(study_result, "issues.csv")   # issues.csv + _skipped + _about
```

Both tables get written every time, for the reason just above. Excel output
needs the `writexl` package.

The saved file has three empty columns — `Status`, `Owner` and `Notes` — for you
to fill in once it is open. Not every finding is something you will fix: some are
expected, some belong to someone else, some are waiting on a data query. Those
decisions are worth recording next to the finding rather than in a separate
document. Pass `tracking = FALSE` if you would rather not have them.

## The rule set

One function answers every question about the rule set.

```{r rules}
rules <- list_rules()
nrow(rules)
table(rules$source)

attr(rules, "rules_version")
```

That last one is the exact CDISC commit the bundled rules came from. Worth
writing down next to your results — though `write_findings()` already records
it in every file it saves.

Not every rule carries the same weight. `source` separates fully-vetted
published rules from deprecated and draft ones; `?list_rules` says what each
means.

Note the count above includes the deprecated ones. `list_rules()` is the
catalog of what is *bundled*; a check excludes superseded rules, because
running one alongside its replacement reports the same problem twice. Listing
is not running.

Ask it what applies to a domain, or what a rule the report named actually
checks:

```{r rules-for-domain}
nrow(list_rules(domain = "AE"))

list_rules(id = "CORE-000547")$issue
```

## The whole API

Six functions, and three of them do the work:

| | |
|---|---|
| `check_dataset(x)` | one dataset — a data frame, or an `.xpt`/`.sas7bdat`/`.csv` |
| `check_study(path)` | a whole folder |
| `write_findings(result, path)` | save to Excel or CSV |
| `list_rules()` | the rule set, one rule, or the rules for a domain |
| `filter_findings(result, ...)` | narrow a result |
| `read_study(path)` | read a folder yourself, when you want to inspect it |

Plus `print()` and `summary()` on a result, which you get by typing the
result's name.

## Two last things

**Nothing leaves your machine.** No internet, no API key, no account. The rules
and the standards metadata are bundled inside the package.

**Think of the accuracy number as a floor.** For most rules CDISC publishes an
answer sheet: some example data, and the exact rows a correct implementation
should flag. Every rule here is run against those examples and compared row by
row, and coreval agrees on about 96% of the fully published ones.

But those examples are small and tidy, and real submissions are neither, so
agreement is a lower bound on correctness rather than a score. The README says
where the remaining gaps are. The advice stays the same either way: run your
qualified tool before you submit.

```{r cleanup, include = FALSE}
unlink(dir, recursive = TRUE)
```