clean_the_nest(): Standardising Surveillance Data

What `clean_the_nest()` does

clean_the_nest() is the foundational layer of the mudnester pipeline. It:

Renames your columns to the mudnester internal schema (standardised names like lettername1, dob, onset_date) so every downstream function knows what to expect
Validates date formats and raises a clear error if a date column is not Date class
Strips and lowercases name fields for linkage consistency
Derives blocking variables (block1, block2, block3) used by starling::murmuration()
Standardises Medicare numbers into 9-, 10-, and 11-digit variants
Derives age, length of stay, ICU outcome, and death outcome automatically when the required columns are present
Pivots long vaccination data to wide (one row per person) via lie_nest_flat = TRUE

Nothing downstream is trustworthy until this layer is structurally sound. Think of it as laying the mud — subsequent functions can only build on what is solid here.

The three data types

data_type is required. The three values map to different source systems in a Queensland public health context.

`data_type`	Typical source	Key mandatory dates
`"cases"`	NNDSS, NoCS, EDIS linelists	`onset_date`
`"hospital"`	iPM, HBCIS, admitted patient collections	`admission_date`
`"vaccination"`	Australian Immunisation Register (AIR), local registers	`vax_date`

Age and age categories are derived for "cases" and "hospital" when dob and onset_date / admission_date are supplied. They are not derived for "vaccination" (the AIR does not reliably carry age at time of vaccination).

Worked examples

Cases (notifiable disease linelist)

set.seed(1)
cases_raw <- data.frame(
  identity          = paste0("PT", 1:20),
  first_name        = sample(c("James","Sarah","Michael"), 20, TRUE),
  surname           = sample(c("Smith","Jones","Williams"), 20, TRUE),
  date_of_birth     = as.Date("1970-01-01") + sample(-5000:5000, 20),
  date_of_onset     = as.Date("2024-03-01") + sample(0:120, 20),
  disease_name      = sample(c("COVID-19","Influenza A","RSV"), 20, TRUE),
  gender            = sample(c("M","F"), 20, TRUE),
  postcode          = sample(c("4556","4557","4560"), 20, TRUE),
  medicare_no       = paste0(sample(2000:9999, 20), sample(10000:99999, 20)),
  indigenous_status = sample(c("Non-Indigenous","Aboriginal","Unknown"), 20, TRUE,
                              prob = c(0.85, 0.10, 0.05)),
  stringsAsFactors  = FALSE
)

df_cases <- clean_the_nest(
  cases_raw,
  data_type   = "cases",
  drop_eggs   = TRUE,
  id_var      = "identity",
  diagnosis   = "disease_name",
  lettername1 = "first_name",
  lettername2 = "surname",
  dob         = "date_of_birth",
  medicare    = "medicare_no",
  gender      = "gender",
  postcode    = "postcode",
  fn          = "indigenous_status",
  onset_date  = "date_of_onset"
)

head(df_cases[, c("lettername1","lettername2","dob","age","diagnosis","block1")], 4)
#>   lettername1 lettername2        dob  age   diagnosis      block1
#> 1       james    williams 1965-10-29 58.7    COVID-19 M 4556 1965
#> 2     michael       smith 1961-08-23 62.8 Influenza A F 4557 1961
#> 3       james       smith 1963-05-17 61.0    COVID-19 M 4556 1963
#> 4       sarah       smith 1960-07-01 63.9    COVID-19 M 4560 1960

Notice that lettername1 is lowercased, punctuation-stripped, and truncated to the first name token only. block1 is gender + postcode + birth_year — used as the primary blocking variable in starling::murmuration().

Hospital admissions

When admission_date and discharge_date are both supplied, length of stay (los, in days) is derived automatically. admission_outcome is a factor indicating whether the person was actually admitted.

set.seed(2)
hosp_raw <- data.frame(
  patient_id         = paste0("UR", 1:15),
  firstname          = sample(c("James","Sarah","Michael"), 15, TRUE),
  last_name          = sample(c("Smith","Jones"), 15, TRUE),
  birth_date         = as.Date("1945-01-01") + sample(0:10000, 15),
  date_of_admission  = as.Date("2024-06-01") + sample(0:180, 15),
  date_of_discharge  = as.Date("2024-06-15") + sample(0:180, 15),
  medicare_number    = paste0(sample(2000:9999, 15), sample(10000:99999, 15)),
  sex                = sample(c("M","F"), 15, TRUE),
  zip_codes          = sample(c("4556","4557"), 15, TRUE),
  icd_codes          = sample(c("J06.9","U07.1","J44.1"), 15, TRUE),
  stringsAsFactors   = FALSE
)

df_hosp <- clean_the_nest(
  hosp_raw,
  data_type      = "hospital",
  drop_eggs      = TRUE,
  id_var         = "patient_id",
  lettername1    = "firstname",
  lettername2    = "last_name",
  dob            = "birth_date",
  medicare       = "medicare_number",
  gender         = "sex",
  postcode       = "zip_codes",
  icd_code       = "icd_codes",
  admission_date = "date_of_admission",
  discharge_date = "date_of_discharge"
)

df_hosp[, c("los","admission_outcome","icd_code")] |> head(4)
#>   los admission_outcome icd_code
#> 1 -18         Admission    J44.1
#> 2  75         Admission    J06.9
#> 3 156         Admission    J44.1
#> 4  46         Admission    U07.1

Vaccination data — long to wide

The Australian Immunisation Register exports one row per vaccination event. lie_nest_flat = TRUE pivots this to one row per person, with vax_date_1, vax_type_1, vax_date_2, vax_type_2, etc.

set.seed(3)
vax_raw <- data.frame(
  patient_id       = rep(paste0("VAX", 1:10), each = 2),
  firstname        = rep(c("Alice","Bob","Carol","Dan","Eve",
                            "Frank","Grace","Henry","Iris","Jack"), each = 2),
  last_name        = rep(c("Smith","Jones","Williams","Taylor","Brown",
                            "White","Black","Green","Blue","Red"), each = 2),
  birth_date       = rep(as.Date("1980-01-01") + sample(-2000:2000, 10), each = 2),
  gender           = rep(sample(c("M","F"), 10, TRUE), each = 2),
  postcode         = rep(sample(c("4556","4557"), 10, TRUE), each = 2),
  medicare_number  = rep(paste0(sample(2000:9999, 10), sample(10000:99999, 10)), each = 2),
  vaccine_delivered = rep(c("COVID-19 XBB.1.5","COVID-19 XBB.1.5"), 10),
  service_date     = c(rbind(
    as.Date("2024-01-15") + sample(0:30, 10),
    as.Date("2024-05-01") + sample(0:30, 10)
  )),
  stringsAsFactors = FALSE
)

df_vax <- clean_the_nest(
  vax_raw,
  data_type     = "vaccination",
  lie_nest_flat = TRUE,
  id_var        = "patient_id",
  lettername1   = "firstname",
  lettername2   = "last_name",
  dob           = "birth_date",
  medicare      = "medicare_number",
  gender        = "gender",
  postcode      = "postcode",
  vax_type      = "vaccine_delivered",
  vax_date      = "service_date"
)

df_vax[, c("id_var","vax_date_1","vax_type_1","vax_date_2","vax_type_2")] |> head(4)
#> # A tibble: 4 × 5
#>   id_var vax_date_1 vax_type_1       vax_date_2 vax_type_2      
#>   <chr>       <dbl> <chr>                 <dbl> <chr>           
#> 1 VAX1        19737 COVID-19 XBB.1.5      19847 COVID-19 XBB.1.5
#> 2 VAX10       19746 COVID-19 XBB.1.5      19870 COVID-19 XBB.1.5
#> 3 VAX2        19745 COVID-19 XBB.1.5      19857 COVID-19 XBB.1.5
#> 4 VAX3        19753 COVID-19 XBB.1.5      19852 COVID-19 XBB.1.5

Birth-cohort studies

In birth-cohort vaccine effectiveness studies (e.g. nirsevimab or Abrysvo effectiveness in infants), the date of birth is the cohort entry date. Pass the same column name to both dob and cohort_entry_date — no duplication needed.

df_cohort <- clean_the_nest(
  birth_cohort_data,
  data_type         = "cases",
  id_var            = "baby_id",
  lettername1       = "first_name",
  lettername2       = "last_name",
  dob               = "babys_date_of_birth",
  cohort_entry_date = "babys_date_of_birth",   # same column — aliased internally
  cohort_exit_date  = "end_of_followup_date",
  gender            = "sex"
)

Australian Medicare numbers

clean_the_nest() understands the full structure of Australian Medicare numbers and produces semantically named output columns for each component, rather than opaque digit-count names.

Structure

A complete Medicare number has up to 11 characters:

Component	Digits	Column	Description
Account identifier	1–8	`medicare08`	Unique household/account number. First digit is always 2–6.
Checksum	9	`medicare09`	Mathematically derived from digits 1–8; used to validate the number. `clean_the_nest()` validates this automatically.
Card issue number	10	`medicare10`	Increments each time a new card is issued (lost, expired, family member added). Never 0.
IRN	11	`medicare_irn`	Individual Reference Number — the digit printed left of a person’s name. Primary cardholder = 1, partner = 2, children = 3, 4, etc. Up to 9 people share a card.

Two additional columns are always produced: medicare_clean (spaces removed, as supplied) and medicare_valid (logical — TRUE if the checksum is mathematically correct).

Which column to use for linkage?

Linkage purpose	Use
Person-level linkage (cases ↔︎ AIR)	`medicare09` (stable across card reissues and family members)
Card-specific linkage (e.g. AIR dose records)	`medicare10`
Household-level blocking	`medicare08`
Distinguishing individuals on the same card	`medicare_irn`

mc_data <- data.frame(
  patient_id = c("PT001", "PT002", "PT003"),
  # PT001: valid 10-digit number (no IRN appended)
  # PT002: valid 11-digit number (IRN = 1)
  # PT003: invalid checksum (digit 9 is 7, should be 3)
  mcare = c("2428778132", "24287781321", "2428778172"),
  stringsAsFactors = FALSE
)

# suppressWarnings() here because PT003 has an invalid checksum —
# that's exactly what we want to demonstrate.
df_mc <- suppressWarnings(suppressMessages(
  clean_the_nest(mc_data, data_type = "cases",
                 id_var = "patient_id", medicare = "mcare")
))

df_mc[, c("id_var", "medicare08", "medicare09", "medicare10",
           "medicare_irn", "medicare_valid")]
#>   id_var medicare08 medicare09 medicare10 medicare_irn medicare_valid
#> 1  PT001   24287781  242877813 2428778132         <NA>           TRUE
#> 2  PT002   24287781  242877813 2428778132            1           TRUE
#> 3  PT003   24287781  242877817 2428778172         <NA>          FALSE

Notice that PT003 has medicare_valid = FALSE — clean_the_nest() warns you about invalid numbers before linkage, because an invalid Medicare number will never match in starling::murmuration(). In real data, investigate these rows: a common cause is the IRN being stored as part of the 10-digit number rather than separately.

The `drop_eggs` argument

drop_eggs = TRUE retains only the columns needed for linkage and downstream analysis. This produces a lean, manageable dataset for early-stage work. Use keep_vars to retain additional columns you need alongside the defaults.

# Without drop_eggs — all original columns plus derived ones
df_full <- clean_the_nest(
  cases_raw, data_type = "cases",
  id_var = "identity", lettername1 = "first_name", lettername2 = "surname",
  dob = "date_of_birth", onset_date = "date_of_onset"
)
ncol(df_full)
#> [1] 19

# With drop_eggs — only the linkage and analysis essentials
df_lean <- clean_the_nest(
  cases_raw, data_type = "cases", drop_eggs = TRUE,
  id_var = "identity", lettername1 = "first_name", lettername2 = "surname",
  dob = "date_of_birth", onset_date = "date_of_onset",
  keep_vars = "disease_name"   # retain this one extra
)
ncol(df_lean)
#> [1] 15
names(df_lean)
#>  [1] "lettername1"                 "lettername2"                
#>  [3] "lettername1_lettername2"     "lettername1_lettername2_dob"
#>  [5] "age"                         "medicare_no"                
#>  [7] "onset_date"                  "block1"                     
#>  [9] "block2"                      "block3"                     
#> [11] "id_var"                      "dob"                        
#> [13] "postcode"                    "gender"                     
#> [15] "disease_name"

What comes next

Once your data is cleaned:

preening() — assign age bands using one of ~50 named schemes (see vignette("preening"))
plumage() — detect chronic comorbidities from ICD-10-AM coding (see vignette("plumage")). For hospital datasets, this is typically the next step after clean_the_nest() — the standardised icd_code column produced here is the direct input to plumage(icd_column = "icd_code").
roost() — aggregate to a time unit for an epi curve (see vignette("roost"))
starling::murmuration() — probabilistic record linkage across the cleaned datasets
molting() — de-identify before sharing (see vignette("molting"))

clean_the_nest(): Standardising Surveillance Data

Preparing cases, hospitalisations, and vaccination records for linkage

Dr Nicolas Smoll, SCPHU, Sunshine Coast Hospital and Health Service

2026-09-23

What `clean_the_nest()` does

The three data types

Worked examples

Cases (notifiable disease linelist)

Hospital admissions

Vaccination data — long to wide

Birth-cohort studies

Australian Medicare numbers

Structure

Which column to use for linkage?

The `drop_eggs` argument

What comes next

clean_the_nest(): Standardising Surveillance Data

Preparing cases, hospitalisations, and vaccination records for linkage

Dr Nicolas Smoll, SCPHU, Sunshine Coast Hospital and Health Service

2026-09-23

What clean_the_nest() does

The three data types

Worked examples

Cases (notifiable disease linelist)

Hospital admissions

Vaccination data — long to wide

Birth-cohort studies

Australian Medicare numbers

Structure

Which column to use for linkage?

The drop_eggs argument

What comes next

What `clean_the_nest()` does

The `drop_eggs` argument