canpumf

The goal of canpumf is to facilitate ingesting, organizing, and working with StatCan PUMF data in R.

Installation

The canpumf package is available on CRAN and can be installed directly.

install.packages(canpumf)

Alternatively the current development version of canpumf from GitHub with:

remotes::install_github("mountainmath/canpumf")

Documentation

Please consult the documentation and example articles for further information.

StatCan publishes an official guide to the Labour Force Survey for different vintages of the LFS.

Cache path

PUMF data can be large and should be cached locally. Set the canpumf.cache_path option to a local directory via options(canpumf.cache_path="<your local path>") in your .Rprofile. Without this, data is stored in tempdir() for the session only.

DuckDB

On first use PUMF data is imported into DuckDB. By default a PUMF DuckDB connection will be shown in the RStudio (or Positron) Connections Pane once a connection is opened, you control the default behaviour by setting the canpumf.register_connection option in your .Rprofile:

options("canpumf.register_connection" = TRUE)

Basic usage

Some PUMF data is available from StatCan via direct download and can be accessed directly via get_pumf(). In other cases, PUMF data must be ordered via EFT and deposited in the cache directory so get_pumf() can find it.

PUMF data can also be loaded from the Borealis Dataverse, which hosts the ODESI collection of Statistics Canada PUMFs. Statistics Canada stays the primary source; Borealis is used automatically for vintages StatCan does not post (the 1971–1986 Census PUMFs), and any other Borealis PUMF dataset can be loaded by its DOI:

cat <- list_borealis_pumf_catalogue()                 # browse the Borealis PUMF collection
list_borealis_pumf_files("doi:10.5683/SP3/EZXFNL")   # inspect a dataset's files
shs_1997 <- get_pumf("SHS", "1997", borealis = "doi:10.5683/SP3/EZXFNL")

The version names the cache directory, and later calls to get_pumf("SHS", "1997") reopen the Borealis copy. Datasets outside the built-in registry are parsed by auto-detection. If the build warns that a variable’s unmatched values become NA, that variable is usually continuous with only a top-code labelled. Pass registry = pumf_registry_entry(data_fixups = list(force_numeric = c("NUMROOM", "AGEREFP"))) together with refresh = TRUE to keep it numeric.

No account is needed. If the BOREALIS_DATAVERSE_KEY environment variable is set, it is sent to Borealis as the API key, which gives access to restricted files the key’s owner is entitled to.

get_pumf() downloads (if needed), parses metadata, applies value labels automatically, and returns a lazy dplyr::tbl() backed by a local DuckDB database. Call dplyr::collect() to load into memory.

Column values are labeled automatically (e.g. province codes become factor levels like "British Columbia"). Column names remain as short coded names by default (e.g. PROV, LFSSTAT). To rename columns to human-readable variable labels, pipe through label_pumf_columns():

tbl <- get_pumf("LFS", "2022") |>
  label_pumf_columns()

When done querying, release the DuckDB connection with close_pumf(tbl).

Label repair

Statistics Canada’s shipped command files routinely carry truncated value and variable labels — hard cuts at 60 characters, dropped leading or interior text. The damage is upstream (SAS, SPSS and Stata versions of the same file agree byte for byte), and it is quiet: the codes and frequencies are correct, only the human-readable label is wrong.

For surveys whose user guide includes a data-dictionary appendix, canpumf parses that appendix and uses it to repair the labels. Because the appendix prints the frequency of every code, the scrape is first reconciled against the actual microdata: a guide whose counts do not reproduce a tabulation of the data and whose printed field positions do not reproduce the command file’s layout is discarded as the wrong document. A label is then only ever replaced when two things hold — the guide’s text demonstrably extends the command file’s, and the command file’s labels show the fingerprint of hard truncation (a spike of labels at a fixed ceiling). Without that second test a guide that prints full question wording where the command file carries a hand-written short label would “repair” perfectly good labels into questions. Two functions expose what happened:

gss <- get_pumf("GSS", "Cycle 16 (2002)")

# per-variable: did the guide's frequencies reconcile with the data?
table(pumf_freq_validation(gss)$status)

# every divergence found, repaired or not
pumf_label_repairs(gss, action = "repaired")
pumf_label_repairs(gss, action = "flagged")

flagged rows are divergences that were recorded but not acted on — most usefully, places where the guide and the command file genuinely disagree rather than one being a truncation of the other. Nothing is repaired silently. Set options(canpumf.pdf_crosscheck = FALSE) to turn the whole step off.

LFS data

LFS data is organized by year, except for the current year where it is organized by month. To access data for a specific year:

lfs_2022 <- get_pumf("LFS", "2022")

This downloads the 2022 LFS PUMF data if needed, parses it, loads labeled data into a shared DuckDB database, and returns a lazy tbl filtered to 2022. To access all LFS data currently in the local database:

lfs_all_local <- get_pumf("LFS")

To ensure the local database contains all available LFS versions, use refresh = "auto". This checks StatCan for versions not yet in the database and imports them:

lfs_all <- get_pumf("LFS", refresh = "auto")

Historical LFS (1976–2005)

Statistics Canada posts the LFS PUMF from 2006 onwards. The monthly files for January 1976 to December 2005 are available from the Borealis Dataverse (ODESI) as series "LFS_HIST". They use the legacy (pre-2017) LFS layout, so they live in their own database rather than being mixed into "LFS".

lfs_1995_06 <- get_pumf("LFS_HIST", "1995-06")  # one month
lfs_1995    <- get_pumf("LFS_HIST", "1995")     # all twelve months of 1995
lfs_hist    <- get_pumf("LFS_HIST", refresh = "auto")  # all 360 months (large)

ODESI labelled the same codes differently in different years (“Unemployed, temporary layoff” vs “Unemploy,temp layoff”). canpumf therefore applies one harmonised bilingual dictionary to every month, so each factor has the same levels across the whole 1976–2005 table. The deposits after 1986 carry weights rebased to a later Census (1987–1995 to 2001, 1996–2000 to 2006, 2001–2005 to 2011), so weighted levels can step at those boundaries.

One LFS timeline, 1976 onward

get_lfs_timeline() stacks whatever is loaded of "LFS_HIST" and "LFS" into one lazy table with a curated set of common variables. It attaches both databases read-only. By default it loads nothing itself. get_lfs_timeline(refresh = "auto") first loads any newly released months, so an analysis script built on it stays up to date.

tl <- get_lfs_timeline()
tl |>
  dplyr::filter(SURVMNTH == 6L) |>
  dplyr::summarise(employed = sum(FINALWT[LFSSTAT %in% c("Employed, at work",
                                                         "Employed, absent from work")]),
                   .by = SURVYEAR) |>
  dplyr::collect()
close_pumf(tl)

Variables with the same codes in both series carry the current LFS labels. A few are recoded to a common scheme: - LFSSTAT: the three historical unemployment categories are collapsed. - GENDER_SEX: sex and gender are combined. - MARSTAT: four categories; the files before November 1999 only have these four. - CMA: Montréal, Toronto, Vancouver or other, and NA before 1987. - Also recoded: SCHOOLN, AGYOWNK, NAICS_18 (18 industry groups), and EDUC (from 1990).

The hours and wage columns are in plain hours and dollars in both series, and the weight is FINALWT. Occupation and the historical family and spouse variables are left out; use get_pumf() on each series for those.

Census data

The canpumf package supports Census PUMF from 1971 through 2021. All releases from 1991 onward are available via direct download from Statistics Canada. Years 1986 and earlier are downloaded automatically from Borealis (English labels only). If you have ordered the Statistics Canada EFT bundle for one of those years and placed it in the cache directory, it is used instead; add "eft" or "borealis" to the version string to pick a source explicitly, e.g. get_pumf("Census", "1971 individuals CMA borealis").

pumf_2021 <- get_pumf("Census", "2021")

By default the package loads the individuals file. Available variants by year:

pumf_h_2016 <- get_pumf("Census", "2016 (hierarchical)")

Verified datasets

The following datasets have been end-to-end tested (metadata parsed, data imported, DuckDB built) without errors or warnings. Versions marked direct download can be fetched automatically by get_pumf() (from Statistics Canada, or from Borealis where marked); others must be placed in the cache directory via Statistics Canada’s EFT portal.

GSS surveys are keyed by their canonical Cycle N (YYYY) version (e.g. get_pumf("GSS", "Cycle 16 (2002)")), since a bare year is not unique across the GSS — several years carry both a regular cycle and a Giving/Volunteering survey. For convenience the cycle number alone ("Cycle 16", "16"), the bare year ("2002"), and the historical theme name ("Aging and Social Support", "Family 2017", "Time Use 2022") all resolve to the canonical key.

CPSS and CCAHS are keyed by their bare cycle number (get_pumf("CPSS", "1"), get_pumf("CCAHS", "1")). StatCan styles these cycles “Series N” (CPSS) and “Cycle N” (CCAHS), so "Series 3", "Cycle 4", and "CPSS 6" resolve to the number; CCAHS additionally accepts its reference year "2022". A bare year is not a CPSS alias, since several CPSS cycles share a calendar year.

Related packages

The cansim package is designed to retrieve and work with public Statistics Canada data tables. cansim prepares retrieved data tables as analysis-ready tidy dataframes and provides a number of convenience tools and functions to make it easier to work with Statistics Canada data. It is available on CRAN and on Github.

The cancensus package is designed to retrieve and work with public Statistics Canada census data via the CensusMapper API. It is available on CRAN and on Github.

Cite canpumf

If you wish to cite the canpumf package in your work:

von Bergmann, J. (2026), canpumf: Import StatCan PUMF data into R. v0.5.2. DOI 10.32614/CRAN.package.canpumf

A BibTeX entry for LaTeX users is

  @Manual{canpumf,
    author = {Jens {von Bergmann}},
    title = {canpumf: Import StatCan PUMF data into R},
    year = {2026},
    doi = {10.32614/CRAN.package.canpumf},
    note = {R package version 0.5.2},
    url = {https://mountainmath.github.io/canpumf/},
  }

Statistics Canada Attribution

Subject to the Statistics Canada Open Data License Agreement, licensed products using Statistics Canada data should employ the following acknowledgement of source:

Acknowledgment of Source

(a) You shall include and maintain the following notice on all licensed rights of the Information:

  - Source: Statistics Canada, name of product, reference date. Reproduced and distributed on an "as is" basis with the permission of Statistics Canada.
 
(b) Where any Information is contained within a Value-added Product, you shall include on such Value-added Product the following notice:

  - Adapted from Statistics Canada, name of product, reference date. This does not constitute an endorsement by Statistics Canada of this product.