---
title: "Getting Started with the ipd Package"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting Started with the ipd Package}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
  %\VignetteDepends{tidyverse}
  %\VignetteDepends{patchwork}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  error = FALSE,
  warning = FALSE,
  message = FALSE,
  comment = "#>"
)
```

```{r, echo=FALSE}
default_options <- options()
```

# Introduction

## Background

With the rapid advancement of artificial intelligence and machine learning (AI/ML), researchers from a wide range of disciplines increasingly use predictions from pre-trained algorithms as outcome variables in statistical analyses. However, reifying algorithmically-derived values as measured outcomes may lead to biased estimates and anti-conservative inference ([Hoffman et al., 2023](https://arxiv.org/abs/2401.08702)). The statistical challenges encountered when drawing inference on predicted data (IPD) include: 

1. Understanding the relationship between predicted outcomes and their true, unobserved counterparts
2. Quantifying the robustness of the AI/ML models to resampling or uncertainty about the training data
3. Appropriately propagating both bias and uncertainty from predictions into downstream inferential tasks

Several works have proposed methods for IPD, including post-prediction inference (PostPI) by [Wang et al., 2020](https://www.pnas.org/doi/suppl/10.1073/pnas.2001238117), prediction-powered inference (PPI) and PPI++ by [Angelopoulos et al., 2023a](https://www.science.org/doi/10.1126/science.adi6000) and [Angelopoulos et al., 2023b](https://arxiv.org/abs/2311.01453), and post-prediction adaptive inference (PSPA) by [Miao et al., 2023](https://arxiv.org/abs/2311.14220). To enable researchers and practitioners interested in these state-of-the-art methods, we have developed `ipd`, a open-source `R` package that implements these methods under the umbrella of IPD.

This vignette provides a guide to using the `ipd` package, including installation instructions, examples of data generation, model fitting, and usage of custom methods. The examples demonstrate the package's functionality.

## Notation

Following the notation of [Miao et al., 2023](https://arxiv.org/abs/2311.14220), we assume we have the following data structure:

1. We have two datasets: a labeled dataset, \(\mathcal{L} = \left\{Y^\mathcal{L}, X^\mathcal{L}, f\left(X^\mathcal{L}\right)\right\}\), and an unlabeled dataset, \(\left\{X^\mathcal{U}, f\left(X^\mathcal{U}\right)\right\}\). The labeled set is typically smaller in size compared to the unlabeled set. 
2. We have access to an algorithm \(f(X)\) that can predict our outcome of interest \(Y\).
3. Our interest is in performing inference on a quantity such as the outcome mean or quantile, or to recover a downstream inferential (mean) model:

$$\mathbb{E}\left[Y^{\mathcal{U}} \mid \boldsymbol{X}^{\mathcal{U}}\right] = g^{-1}\left(\boldsymbol{X}^{\mathcal{U}'}\beta\right),$$

where \(\beta\) is a vector of regression coefficients and \(g(\cdot)\) is a given link function, such as the identity link for linear regression, the logistic link for logistic regression, or the log link for Poisson regression. However, in practice, we do not observe \(Y^\mathcal{U}\) in the 'unlabeled' subset of the data. Instead, these values are replaced by the predicted \(f(X^\mathcal{U})\). We can use methods for IPD to obtain corrected estimates and standard errors when we replace these unobserved \(Y^\mathcal{U}\) by \(f(X^\mathcal{U})\).

## Installation

To install the development version of `ipd` from [GitHub](https://github.com/ipd-tools/ipd), you can use the `devtools` package:

```{r, eval = F}
#-- Install devtools if it is not already installed

install.packages("devtools")   

#-- Install the ipd package from GitHub

devtools::install_github("ipd-tools/ipd")
```

# Usage

We provide a simple example to demonstrate the basic use of the functions included in the `ipd` package.

```{r setup}
#-- Load ipd Package

library(ipd)
```

## Data Generation

The `ipd` packages provides a unified function, `simdat`, for generating synthetic datasets for various models. The function currently supports "mean", "quantile", "ols", "logistic", and "poisson" models. 

### Function Arguments

- `n`: A vector of size 3 indicating the sample size in the training, labeled, and unlabeled data sets.
- `effect`: A float specifying the regression coefficient for the first variable of interest (defaults to 1).
- `sigma_Y`: A float specifying the residual variance for the generated outcome.
- `model`: The type of model to be generated. Must be one of `"mean"`, `"quantile"`, `"ols"`, `"logistic"`, or `"poisson"`.

The `simdat` function generate a data.frame with three subsets: (1) an independent "training" set with additional observations used to fit a prediction model, and "labeled" and "unlabeled" sets which contain the observed and predicted outcomes and the simulated features of interest. 

### Generating Data for Linear Regression

We can generate a continuous outcome and relevant predictors for linear regression as follows. The `simdat` function generates four independent covariates, $X_1$, $X_2$, $X_3$, and $X_4$, and the outcome:

$$Y = \text{effect}\times X_1 + \frac{1}{2}\times X_2^2 + \frac{1}{3}\times X_3^3 + \frac{1}{4}\times X_4^2 + \varepsilon_y$$

where `effect` is one of the function arguments and $\varepsilon_y \sim N(0, \text{sigma_Y})$, with `sigma_Y` being another argument. Here, the `simdat` function generates three subsets of data, a "training" subset, a "labeled" subset, and an "unlabeled" subset, based on the sizes in `n`. It then learns the prediction rule for the outcome in the "training" subset using a generalized additive model and predicts these outcomes in the "labeled" and "unlabeled" subsets:

```{r simols}
#-- Generate a Dataset for Linear Regression

set.seed(123)

n <- c(10000, 500, 1000)

dat_ols <- simdat(n = n, effect = 1, sigma_Y = 4, model = "ols")

#-- Print First 6 Rows of Training, Labeled, and Unlabeled Subsets

options(digits=2)

head(dat_ols[dat_ols$set == "training",])

head(dat_ols[dat_ols$set == "labeled",])

head(dat_ols[dat_ols$set == "unlabeled",])
```

The `simdat` function provides observed and unobserved outcomes for both the labeled and unlabeled datasets, though in practice the observed outcomes are not in the unlabeled set. We can visualize the relationships between these variables in the labeled data subset:

```{r plot, echo = F, error = F, warning = F, message = F, comment = NA, fig.height=4, fig.width=7}

library(tidyverse)

library(patchwork)

#-- Plot example labeled data

dat_labeled <- dat_ols[dat_ols$set == "labeled",]

p1 <- dat_labeled |>

  pivot_longer(Y:f) |>

  mutate(

    name = factor(name, levels = c("f", "Y"),

      labels = c("Predicted ", "Observed ")) |> fct_rev()) |>

  ggplot(aes(x = X1, y = value, group = name, color = name, fill = name)) +

    theme_bw() + coord_fixed(1/3) +
  
    geom_abline(slope = 1, intercept = 0) +

    geom_point(alpha = 0.5) +

    geom_smooth(method = "lm") +

    scale_x_continuous(limits = c(-2.5, 2.5)) +

    scale_y_continuous(limits = c(-7.5, 7.5)) +

    scale_color_manual(values = c("#AA4AC4", "#00C1D5")) +

    scale_fill_manual(values = c("#AA4AC4", "#00C1D5")) +

    labs(
      
      x = "\nCovariate", y = "Outcome\n", fill = NULL, color = NULL,
       
      title = "Outcomes vs. Covariate") +

    theme(

      legend.direction = "horizontal",

      legend.position = "inside",

      legend.position.inside = c(0.5, 0.91),

      axis.title = element_text(size = 8),

      axis.text = element_text(size = 8),
      
      title = element_text(size = 8))

p2 <- dat_labeled |> 
  
  ggplot(aes(x = f, y = Y)) +

    theme_bw() + coord_fixed(1) +
  
    geom_abline(slope = 1, intercept = 1) +

    geom_point(color = "#00C1D5", alpha = 0.5) +

    geom_smooth(method = "lm", color = "#00C1D5") +

    scale_x_continuous(limits = c(-7.5, 7.5)) +

    scale_y_continuous(limits = c(-7.5, 7.5)) +

    labs(
      
      x = "\nPredicted Outcome", y = "Observed Outcome\n",
       
      title = "Predicted vs. Observed Outcomes") +

    theme(

      axis.title = element_text(size = 8),

      axis.text = element_text(size = 8),
      
      title = element_text(size = 8))

fig1 <- (p1 + theme(plot.margin = unit(c(0,20,0, 0), "pt"))) +

  (p2 + theme(plot.margin = unit(c(0,20,0,20), "pt"))) +

  plot_annotation(tag_levels = "A")

fig1

```

We can see that:

-   The predicted outcomes are more correlated with the covariate than the true outcomes (plot A)
-   The predicted outcomes are not perfect substitutes for the true outcomes (plot B)

### Generating Data for Logistic Regression

As another example, we can generate a binary outcome and relevant predictors for logistic regression as follows:

```{r simlogistic}
#-- Generate a Dataset for Logistic Regression

set.seed(123)

dat_logistic <- simdat(n = n, effect = 3, sigma_Y = 1, 
                       
  model = "logistic")

#-- Print First 6 Rows of Training, Labeled, and Unlabeled Subsets

head(dat_logistic[dat_logistic$set == "training",])

head(dat_logistic[dat_logistic$set == "labeled",])

head(dat_logistic[dat_logistic$set == "unlabeled",])
```

```{r logsum, echo = F}

dat_logistic_labeled <- dat_logistic[dat_logistic$set == "labeled",]

dat_logistic_labeled_summ <- dat_logistic_labeled |>
  
  group_by(Y, f) |> count() |> ungroup() |>
  
  mutate(Y = factor(Y), f = factor(f), 
         
    pct = n / sum(n) * 100,
    
    fill = if_else(Y == f, 1, 0))

```

We can again visualize the relationships between the true and predicted outcome variables in the labeled data subset and see that `r paste0(sprintf("%2.1f", sum(dat_logistic_labeled_summ[dat_logistic_labeled_summ$fill == 1, "pct"])), "%")` observations are correctly predicted:

```{r plot2, echo = F, fig.width=7}

dat_logistic_labeled_summ |>
  
  ggplot(aes(x = f, y = Y, fill = fill)) + 
  
    geom_tile() + coord_equal() +
  
    geom_text(
      
      aes(label = paste0(n, " (", sprintf("%2.1f", pct), "%)")), vjust = 1) +
  
    scale_x_discrete(expand = c(0,0), limits = rev) +
  
    scale_y_discrete(expand = c(0,0)) +
  
    scale_fill_gradient(high = "#00C1D5", low = "#FFB500") + 
  
    theme(legend.position = "none")
```

## Model Fitting

### Linear Regression

We compare two non-IPD approaches to analyzing the data to methods included in the `ipd` package.

#### 0.1 'Naive' Regression Using the Predicted Outcomes

```{r naive}
#--- Fit the Naive Regression

lm(f ~ X1, data = dat_ols[dat_ols$set == "unlabeled",]) |> 
  
  summary()
```

#### 0.2 'Classic' Regression Using only the Labeled Data

```{r classic}
#--- Fit the Classic Regression

lm(Y ~ X1, data = dat_ols[dat_ols$set == "labeled",]) |> 
  
  summary()
```

You can fit the various IPD methods to your data and obtain summaries using the provided wrapper function, `ipd()`:

#### 1.1 PostPI Bootstrap Correction (Wang et al., 2020)

```{r postpi_boot_ols}
#-- Specify the Formula

formula <- Y - f ~ X1

#-- Fit the PostPI Bootstrap Correction

nboot <- 200

ipd::ipd(formula, 
         
  method = "postpi_boot", model = "ols", data = dat_ols, label = "set", 
  
  nboot = nboot) |> 
  
  summary()
```

#### 1.2 PostPI Analytic Correction (Wang et al., 2020)

```{r postpi_analytic_ols}
#-- Fit the PostPI Analytic Correction

ipd::ipd(formula, 
         
  method = "postpi_analytic", model = "ols", data = dat_ols, label = "set") |> 
  
  summary()
```

#### 2. Prediction-Powered Inference (PPI; Angelopoulos et al., 2023)

```{r ppi_ols}
#-- Fit the PPI Correction

ipd::ipd(formula, 
         
  method = "ppi", model = "ols", data = dat_ols, label = "set") |> 
  
  summary()
```

#### 3. PPI++ (Angelopoulos et al., 2023)

```{r ppi_plusplus}
#-- Fit the PPI++ Correction

ipd::ipd(formula, 
         
  method = "ppi_plusplus", model = "ols", data = dat_ols, label = "set") |> 
  
  summary()
```

#### 4. post-prediction adaptive inference (PSPA; Miao et al., 2023)

```{r pspa}
#-- Fit the PSPA Correction

ipd::ipd(formula, 
         
  method = "pspa", model = "ols", data = dat_ols, label = "set") |> 
  
  summary()
```

### Logistic Regression

We also show how these methods compare for logistic regression.

#### 0.1 'Naive' Regression Using the Predicted Outcomes

```{r naive2}
#--- Fit the Naive Regression

glm(f ~ X1, family = binomial, 
    
  data = dat_logistic[dat_logistic$set == "unlabeled",]) |> 
  
  summary()
```

#### 0.2 'Classic' Regression Using only the Labeled Data

```{r classic2}
#--- Fit the Classic Regression

glm(Y ~ X1, family = binomial,
   
  data = dat_logistic[dat_logistic$set == "labeled",]) |> 
  
  summary()
```

You can again fit the various IPD methods to your data and obtain summaries using the provided wrapper function, `ipd()`:

#### 1. PostPI Bootstrap Correction (Wang et al., 2020)

```{r postpi_boot2}
#-- Specify the Formula

formula <- Y - f ~ X1

#-- Fit the PostPI Bootstrap Correction

nboot <- 200

ipd::ipd(formula, method = "postpi_boot", model = "logistic", 
         
  data = dat_logistic, label = "set", nboot = nboot) |> 
  
  summary()
```

#### 2. Prediction-Powered Inference (PPI; Angelopoulos et al., 2023)

```{r ppi2}
#-- Fit the PPI Correction

ipd::ipd(formula, method = "ppi", model = "logistic", 
         
  data = dat_logistic, label = "set") |> 
  
  summary()
```

#### 3. PPI++ (Angelopoulos et al., 2023)

```{r ppi_plusplus2}
#-- Fit the PPI++ Correction

ipd::ipd(formula, method = "ppi_plusplus", model = "logistic", 
         
  data = dat_logistic, label = "set") |> 
  
  summary()
```

#### 4. Post-Prediction Adaptive Inference (PSPA; Miao et al., 2023)

```{r pspa2}
#-- Fit the PSPA Correction

ipd::ipd(formula, method = "pspa", model = "logistic", 
         
  data = dat_logistic, label = "set") |> 
  
  summary()
```

## Printing, Summarizing, and Tidying

The package also provides custom `print`, `summary`, `tidy`, `glance`, and `augment` methods to facilitate easy model inspection:

```{r methods}
#-- Fit the PostPI Bootstrap Correction

nboot <- 200

fit_postpi <- ipd::ipd(formula, 
         
  method = "postpi_boot", model = "ols", data = dat_ols, label = "set", 
  
  nboot = nboot)

```

### Print Method

The `print` method gives an abbreviated summary of the output from the `ipd` function:

```{r print}
#-- Print the Model

print(fit_postpi)

```

### Summary Method

The `summary` method gives more detailed information about the estimated coefficients, standard errors, and confidence limits:

```{r summary}
#-- Summarize the Model

summ_fit_postpi <- summary(fit_postpi)
  
#-- Print the Model Summary

print(summ_fit_postpi)
```

### Tidy Method

The `tidy` method organizes the model coefficients into a [tidy](https://broom.tidymodels.org/) format.

```{r tidy}
#-- Tidy the Model Output

tidy(fit_postpi)
```

### Glance Method

The `glance` method returns a one-row summary of the model fit.

```{r glance}
#-- Get a One-Row Summary of the Model

glance(fit_postpi)
```

### Augment Method

The `augment` method adds model predictions and residuals to the original dataset.

```{r augment}
#-- Augment the Original Data with Fitted Values and Residuals

augmented_df <- augment(fit_postpi)

head(augmented_df)
```

```{r, echo=FALSE}
options(default_options)
```

# Conclusions

The `ipd` package offers a suite of functions for conducting inference on predicted data. With custom methods for printing, summarizing, tidying, glancing, and augmenting model outputs, `ipd` streamlines the process of IPD-based inference in `R`. We will continue to develop this package to include more targets of inference and IPD methods as they are developed, as well as additional functionality for analyzing such data. For further information and detailed documentation, please refer to the function help pages within the package, e.g.,

```{r help, eval=F}
?ipd
```

## Feedback

For questions, comments, or any other feedback, please contact the developers (<ssalerno@fredhutch.org>).

## Contributing

Contributions are welcome! Please open an issue or submit a pull request on [GitHub](https://github.com/ipd-tools/ipd).

## License

This package is licensed under the MIT License.