---
title: "Creating denominator cohorts"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{a02_Creating_denominator_cohorts}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  chunk_output_type: console
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5,
  eval = Sys.getenv("$RUNNER_OS") != "macOS"
)
```

# Introduction
Calculating incidence or prevalence requires first identifying an appropriate denominator population. To find such a denominator population (or multiple denominator populations) we can use the `generateDenominatorCohortSet()` function. This function will identify the time that people in the database satisfy a set of criteria related to the study period and individuals´ age, sex, and amount of prior observed history. 

When using `generateDenominatorCohortSet()` individuals will enter a denominator population on the respective date of the latest of the following: 

1. Study start date
2. Date at which they have sufficient prior observation 
3. Date at which they reach a minimum age

They will then exit on the respective date of the earliest of the following: 

1. Study end date
2. Date at which their observation period ends
3. The last day in which they have the maximum age

Let´s go through a few examples to make this logic a little more concrete.

### No specific requirements
The simplest case is that no study start and end dates are specified, no prior history requirement is imposed, nor any age or sex criteria. In this case individuals will enter the denominator population once they have entered the database (start of observation period) and will leave when they exit the database (end of observation period). Note that in some databases a person can have multiple observation periods, in which case their contribution of person time would look like the the last person below.

```{r,echo=FALSE, message=FALSE, out.width="80%", warning=FALSE}
library(knitr)
library(here)
knitr::include_graphics(here("vignettes/dpop1.png"))
```

### Specified study period
If we specify a study start and end date then only observation time during this period will be included.

```{r,echo=FALSE, out.width="80%"}
knitr::include_graphics(here("vignettes/dpop2.png"))
```

### Specified study period and prior history requirement
If we also add some requirement of prior history then somebody will only contribute time at risk once this is reached.

```{r,echo=FALSE, out.width="80%"}
knitr::include_graphics(here("vignettes/dpop3.png"))
```


### Specified study period, prior history requirement, and age and sex criteria

Lastly we can also impose age and sex criteria, and now individuals will only contribute time when they also satisfy these criteria. Not shown in the below figure is a person´s sex, but we could also stratify a denominator population by this as well.

```{r,echo=FALSE, out.width="80%"}
knitr::include_graphics(here("vignettes/dpop4.png"))
```

# Using generateDenominatorCohortSet()
`generateDenominatorCohortSet()` is the function we use to identify a set of denominator populations. To demonstrate its use, let´s load the IncidencePrevalence package (along with a couple of packages to help for subsequent plots) and generate 500 example patients using the `mockIncidencePrevalenceRef()` function.

```{r, message=FALSE, warning=FALSE}
library(CDMConnector)
library(IncidencePrevalence)
library(ggplot2)
library(tidyr)
library(dplyr)
```

```{r, message=TRUE}
cdm <- mockIncidencePrevalenceRef(sampleSize = 500)
```


### No specific requirements

We can get a denominator population without including any particular requirements like so

```{r, message=FALSE, warning=FALSE}
cdm <- generateDenominatorCohortSet(
  cdm = cdm, 
  name = "denominator",
  cohortDateRange = as.Date(c(NA,NA)),
  ageGroup = list(c(0, 150)),
  sex = "Both",
  daysPriorObservation = 0
)
cdm$denominator

cdm$denominator %>%
  filter(subject_id %in% c("1", "2", "3", "4", "5"))
```

Let´s have a look at the included time of the first five patients. We can see that people enter and leave at different times.

```{r, message=FALSE, warning=FALSE, echo=FALSE}
cdm$denominator %>%
  collect() %>%
  filter(subject_id %in% c("1", "2", "3", "4", "5")) %>%
  pivot_longer(cols = c(
    "cohort_start_date",
    "cohort_end_date"
  )) %>%
  ggplot() +
  geom_point(aes(x = value, y = subject_id)) +
  geom_line(aes(x = value, y = subject_id)) +
  theme_minimal() +
  xlab("Year")
```

We can also plot a histogram of start and end dates of the 500 simulated patients 
```{r, message=FALSE, warning=FALSE}
cdm$denominator %>%
  collect() %>%
  ggplot() +
  theme_minimal() +
  geom_histogram(aes(cohort_start_date),
    colour = "black", fill = "grey"
  )
```

```{r, message=FALSE, warning=FALSE}
cdm$denominator %>%
  collect() %>%
  ggplot() +
  theme_minimal() +
  geom_histogram(aes(cohort_end_date),
    colour = "black", fill = "grey"
  )
```

### Specified study period
We can get specify a study period like so

```{r, message=FALSE, warning=FALSE}
cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator", 
  cohortDateRange = c(as.Date("1990-01-01"), as.Date("2009-12-31")),
  ageGroup = list(c(0, 150)),
  sex = "Both",
  daysPriorObservation = 0
)
cdm$denominator

cohortCount(cdm$denominator)

cdm$denominator %>%
  filter(subject_id %in% c("1", "2", "3", "4", "5"))
```

Now we can see that many more people share the same cohort entry (the study start date) and cohort exit (the study end date).

```{r, message=FALSE, warning=FALSE}
cdm$denominator %>%
  collect() %>%
  ggplot() +
  theme_minimal() +
  geom_histogram(aes(cohort_start_date),
    colour = "black", fill = "grey"
  )
```

```{r, message=FALSE, warning=FALSE}
cdm$denominator %>%
  collect() %>%
  ggplot() +
  theme_minimal() +
  geom_histogram(aes(cohort_end_date),
    colour = "black", fill = "grey"
  )
```


### Specified study period and prior history requirement
We can add some requirement of prior history

```{r, message=FALSE, warning=FALSE}
cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator",
  cohortDateRange = c(as.Date("1990-01-01"), as.Date("2009-12-31")),
  ageGroup = list(c(0, 150)),
  sex = "Both",
  daysPriorObservation = 365
)
cdm$denominator

cohortCount(cdm$denominator)

cdm$denominator %>%
  filter(subject_id %in% c("1", "2", "3", "4", "5"))
```

### Specified study period, prior history requirement, and age and sex criteria
In addition to all the above we could also add some requirements around age and sex. One thing to note is that the age upper limit will include time from a person up to the day before their reach the age upper limit + 1 year. For instance, when the upper limit is 65, that means we will include time from a person up to and including the day before their 66th birthday.

```{r, message=FALSE, warning=FALSE}
cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator",
  cohortDateRange = c(as.Date("1990-01-01"), as.Date("2009-12-31")),
  ageGroup = list(c(18, 65)),
  sex = "Female",
  daysPriorObservation = 365
)
cdm$denominator %>%
  glimpse()

cohortCount(cdm$denominator)

cdm$denominator %>%
  filter(subject_id %in% c("1", "2", "3", "4", "5"))
```

### Multiple options to return multiple denominator populations 
More than one age, sex and prior history requirements can be specified at the same time. First, we can take a look at having two age groups. We can see below that those individuals who have their 41st birthday during the study period will go from the first cohort (age_group: 0;40) to the second (age_group: 41;100) on this day.
```{r, message=FALSE, warning=FALSE}
cdm <- mockIncidencePrevalenceRef(sampleSize = 500, 
                                  earliestObservationStartDate = as.Date("2000-01-01"),
                                  latestObservationStartDate =  as.Date("2005-01-01"),
                                  minDaysToObservationEnd = 10000, 
                                  maxDaysToObservationEnd = NULL,
                                  earliestDateOfBirth = as.Date("1960-01-01"), 
                                  latestDateOfBirth = as.Date("1980-01-01"))

cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator",
  ageGroup = list(
    c(0, 40),
    c(41, 100)
  ),
  sex = "Both",
  daysPriorObservation = 0
)
cdm$denominator %>%
  filter(subject_id %in% !!as.character(seq(1:30))) %>%
  collect() %>%
  left_join(settings(cdm$denominator),
            by = "cohort_definition_id") %>%
  pivot_longer(cols = c(
    "cohort_start_date",
    "cohort_end_date"
  )) %>%
  mutate(subject_id = factor(as.numeric(subject_id))) %>%
  ggplot(aes(x = subject_id, y = value, colour = age_group)) +
  geom_point(position = position_dodge(width = 0.5)) +
  geom_line(position = position_dodge(width = 0.5)) +
  theme_minimal() +
  theme(legend.position = "top", 
        legend.title = element_blank()) +
  ylab("Year") +
  coord_flip()
```

We can then also sex specific denominator cohorts.
```{r, message=FALSE, warning=FALSE, fig.height=8}
cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator",
  ageGroup = list(
    c(0, 40),
    c(41, 100)
  ),
  sex = c("Male", "Female", "Both"),
  daysPriorObservation = 0
)
cdm$denominator %>%
  filter(subject_id %in% !!as.character(seq(1:15))) %>%
  collect() %>%
  left_join(settings(cdm$denominator)) %>%
  pivot_longer(cols = c(
    "cohort_start_date",
    "cohort_end_date"
  )) %>%
  mutate(subject_id = factor(as.numeric(subject_id))) %>%
  ggplot(aes(x = subject_id, y = value, colour = age_group)) +
    facet_grid(sex ~ ., space = "free_y") +
  geom_point(position = position_dodge(width = 0.5)) +
  geom_line(position = position_dodge(width = 0.5)) +
  theme_bw() +
  theme(legend.position = "top", 
        legend.title = element_blank()) +
  ylab("Year") +
  coord_flip()
```

And we could also specifying multiple prior history requirements 
```{r, message=FALSE, warning=FALSE, fig.height=10}
cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator",
  ageGroup = list(
    c(0, 40),
    c(41, 100)
  ),
  sex = c("Male", "Female", "Both"),
  daysPriorObservation = c(0, 365)
)
cdm$denominator %>%
  filter(subject_id %in% !!as.character(seq(1:8))) %>%
  collect() %>%
  left_join(cohortSet(cdm$denominator)) %>%
  pivot_longer(cols = c(
    "cohort_start_date",
    "cohort_end_date"
  )) %>%
  mutate(subject_id = factor(as.numeric(subject_id))) %>%
  ggplot(aes(x = subject_id, y = value, colour = age_group, 
             linetype = sex, shape = sex
             )) +
  facet_grid(sex + days_prior_observation ~ ., space = "free", 
             scales = "free") +
  geom_point(position = position_dodge(width = 0.5)) +
  geom_line(position = position_dodge(width = 0.5)) +
  theme_bw() +
  theme(legend.position = "top") +
  ylab("Year") +
  coord_flip()
```

Note, setting requirementInteractions to FALSE would mean that only the first value of other age, sex, and prior history requirements are considered for a given characteristic. In this case the order of the values will be important and generally the first values will be the primary analysis settings while subsequent values are for secondary analyses.
```{r, message=FALSE, warning=FALSE, fig.height=10}
cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator",
  ageGroup = list(
    c(0, 40),
    c(41, 100)
  ),
  sex = c("Male", "Female", "Both"),
  daysPriorObservation = c(0, 365),
  requirementInteractions = FALSE
)

cdm$denominator %>%
  filter(subject_id %in% !!as.character(seq(1:8))) %>%
  collect() %>%
  left_join(cohortSet(cdm$denominator)) %>%
  pivot_longer(cols = c(
    "cohort_start_date",
    "cohort_end_date"
  )) %>%
  mutate(subject_id = factor(as.numeric(subject_id))) %>%
  ggplot(aes(x = subject_id, y = value, colour = age_group, 
             linetype = sex, shape = sex
             )) +
  facet_grid(sex + days_prior_observation ~ ., space = "free", 
             scales = "free") +
  geom_point(position = position_dodge(width = 0.5)) +
  geom_line(position = position_dodge(width = 0.5)) +
  theme_bw() +
  theme(legend.position = "top") +
  ylab("Year") +
  coord_flip()
```

### Output 
`generateDenominatorCohortSet()` will generate a table with the denominator population, which includes the information on all the individuals who fulfill the given criteria at any point during the study period. It also includes information on the specific start and end dates in which individuals contributed to the denominator population (cohort_start_date and cohort_end_date). Each patient is recorded in a different row. For those databases that allow individuals to have multiple non-overlapping observation periods, one row for each patient and observation period is considered.

Considering the following example, we can see:

```{r, message=TRUE, warning=FALSE, message=FALSE}
cdm <- generateDenominatorCohortSet(
  cdm = cdm,
  name = "denominator",
  cohortDateRange = c(as.Date("1990-01-01"), as.Date("2009-12-31")),
  ageGroup = list(
    c(0, 18),
    c(19, 100)
  ),
  sex = c("Male", "Female"),
  daysPriorObservation = c(0, 365)
)

head(cdm$denominator, 8)
```

The output table will have several attributes. With `settings()` we can see the options used when defining the set of denominator populations. More than one age, sex and prior history requirements can be specified at the same time and each combination of these variables will result in a different cohort, each of which has a corresponding cohort_definition_id. In the above example, we identified 8 different cohorts:

```{r, message=TRUE, warning=FALSE}
settings(cdm$denominator) %>% 
  glimpse()
```

With `cohortCount()` we can see the number of individuals who entered each study cohort
```{r, message=TRUE, warning=FALSE}
cohortCount(cdm$denominator) %>% 
  glimpse()
```

With `attrition()` we can see the number of individuals in the database who were excluded from entering a given denominator population along with the reason (such as missing crucial information or not satisfying the sex or age criteria required, among others):

```{r, message=TRUE, warning=FALSE}
attrition(cdm$denominator) %>% 
  glimpse()
```