---
title: "Analysing and visualising a literature"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Analysing and visualising a literature}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = FALSE, comment = "")
# Console colour carries no meaning on a rendered page. pkgdown turns it on for
# its own build, and the escape sequences then reach the reader as literal text,
# so colour is switched off here for a plain vignette render and a site build
# alike. The fixed width keeps tibbles inside the documentation column.
options(cli.num_colors = 1, cli.hyperlink = FALSE, crayon.enabled = FALSE,
        width = 80)
# Print data frames and tibbles as formatted tables.
local({
  kp <- function(x, ...) {
    if (any(vapply(x, is.list, logical(1)))) return(knitr::normal_print(x))
    knitr::knit_print(knitr::kable(x))
  }
  for (cls in c("data.frame", "tbl_df", "tbl")) {
    registerS3method("knit_print", cls, kp, envir = asNamespace("knitr"))
  }
})
has_ggplot2 <- requireNamespace("ggplot2", quietly = TRUE)
# Draw the figures on a transparent background so they sit on whatever colour
# the page behind them happens to be. An opaque white matte reads as a white
# slab on the site's dark theme, and worse once pkgdown's dark-mode filter
# inverts it into a black one. Both halves below are needed: the device option
# gives the file an alpha channel, and the theme override clears the white
# rectangle that ggplot2's complete themes paint over it regardless. The
# override belongs here, in the vignette, because a figure saved for a paper
# usually does want a background of its own, so the package's plotting
# functions leave it alone.
knitr::opts_chunk$set(dev.args = list(bg = "transparent"))
if (has_ggplot2) {
  sf_on_page <- ggplot2::theme(
    plot.background  = ggplot2::element_rect(fill = "transparent", colour = NA),
    panel.background = ggplot2::element_rect(fill = "transparent", colour = NA)
  )
  knitr::opts_chunk$set(render = function(x, ...) {
    if (inherits(x, "ggplot")) x <- x + sf_on_page
    knitr::knit_print(x, ...)
  })
}
```

```{r setup}
library(scopusflow)
```

Once a set of records is in hand, the package offers a small analysis layer that
turns it into the figures a bibliometric study usually needs. The steps that
contact the API are shown but not run. The rest run on `example_records`, the
corpus of 138 real journal articles on graphene supercapacitors that the package
bundles because Scopus records may not be redistributed, and which carries the
schema a retrieval returns. `vignette("scopusflow")` describes where it comes
from.

## What is in a record set

`summary()` is the quickest way to see what a retrieval holds, and it is worth
running before any closer analysis.

```{r}
records <- example_records
summary(records)
```

`scopus_top()` tallies the most frequent sources or authors.

```{r}
scopus_top(records, by = "source")
scopus_top(records, by = "author", n = 5)
```

The source tally is long-tailed, as a real literature is. These 138 articles are
spread across 90 journals, and only *ACS Applied Materials & Interfaces*, with
eight, appears more than five times. The author tally is flatter still, with 119
distinct authors, the most prolific of whom contributed three papers.

An author string that holds several names is split, so each contributor is
counted once per record. The bundled corpus names only the first author of each
paper, so it cannot show that happening. Two records with placeholder names can,
and the same splitting applies to the semicolon-joined author lists a live
harvest returns.

```{r}
multi <- scopus_records(list(entry = list(
  list(`dc:creator` = "Author A.; Author B."),
  list(`dc:creator` = "Author B.")
)))
scopus_top(multi, by = "author")
```

```{r eval = has_ggplot2, fig.alt = "A horizontal bar chart of the most frequent sources", fig.width = 7, fig.height = 3.5}
plot_scopus_top(scopus_top(records, by = "source"))
```

The same plot works on the author tally.

```{r eval = has_ggplot2, fig.alt = "A horizontal bar chart of the most frequent authors", fig.width = 7, fig.height = 3.5}
plot_scopus_top(scopus_top(records, by = "author", n = 5))
```

A record set also has an honest default view. `autoplot()` draws its records per
year. Because this corpus is a complete harvest of one query, those bars are the
real number of publications per year for that query. The same `autoplot()`
generic dispatches on `scopus_trend` and `scopus_top` objects too, delegating to
the plots above.

```{r eval = has_ggplot2, fig.alt = "A bar chart of publications per year from 2015 to 2024, fluctuating around fifteen a year", fig.width = 7, fig.height = 3.5}
ggplot2::autoplot(records)
```

## How a literature grows

`scopus_trend()` counts how many records match a query in each year, which is the
size of a literature over time. It issues one count request per year, so it needs
the API.

```{r eval = FALSE}
tr <- scopus_trend("graphene supercapacitor", years = 2015:2024,
                   field = "TITLE-ABS-KEY")
plot_scopus_trend(tr)
```

The offline equivalent costs no requests at all. A complete harvest already
contains its own yearly counts, so tallying the records by year gives what a
count per year would have returned, in the shape `scopus_trend()` returns it.

```{r}
by_year <- table(records$year)
tr <- tibble::tibble(
  query = "TITLE-ABS-KEY(graphene supercapacitor)",
  year  = as.integer(names(by_year)),
  n     = as.numeric(by_year)
)
class(tr) <- c("scopus_trend", class(tr))
tr
```

```{r eval = has_ggplot2, fig.alt = "A line and area chart of publications per year from 2015 to 2024, peaking in 2019", fig.width = 7.5, fig.height = 4}
plot_scopus_trend(tr)
```

The figure draws the same counts as the bar chart above, and the agreement is
worth noticing. A trend is something a record set already knows, and something
`scopus_trend()` can find out for a query whose records you never download. The
caption is the function's own attribution, since a `scopus_trend` object
ordinarily comes from the Search API, whereas this one was tallied from the
bundled stand-in. The
curve itself is flat because the query is a narrow phrase, some fifteen papers a
year over a decade. A broader query, 'graphene' on its own, would show the steep
growth the field is better known for, on a scale that runs into the offset
ceiling discussed below.

## Where a niche sits

A study that crosses two or more fields is best introduced by sizing those
fields and their overlap. Each parent literature may hold thousands of records
while their intersection holds a handful, which is the niche the study
occupies. `scopus_intersections()` counts a named set of concepts and any
requested intersections of them, at one count request per row, so the whole
landscape is as cheap as a few `scopus_count()` calls. Concept values may be
bare terms, wrapped in `field` for you, or complete field-tagged expressions,
used exactly as given.

```{r eval = FALSE}
sets <- scopus_intersections(
  concepts = c(
    "semantic priming"  = "semantic priming",
    "mental simulation" = "mental simulation",
    # A synonym set, given as a complete expression and used exactly as given.
    "embodied simulation" =
      'TITLE-ABS-KEY("mental simulation") OR TITLE-ABS-KEY("embodied simulation")'
  ),
  intersections = list(c("semantic priming", "mental simulation")),
  field = "TITLE-ABS-KEY"
)
plot_scopus_intersections(
  sets,
  highlight = sets$label[sets$type == "intersection"]
)
```

Here `field` leaves the third value untouched, since it already reads as a
complete field-tagged expression, so a concept can be a whole synonym set.

This one cannot be derived from a record set, because it counts whole
literatures, where a record set holds only the records already in hand, so here
the result is rebuilt in its own shape with illustrative counts, purely to show
the plot. The lollipop
chart uses a log-scale axis, so the small intersection stays legible beside its
large parent fields, and the highlighted row draws the eye to the niche itself.

```{r eval = has_ggplot2, fig.alt = "A log-scale lollipop chart showing three concepts and a small intersection, with the intersection highlighted", fig.width = 7.5, fig.height = 3}
sets <- tibble::tibble(
  label = c("semantic priming", "mental simulation", "embodied simulation",
            "semantic priming × mental simulation"),
  query = c("TITLE-ABS-KEY(semantic priming)",
            "TITLE-ABS-KEY(mental simulation)",
            'TITLE-ABS-KEY("mental simulation") OR TITLE-ABS-KEY("embodied simulation")',
            "(TITLE-ABS-KEY(semantic priming)) AND (TITLE-ABS-KEY(mental simulation))"),
  n = c(6600, 2100, 3400, 15),
  type = c("concept", "concept", "concept", "intersection"),
  size = c(1L, 1L, 1L, 2L),
  members = c("semantic priming", "mental simulation", "embodied simulation",
              "semantic priming; mental simulation")
)
class(sets) <- c("scopus_intersections", class(sets))
plot_scopus_intersections(
  sets,
  highlight = sets$label[sets$type == "intersection"]
)
```

## Reading the fuller record

The Search API returns a few fields per record. To read the abstract and the
fuller metadata for a record you already know, `scopus_abstract()` calls the
Abstract Retrieval API, by DOI or 'Scopus' identifier. A batch is resilient, so
an identifier that cannot be found yields a row of `NA`s with a warning rather
than stopping the run.

```{r eval = FALSE}
ab <- scopus_abstract(head(scopus_extract_dois(records), 2))
```

The result is a tibble of class `scopus_abstracts`, one row per identifier. To
show its shape without a key, here is a stand-in built from the two most-cited
records of the corpus, which supplies every column but the abstract itself. The
abstract is the one thing a live call adds and the corpus does not carry, so it
is marked as a placeholder, with nothing invented to fill it, and the columns
are listed by
name because the prose is far too wide to typeset.

```{r}
top2 <- records[order(-records$citations), ][1:2, ]
ab <- tibble::tibble(
  id          = top2$doi,
  scopus_id   = NA_character_,
  doi         = top2$doi,
  title       = top2$title,
  abstract    = "<abstract text, as the API returns it>",
  publication = top2$publication,
  year        = top2$year,
  citations   = top2$citations
)
class(ab) <- c("scopus_abstracts", class(ab))
names(ab)

ab[, c("title", "publication", "year", "citations")]
```

## Beyond five thousand records

A single Search API query returns at most its first 5000 records under the
ordinary offset paging. When you need the whole of a larger result set in one
pass, `scopus_fetch(cursor = TRUE)` follows the API's cursor instead, which has no
such ceiling.

```{r eval = FALSE}
recs <- scopus_fetch("TITLE-ABS-KEY(microplastics)", cursor = TRUE)
nrow(recs)
```

That query matched 38,374 records when this article was written (the literature
keeps growing), several times the offset ceiling, and the cursor retrieves all
of them in one call. The records then arrive in the API's deep-paging order,
which is the right trade for a complete harvest and the wrong one when you want
the most relevant few hundred. This is the one-call alternative to the
year-partitioned plan in the *Search plans and quota-aware retrieval* article. A
plan keeps each cell under the ceiling and preserves relevance order, whereas
`cursor = TRUE` harvests the whole set in a single pass. Reach for the plan when
you want cached, resumable cells and the cursor when you want the complete set
at once.
