---
title: "Author keywords and references"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Author keywords and references}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = FALSE, comment = "")
# Console colour carries no meaning on a rendered page. pkgdown turns it on for
# its own build, and the escape sequences then reach the reader as literal text,
# so colour is switched off here for a plain vignette render and a site build
# alike. The fixed width keeps tibbles inside the documentation column.
options(cli.num_colors = 1, cli.hyperlink = FALSE, crayon.enabled = FALSE,
        width = 80)
# Print data frames and tibbles as formatted tables.
local({
  kp <- function(x, ...) {
    if (any(vapply(x, is.list, logical(1)))) return(knitr::normal_print(x))
    knitr::knit_print(knitr::kable(x))
  }
  for (cls in c("data.frame", "tbl_df", "tbl")) {
    registerS3method("knit_print", cls, kp, envir = asNamespace("knitr"))
  }
})
```

Keyword co-occurrence and citation-network analysis both need something the
Search API alone does not return: a document's author-supplied keywords, and
its own reference list. This walks through retrieving both, what each costs,
and what your 'Scopus' entitlement needs to cover.

The API calls below are shown exactly as you would run them with a configured
key. No key is available when the site and the package are built, and Elsevier's
terms do not permit redistributing retrieved records in any case, so each call is
paired with its output assembled offline from one representative example, the
2015 *Nature* review 'Deep learning'. The corpus of real articles that the other
articles run on carries neither author keywords nor reference lists, the two
fields this one is about, so it cannot stand in here. The vignette therefore
renders the same with or without a key, and `scopus_has_key()` is the switch you
would use to run the calls for real.

```{r setup}
library(scopusflow)
```

## Author keywords from a search

The Search API's `COMPLETE` view carries an `authkeywords` field alongside the
usual title, DOI and date. Requesting it costs no extra request beyond
`COMPLETE`'s own smaller page size (25 records per page, against 200 for
`STANDARD`), which already means more requests, and so more quota, for the
same number of records.

```{r, eval = FALSE}
recs <- scopus_fetch("DOI(10.1038/nature14539)", view = "COMPLETE")
recs$authkeywords
```

```{r, include = FALSE}
# Representative COMPLETE-view result, assembled offline. authkeywords is the
# single string scopus_fetch() returns under view = "COMPLETE", one element
# per record, in Scopus' own " | "-delimited form. The bibliographic fields are
# the document's own. The 'Scopus' identifier and citation count are left NA,
# since nothing on the page prints them and a guess would be a fabrication.
recs <- tibble::tibble(
  entry_number = 1L, scopus_id = NA_character_, doi = "10.1038/nature14539",
  title = "Deep learning", authors = "LeCun Y.; Bengio Y.; Hinton G.",
  year = 2015L, date = "2015-05-28", publication = "Nature",
  citations = NA_integer_,
  authkeywords =
  "deep learning | neural networks | representation learning | backpropagation"
)
```

```{r}
recs$authkeywords
```

In development, this field came back `NA` even on a live, otherwise
fully-entitled key, for documents that do carry author keywords in 'Scopus'
itself. Those documents do have keywords, so what this points to is an
entitlement gap specific to this one field. If your own keywords come back all
`NA`, it is worth raising with your Scopus/Elsevier account contact.

## References via Abstract Retrieval

The reference list is not available from Search under any view. It needs
Abstract Retrieval's `FULL` or `REF` view, an entitlement separate from
ordinary abstract access and from Search access. This is a per-document
endpoint, so retrieving references for *n* documents costs *n* requests
against Abstract Retrieval's own, smaller weekly quota, separate from
Search's. Each row of the returned `references` data frame is one cited work.

```{r, eval = FALSE}
ab <- scopus_abstract(
  "10.1038/nature14539",
  view = "FULL", include = c("references", "keywords")
)
ab$references[[1]][, c("title", "authors", "source", "year")]
```

```{r, include = FALSE}
# Representative Abstract Retrieval result: one row carrying a `references`
# list-column (a data frame of cited works, in scopus_abstract()'s schema) and
# the same n_requests / quota attributes the function attaches.
refs <- tibble::tibble(
  position = as.character(1:4),
  # The 'Scopus' identifier of each cited work is left NA, as it is whenever
  # the API does not resolve one. Inventing one for the illustration would
  # misrepresent what a real call returns.
  id = NA_character_,
  doi = c("10.1109/5.726791", "10.1038/nature14236", NA, NA),
  title = c(
    "Gradient-based learning applied to document recognition",
    "Human-level control through deep reinforcement learning",
    "ImageNet classification with deep convolutional neural networks",
    "Learning representations by back-propagating errors"
  ),
  authors = c(
    "LeCun Y.; Bottou L.; Bengio Y.; Haffner P.",
    "Mnih V.; Kavukcuoglu K.; Silver D.",
    "Krizhevsky A.; Sutskever I.; Hinton G.",
    "Rumelhart D.; Hinton G.; Williams R."
  ),
  source = c(
    "Proceedings of the IEEE", "Nature",
    "Advances in Neural Information Processing Systems", "Nature"
  ),
  year = c(1998L, 2015L, 2012L, 1986L),
  # NA is what view = "FULL" actually returns for this column. Only view =
  # "REF" populates a cited work's own citation count.
  citedbycount = NA_integer_
)
ab <- tibble::tibble(
  id = "10.1038/nature14539",
  doi = "10.1038/nature14539",
  title = "Deep learning",
  year = 2015L,
  # scopus_abstract() joins keywords "; ", like its authors column. Only
  # Search's COMPLETE view uses the " | " form shown above.
  authkeywords =
  "deep learning; neural networks; representation learning; backpropagation",
  references = list(refs)
)
attr(ab, "n_requests") <- 1L
attr(ab, "quota") <- list(remaining = 24999L, reset = NA_character_)
```

```{r}
ab$references[[1]][, c("title", "authors", "source", "year")]
```

`view = "FULL"` is the recommended default. In development, it returned a
complete, correctly counted reference list for every document tried, while
`view = "REF"` returned an inconsistent, sometimes-truncated subset, on an
otherwise identical request made moments apart. `scopus_abstract()` warns
when the number of references returned does not match the document's own
reported count, so a partial list never arrives unannounced.

The number of requests spent and the remaining Abstract Retrieval quota are
attached as attributes, since this endpoint draws on a separate, smaller quota
that is easy to exhaust unnoticed.

```{r}
attr(ab, "n_requests")        # requests spent so far
attr(ab, "quota")$remaining   # Abstract Retrieval quota left
```

A key or subscription tier that does not cover the requested view raises a
`scopus_error_forbidden` condition that names the view, where a generic HTTP
failure would leave you guessing. It stops the whole batch too, since
entitlement is a property of the account, so the same refusal would only recur
on every remaining identifier.

For more than a handful of identifiers, pass `cache_dir` so an interrupted or
quota-limited batch resumes without re-spending quota already spent.

```r
ab <- scopus_abstract(
  dois, view = "FULL", include = c("references", "keywords"),
  cache_dir = "abstract-cache"
)
```

## A minimal, cross-tool corpus

`scopus_corpus()` combines a search result with this Abstract Retrieval step,
returning a minimal shape close to what OpenAlex's `works` API already
returns: `id`, `title`, `year`, `keywords` (a list-column of character
vectors) and `references` (a list-column of data frames), where bibliometrix
would give you semicolon-joined citation strings. It does not replace
[`as_bibliometrix()`], which keeps its own field-mapping convention for users
who want that.

```{r, eval = FALSE}
recs <- scopus_fetch("DOI(10.1038/nature14539)", max_results = 1)
corpus <- scopus_corpus(recs, view = "FULL")
corpus$keywords[[1]]
nrow(corpus$references[[1]])
```

```{r, include = FALSE}
# scopus_corpus() pairs each record with its Abstract Retrieval references and
# splits scopus_abstract()'s "; "-joined authkeywords string into a character
# vector per record.
corpus <- tibble::tibble(
  id = "10.1038/nature14539",
  title = "Deep learning",
  year = 2015L,
  keywords = list(trimws(strsplit(ab$authkeywords, ";", fixed = TRUE)[[1]])),
  references = list(refs)
)
```

```{r}
corpus$keywords[[1]]
nrow(corpus$references[[1]])
```

The keywords list-column is ready for the co-occurrence analysis this article
is named for, unnested and tallied into a per-keyword document count across the
corpus.

```{r}
sort(table(unlist(corpus$keywords)), decreasing = TRUE)
```

This costs one Abstract Retrieval request per record in `recs`, on top of
whatever retrieved `recs` in the first place.
