--- title: "Batch occurrence pipelines in biofetchR" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Batch occurrence pipelines in biofetchR} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} params: run_live: false --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", message = TRUE, warning = TRUE ) # Read the vignette parameter that controls whether live examples are run. # When run_live is FALSE, live GBIF and provider-download examples are skipped. # When run_live is TRUE, those examples are executed during rendering. run_live <- isTRUE(params$run_live) # Read GBIF credentials from environment variables. # This avoids writing usernames, passwords, or email addresses directly into # scripts, vignettes, or shared project files. gbif_user <- Sys.getenv("GBIF_USER") gbif_pwd <- Sys.getenv("GBIF_PWD") gbif_email <- Sys.getenv("GBIF_EMAIL") # Check whether all three GBIF credential fields are available. # Live GBIF examples are only attempted when this is TRUE. gbif_ready <- all(nzchar(c(gbif_user, gbif_pwd, gbif_email))) # Run a live example without stopping the whole vignette if an external service, # provider download, or GBIF request fails. The error message is printed so the # user can see what happened, and NULL is returned for that example. run_live_safely <- function(expr) { tryCatch( force(expr), error = function(e) { message("Live example did not complete: ", conditionMessage(e)) NULL } ) } # Print a clear message when a live example is skipped because credentials or # another required setup step are missing. skip_live_message <- function(reason) { message("Live example skipped: ", reason) invisible(NULL) } ``` ```{r load-package} library(biofetchR) ``` # Purpose This vignette introduces the main batch-processing workflows in `biofetchR`. It focuses on moving from a simple input table to processed occurrence outputs, using examples that can be adapted for small tests or larger data-production runs. The terrestrial and freshwater pipeline starts from a species-country table. In most cases, this means one row per target taxon and recipient country, with columns such as `species` and `iso2c`. For each row, `biofetchR` can request the matching GBIF occurrence records, import the completed download, apply the selected cleaning and thinning steps, assign records to the chosen spatial context, and return or export the processed data. For larger batches, outputs can be written directly to a designated folder, including separate files for individual species-country combinations. For smaller runs, users can also keep the processed records in the R session for immediate inspection. The marine pipeline follows a similar batch structure, but uses species-level marine input rather than species-country rows. Marine records can be assigned to a selected marine overlay, such as an EEZ-style layer, Marine Ecoregions of the World, IHO sea areas, or Large Marine Ecosystem (LME) polygons, which represent broad coastal and ocean regions used for ecosystem-scale marine summaries. This vignette covers two high-level functions: - `process_gbif_terrestrial_freshwater_pipeline()` for terrestrial and freshwater species-country workflows; - `process_gbif_marine_pipeline()` for marine occurrence workflows. The examples show how to prepare input tables, provide GBIF credentials, choose the main spatial-attribution setting, control cleaning and thinning, and inspect the summary outputs produced by the pipeline. Other vignettes cover specialist topics in more detail. Use the origin-evidence guide when the main focus is native-range, introduced-status or invasive-status classification; use the spatial-overlays guide for a wider comparison of overlay options; and use the auditing-and-scaling guide when planning larger runs. # Using the examples The examples in this vignette are written as project templates. They can be copied into an analysis script and adapted to different input tables, output folders, spatial settings, cleaning options and batch sizes. Some examples can submit live GBIF downloads or retrieve external spatial resources. These steps require internet access, GBIF credentials and access to the relevant provider sources. To keep the examples safe to read and render, live code is controlled by the `run_live` parameter. When `run_live = FALSE`, the vignette shows the workflow structure without submitting downloads or retrieving remote resources. When `run_live = TRUE`, the same examples are run against GBIF and the selected spatial-data providers. To run the live examples while rendering this vignette locally, use: ```{r render-live-version, eval = FALSE} rmarkdown::render( "vignettes/biofetchR-batch-pipelines.Rmd", params = list(run_live = TRUE) ) ``` This lets users choose how to use the vignette: as a static guide to the pipeline structure, or as a live workflow template for testing the package with real downloads and provider data. # Batch input tables The terrestrial and freshwater pipeline uses a species-country input table. Each row defines one taxon-country combination to process. The minimal input has two columns: - `species`: scientific name to resolve and submit to GBIF; - `iso2c`: two-letter ISO country code for the country to search. ```{r terrestrial-batch-input} terrestrial_batch <- data.frame( species = c( "Xenopus laevis", "Trachemys scripta", "Rattus rattus", "Sturnus vulgaris" ), iso2c = c( "FR", "GB", "GB", "DE" ), stringsAsFactors = FALSE ) terrestrial_batch ``` Project-specific columns can also be included in the input table. For example, users may want to keep taxonomic group, habitat, pathway, evidence source or review notes alongside the required `species` and `iso2c` columns. These fields can help organise larger batch files and may be carried into summaries or audit outputs where supported by the workflow. ```{r terrestrial-batch-input-with-metadata} terrestrial_batch_metadata <- data.frame( species = c( "Xenopus laevis", "Trachemys scripta" ), iso2c = c( "FR", "GB" ), taxon_group = c( "Amphibian", "Reptile" ), notes = c( "low-memory smoke test", "small batch example" ), stringsAsFactors = FALSE ) terrestrial_batch_metadata ``` Marine workflows use a simpler species-level input table. The marine pipeline downloads records by species first, then assigns the returned coordinates to the selected marine overlay. ```{r marine-batch-input} marine_batch <- data.frame( species = c( "Carcinus maenas", "Mnemiopsis leidyi", "Undaria pinnatifida" ), stringsAsFactors = FALSE ) marine_batch ``` This distinction matters when preparing batch files. Terrestrial and freshwater runs are usually organised around species-country combinations, whereas marine runs are usually organised around species and the marine overlay selected later in the workflow. # Credential and output setup GBIF downloads require user credentials. In project scripts, it is best to read these from environment variables rather than writing usernames, passwords or email addresses directly into the code. The examples in this vignette use three environment variables: - `GBIF_USER` - `GBIF_PWD` - `GBIF_EMAIL` The code below also creates an output folder for the vignette run. Here, `tempdir()` is used so the example can run without writing files into the working directory. For a real analysis, use a stable project folder instead, such as `outputs/gbif_batches`. ```{r credentials-and-output} # Create a temporary output folder for this vignette run. # In a real project, replace tempdir() with a stable project directory, # such as "outputs/gbif_batches". output_root <- file.path(tempdir(), "biofetchR_batch_vignette") # Create the output folder if it does not already exist. # recursive = TRUE also creates any missing parent folders. # showWarnings = FALSE avoids unnecessary messages if the folder already exists. dir.create(output_root, recursive = TRUE, showWarnings = FALSE) # Create a small credential-check table. # This does not print the credential values themselves; it only reports whether # each required environment variable is available. credential_status <- data.frame( variable = c("GBIF_USER", "GBIF_PWD", "GBIF_EMAIL"), available = nzchar(c(gbif_user, gbif_pwd, gbif_email)), stringsAsFactors = FALSE ) # Display the credential check and the output folder being used for this run. credential_status output_root ``` Before running a live batch, check that all three credential fields return `TRUE`. The output folder should also be changed from the temporary vignette location to a project directory that can be inspected later. For larger runs, keep the main files produced by the workflow together: the input table, exported occurrence files, `gbif_summary.csv`, any evidence or diagnostic outputs, and the package version used for the run. # Coordinate cleaning and spatial thinning Most batch workflows should apply coordinate cleaning before outputs are used for spatial interpretation. GBIF occurrence tables can include missing coordinates, zero-zero coordinates, impossible coordinates, duplicated points, high coordinate uncertainty, country-centroid records, institution coordinates or other records that are not suitable for spatial attribution. In `biofetchR`, coordinate cleaning is controlled with `apply_cleaning`. When enabled, the pipeline removes records that fail the selected coordinate-quality checks before records are exported. Spatial thinning is optional and is controlled with `apply_thinning` and `dist_km`. Thinning reduces dense clusters of nearby records by retaining records separated by a minimum distance. This can be useful when occurrence data are strongly concentrated around well-sampled areas, such as cities, roads, field stations, protected areas, ports, coastlines or frequently sampled localities. The thinning distance should match the scale of the analysis. A small distance may be suitable for local or subnational studies, while a larger distance may be more appropriate for regional or macroecological summaries. Thinning should be treated as a sampling-bias reduction step, not as a replacement for checking the spatial structure of the final dataset. ```{r cleaning-thinning-template, eval = FALSE} process_gbif_terrestrial_freshwater_pipeline( df = terrestrial_batch, output_dir = file.path(output_root, "cleaned_thinned_batch"), user = Sys.getenv("GBIF_USER"), pwd = Sys.getenv("GBIF_PWD"), email = Sys.getenv("GBIF_EMAIL"), region_source = "gadm", gadm_unit = 1, apply_cleaning = TRUE, apply_thinning = TRUE, dist_km = 5, export_summary = TRUE, store_in_memory = FALSE, return_all_results = FALSE ) ``` After a run, use `gbif_summary.csv` to compare the number of records returned from GBIF, retained after cleaning, and retained after thinning. For analyses that will be reported or reused, record whether cleaning was enabled, whether thinning was applied, and the thinning distance used. # A low-memory terrestrial smoke test Before running a larger batch, start with a small smoke test. This is a short run that checks whether the main workflow components are working together before processing many species-country combinations. The example below uses `Xenopus laevis` in France and assigns records to GADM level-1 units. It also enables coordinate cleaning, 5 km spatial thinning, taxonomy preparation, GRIIS evidence and native-range evidence in audit mode. This makes the test small enough to run quickly, while still checking the main setup requirements: GBIF credentials, provider access, cache folders, spatial attribution and output writing. ```{r low-memory-terrestrial-live, eval = run_live} if (!gbif_ready) { skip_live_message( "set GBIF_USER, GBIF_PWD, and GBIF_EMAIL to run the GBIF smoke test" ) } else { # Use one species-country pair for the first test run. # This is enough to check credentials, GBIF access, spatial attribution, # cleaning, thinning, evidence attachment, and output writing. low_memory_input <- data.frame( species = "Xenopus laevis", iso2c = "FR", stringsAsFactors = FALSE ) low_memory_result <- run_live_safely( process_gbif_terrestrial_freshwater_pipeline( df = low_memory_input, # Write all example outputs into the vignette output folder. output_dir = file.path(output_root, "terrestrial_low_memory"), # GBIF credentials are read from environment variables above. user = gbif_user, pwd = gbif_pwd, email = gbif_email, # Assign records to GADM level-1 administrative units. region_source = "gadm", gadm_unit = 1, cache_dir_gadm = file.path(output_root, "cache", "gadm"), # Keep the first run deliberately small and conservative. batch_size = 1, # Apply coordinate quality control and then thin clustered records. apply_cleaning = TRUE, apply_thinning = TRUE, dist_km = 5, # Prepare taxon names before submitting the GBIF download. prepare_taxonomy = TRUE, # Attach GRIIS evidence in audit mode. # Audit mode keeps the records and adds evidence fields for inspection. use_griis_status = TRUE, griis_filter_mode = "audit_only", griis_cache_dir = file.path(output_root, "cache", "griis"), # Attach native-range evidence from selected web sources. # For marine workflows, WoRMS can be useful alongside SInAS and GBIF. use_native_status = TRUE, use_native_web = TRUE, native_web_sources = c("sinas", "gbif", "worms"), native_web_cache_dir = file.path(output_root, "cache", "native_web"), native_filter_mode = "audit_only", # Write origin-evidence and processing summaries. reconcile_origin_evidence = TRUE, export_origin_audit = TRUE, export_summary = TRUE, # Write occurrence outputs to disk, but also return a result object # so the smoke test can be inspected immediately in the R session. return_all_results = TRUE, store_in_memory = FALSE, # Show progress messages during the live run. quiet = FALSE ) ) low_memory_result } ``` After a live run, inspect `gbif_summary.csv` first. This file provides a compact row-level summary of the run, including record counts, processing status, failure messages where relevant and the path to any exported occurrence file. ```{r low-memory-summary-live, eval = run_live} # Build the path to the summary file written by the smoke-test run. # The summary is stored inside the output directory used for this example. summary_path <- file.path( output_root, "terrestrial_low_memory", "gbif_summary.csv" ) # Read and inspect the summary only if the file was created. # This avoids an error if the previous live example was skipped or did not finish. if (file.exists(summary_path)) { # Import the summary table as a plain data frame. low_memory_summary <- read.csv(summary_path, stringsAsFactors = FALSE) # Display the most useful audit columns. # intersect() is used so the code still works if some optional columns are not # present in a particular pipeline run or package version. low_memory_summary[ , intersect( c( "species", # taxon processed "region_id", # spatial unit or recipient-region identifier "region_type", # spatial framework used, such as GADM "n_total", # records before cleaning "n_cleaned", # records retained after coordinate cleaning "n_thinned", # records retained after spatial thinning "status", # success, failure, or no-record status "fail_stage", # pipeline stage where a failure occurred, if any "fail_reason", # short explanation of the failure, if any "output_file" # exported occurrence file written for the row ), names(low_memory_summary) ) ] } else { # This message is shown when the live example was skipped or no summary was # written. It keeps the vignette readable without stopping execution. skip_live_message("summary file was not created by the previous live chunk") } ``` # Example summary output The table below is a small mock version of `gbif_summary.csv`. It shows the kinds of fields users should inspect after a batch run: the taxon processed, the region identifier, record counts before and after processing, the row status, any failure information and the exported file path. ```{r synthetic-terrestrial-summary} # In a real workflow, this information is read from gbif_summary.csv. summary_example <- data.frame( # Taxon processed by the pipeline. species = c("Xenopus laevis", "Trachemys scripta", "Rattus rattus"), # Recipient region or spatial unit associated with each row. # For this terrestrial example, these are ISO2 country codes. region_id = c("FR", "GB", "GB"), # Spatial framework used for the exported records. # Here, records are assumed to have been assigned to GADM level-1 units. region_type = c("GADM_1", "GADM_1", "GADM_1"), # Number of records returned before coordinate cleaning. n_total = c(1340, 220, 0), # Number of records retained after coordinate quality control. n_cleaned = c(1309, 198, 0), # Number of records retained after spatial thinning. n_thinned = c(861, 174, 0), # Processing outcome for each species-region row. status = c("success", "success", "no_records"), # Pipeline stage where a row failed or stopped, if applicable. fail_stage = c(NA, NA, "import"), # Short explanation of the failure or stopping reason, if applicable. fail_reason = c(NA, NA, "No records returned"), # Path to the exported occurrence file. # Rows with no usable records may not produce an output file. output_file = c( "Xenopus_laevis_FR_gadm1.csv", "Trachemys_scripta_GB_gadm1.csv", NA ), stringsAsFactors = FALSE ) # Display the example summary table. summary_example ``` A useful first check is the `status` column. Successful rows can usually be followed through to their exported occurrence files, while rows marked as failed, skipped or `no_records` should be reviewed before the batch output is used in an analysis. ```{r inspect-synthetic-terrestrial-summary} table(summary_example$status, useNA = "ifany") summary_example[summary_example$status != "success", ] ``` # Scaling to a terrestrial or freshwater batch After the one-row smoke test works, increase the number of species-country rows gradually. The `batch_size` argument controls how many species are submitted in each batch, which helps pace live GBIF requests and reduces the chance of running into provider limits during larger runs. Smaller batches also make it easier to identify which species or countries caused a problem if a download, import or spatial-attribution step fails. For larger workflows, `store_in_memory = FALSE` is recommended. This writes processed occurrence tables to the output folder instead of keeping every table in the active R session. Small batches can still be returned to the session for interactive inspection, but disk-based outputs are safer when processing many taxa or recipient countries. ```{r terrestrial-batch-live, eval = run_live} if (!gbif_ready) { skip_live_message("set GBIF credentials to run the terrestrial batch example") } else { # Run the same terrestrial/freshwater workflow on a slightly larger # species-country table. This is the next step after the one-row smoke test. terrestrial_batch_result <- run_live_safely( process_gbif_terrestrial_freshwater_pipeline( df = terrestrial_batch, # Write this batch into its own output folder so it is easy to inspect # separately from the smoke-test output. output_dir = file.path(output_root, "terrestrial_batch"), # GBIF credentials are read from environment variables defined earlier. user = gbif_user, pwd = gbif_pwd, email = gbif_email, # Assign records to GADM level-1 administrative units. region_source = "gadm", gadm_unit = 1, cache_dir_gadm = file.path(output_root, "cache", "gadm"), # Keep batches small in this example. # For larger projects, increase batch_size gradually after checking # that the first few rows complete successfully. batch_size = 1, # Apply coordinate cleaning and 5 km spatial thinning. apply_cleaning = TRUE, apply_thinning = TRUE, dist_km = 5, # Standardise and resolve taxon names before GBIF download submission. prepare_taxonomy = TRUE, # Attach GRIIS evidence without filtering rows out of this example. # This keeps the evidence available for inspection in the audit outputs. use_griis = TRUE, filter_griis_invasive = FALSE, griis_cache_dir = file.path(output_root, "cache", "griis"), # Attach native-range evidence from selected web sources. # audit_only keeps all records and writes evidence for later checking. use_native_range = TRUE, use_native_web = TRUE, native_web_sources = c("sinas", "gbif"), native_web_cache_dir = file.path(output_root, "cache", "native_web"), native_filter_mode = "audit_only", # Reconcile GRIIS and native-origin evidence and write audit outputs. reconcile_origin_evidence = TRUE, export_origin_audit = TRUE, # Write gbif_summary.csv for the batch. export_summary = TRUE, # For batch workflows, write occurrence tables to disk rather than # keeping all processed records in memory. store_in_memory = FALSE, # Return only the main run object because outputs are written to disk. return_all_results = FALSE, # Print progress messages during the live run. quiet = FALSE ) ) # Display the returned run object. terrestrial_batch_result } ``` For most projects, the key setting is `batch_size`. This controls how many species are submitted together in each GBIF batch, helping the workflow avoid submitting too many live requests at once. A smaller `batch_size` is usually safer for large or provider-dependent runs, while a larger value may be suitable once the input table and selected settings are straightforward. After the run, inspect `gbif_summary.csv` to check which rows completed, which rows returned no records, whether any provider or import steps failed, and where the exported occurrence files were written. This is usually enough to identify problem rows without needing to build the workflow through multiple staged runs. # Adding terrestrial and freshwater overlays The terrestrial and freshwater pipeline can assign records to more than one spatial framework in the same run. This is useful when one layer is needed for administrative reporting and another layer is needed for ecological or hydrological interpretation. In the example below, GADM level 1 is kept as the administrative reference, while FEOW is added as a freshwater ecoregion layer. The important change is `region_source = c("gadm", "feow")`. Setting `overlay_mode = "dual_separate"` keeps the two joins separate, so users can inspect the GADM and FEOW attributions independently rather than forcing them into a single combined spatial unit. ```{r terrestrial-overlay-live, eval = run_live} if (!gbif_ready) { skip_live_message("set GBIF credentials to run the overlay example") } else { # Run a small overlay example using one species-country pair. # This keeps the example manageable while showing how to combine # administrative and ecological spatial attribution. feow_result <- run_live_safely( process_gbif_terrestrial_freshwater_pipeline( # Input table for the example. # The species is processed in France as the recipient country. df = data.frame( species = "Xenopus laevis", iso2c = "FR", stringsAsFactors = FALSE ), # Write the overlay example into its own output folder. output_dir = file.path(output_root, "terrestrial_feow"), # GBIF credentials are read from environment variables defined earlier. user = gbif_user, pwd = gbif_pwd, email = gbif_email, # Use both GADM and FEOW as spatial frameworks. # GADM provides administrative attribution. # FEOW provides freshwater ecoregion attribution. region_source = c("gadm", "feow"), # Keep the GADM and FEOW joins as separate outputs. # This is useful when the two spatial frameworks should be inspected # independently rather than merged into one combined unit. overlay_mode = "dual_separate", # Use GADM level 1 for subnational administrative attribution. gadm_unit = 1, # Cache downloaded or loaded spatial layers so they can be reused. cache_dir_gadm = file.path(output_root, "cache", "gadm"), feow_cache_dir = file.path(output_root, "cache", "feow"), # Keep the example to one GBIF request. batch_size = 1, # Apply coordinate cleaning and 5 km spatial thinning before export. apply_cleaning = TRUE, apply_thinning = TRUE, dist_km = 5, # Write gbif_summary.csv for the overlay run. export_summary = TRUE, # Write processed occurrence tables to disk rather than storing # all records in the active R session. store_in_memory = FALSE, # Return only the main run object because detailed outputs are written # to the output directory. return_all_results = FALSE, # Show progress messages during the live run. quiet = FALSE ) ) # Display the returned run object. feow_result } ``` This pattern can be adapted to other supported overlays by changing `region_source` and the relevant cache settings. Use separate overlays when the spatial frameworks answer different questions, such as administrative reporting versus freshwater ecoregion context. # Marine batch pipeline Marine workflows use a separate pipeline because the input table is organised by species rather than by species-country combinations. Records are downloaded for each species, then assigned to the selected marine overlay after the coordinates have been imported. This is the basic marine batch pattern. Here, records are assigned to EEZ polygons, while more detailed marine-overlay options are covered in the dedicated marine workflow vignette. ```{r marine-batch-live, eval = run_live} if (!gbif_ready) { skip_live_message("set GBIF credentials to run the marine batch example") } else { # Run a small marine batch example. # Unlike the terrestrial workflow, the marine pipeline starts from a species # list and assigns records to marine regions after the GBIF records are imported. marine_batch_result <- run_live_safely( process_gbif_marine_pipeline( df = marine_batch, # Write the marine batch outputs into their own folder. output_dir = file.path(output_root, "marine_batch"), # GBIF credentials are read from environment variables defined earlier. user = gbif_user, pwd = gbif_pwd, email = gbif_email, # Assign imported occurrence records to Exclusive Economic Zones. # Other marine overlays can be used in the dedicated marine workflow. overlay = "eez", # Cache the marine overlay so it can be reused in later runs. overlay_cache_dir = file.path(output_root, "cache", "marine_eez"), # Allow the example to continue more flexibly if a remote overlay source # is temporarily unavailable or needs to be resolved from cache. strict_overlay_loading = FALSE, # Keep the example small by processing one species at a time. batch_size = 1, # Apply coordinate cleaning and 5 km spatial thinning before export. apply_cleaning = TRUE, apply_thinning = TRUE, dist_km = 5, # Standardise and resolve taxon names before submitting GBIF downloads. prepare_taxonomy = TRUE, # Attach GRIIS evidence in audit mode. # Audit mode keeps the records and adds evidence fields for inspection. use_griis_status = TRUE, griis_filter_mode = "audit_only", # Attach native-range evidence from selected web sources. # For marine workflows, WoRMS can be useful alongside SInAS and GBIF. use_native_status = TRUE, use_native_web = TRUE, native_web_sources = c("sinas", "gbif", "worms"), native_filter_mode = "audit_only", # Compare and reconcile GRIIS and native-origin evidence where available. reconcile_griis_native = TRUE, # Write gbif_summary.csv for the marine batch. export_summary = TRUE, # Write processed occurrence tables to disk rather than storing all # records in the active R session. store_in_memory = FALSE, # Return only the main run object because detailed outputs are written # to the output directory. return_all_results = FALSE, # Show progress messages during the live run. quiet = FALSE ) ) # Display the returned run object. marine_batch_result } ``` The main difference from the terrestrial and freshwater examples is where the spatial filter is applied. In the terrestrial workflow, the country code is part of the GBIF request. In the marine workflow, the overlay assignment happens after the species-level records have been imported. # Output files and audit trail A batch run produces more than the processed occurrence tables. The supporting files are useful because they show what was submitted, what was returned, which steps were applied, and where each output was written. For project work, keep the main run materials together: - the original input table; - the pipeline call and settings used; - `gbif_summary.csv`; - exported occurrence files; - taxonomy outputs, if taxonomy preparation was enabled; - GRIIS or native-range evidence tables, if these options were used; - cache notes or source details for external spatial layers; - the `biofetchR` package version and R session information. Start with `gbif_summary.csv` after each run. This file gives a row-level view of the batch, including record counts, processing status, failure messages where relevant, and exported file paths. Rows marked as `success` should have sensible record counts and an output file path. Rows marked as failed, skipped or `no_records` should be checked before the exported data are used. ```{r output-audit-example} # Create a compact audit view from the summary table. # This keeps the columns most useful for checking whether each row completed # successfully and where the exported occurrence file was written. audit_view <- summary_example[ , intersect( c( "species", # taxon processed "region_id", # recipient region or spatial unit "region_type", # spatial framework used "status", # processing outcome for the row "fail_stage", # stage where the row failed or stopped, if applicable "fail_reason", # explanation of the failure or stopping reason "output_file" # exported occurrence file path, if one was written ), names(summary_example) ) ] # Display the audit view. audit_view ``` This compact view is useful when scanning larger runs because it separates rows that produced usable files from rows that need follow-up. The full summary table can then be used to inspect record counts, download information and any additional fields written by the selected workflow settings. # Practical recommendations For small exploratory analyses, it is usually enough to run the pipeline, inspect `gbif_summary.csv`, and open the exported occurrence files. For larger batches, the main priorities are to pace GBIF requests sensibly, keep outputs organised, and preserve the files needed to check how each row was processed. Recommended practice is to: - run a one-row smoke test before a larger live run; - choose a conservative `batch_size` when submitting many species to GBIF; - use `store_in_memory = FALSE` when processing larger batches; - write outputs to a stable project directory rather than a temporary folder; - keep provider caches separate from final analysis outputs; - keep origin and invasive-status evidence in `audit_only` mode until the evidence fields have been reviewed; - save the original input table with the exported dataset; - use `gbif_summary.csv` as the main record of row status, record counts, failures and output paths; - rerun failed or incomplete rows separately after checking `fail_stage` and `fail_reason`. These habits keep the workflow manageable without requiring users to build the analysis through multiple staged runs. The key is to keep the batch inputs, summary file, exported occurrence tables and evidence outputs together so the processed dataset can be inspected and reused later.