Principles of Reproducible Research

Science (2011) Reproducible Research in Computational Science Peng, Roger D.

Providing sufficient materials and information for reproducing your results

Making it easy for existing and new collaborators to understand, contribute and build upon a project

Your most important collaborator is your past-self but he does not respond to e-mails

The journey of reproducibility in R

Case study

• Script to compare alternative confidence intervals for a ratio of variables

• Use survey data on households

  Variable	Name
  Number of dogs	Ndog
  Number of humans	Nhum
  Urban / Rural	zone

• Are dog-human ratios in rural and urban areas different?

Household dataset

# A tibble: 334 × 5
   id    zone    Nhum  Ndog dh_ratio
   <chr> <chr>  <dbl> <dbl>    <dbl>
 1 18    URBAIN     3     0    0    
 2 20    URBAIN     1     0    0    
 3 79    URBAIN     4     0    0    
 4 339   URBAIN     3     0    0    
 5 418   URBAIN     6     0    0    
 6 419   URBAIN     4     0    0    
 7 551   URBAIN     4     0    0    
 8 591   URBAIN     4     0    0    
 9 740   URBAIN     6     1    0.167
10 883   URBAIN     2     0    0    
# … with 324 more rows

1. The R script

Initial script

rm(list = ls())
setwd('C:\Users\facu\My Documents\some project')


library(tidyverse)

library(readxl)
dat <- read_excel("HumanDogRatioData.xlsx")

dat <- dat |>
    select(Idmenage, zone = ru_urb, Nbh, Ndog) |>
    mutate(dh_ratio = Ndog/Nbh)

str(dat)

dat |>
  pivot_longer(
    Nbh:dh_ratio,
    names_to = "variable",
    values_to = "value"
  ) |>
  ggplot(aes(value)) +
  geom_histogram(bins = 15) +
  facet_grid(zone ~ variable, scales = "free")

res_normal <-
  dat |>
  select(zone, y = dh_ratio) |>
  group_by(zone) |>
  summarise(
    Mean = mean(y),
    Variance = var(y),
    CI95_a = mean(y) - qt(1 - 0.05 / 2, length(y) - 1) * sqrt(var(y) / length(y)),
    CI95_b = mean(y) + qt(1 - 0.05 / 2, length(y) - 1) * sqrt(var(y) / length(y)),
    .groups = "drop"
  )
res_normal

library(kableExtra)
kbl(
  res_normal,
  booktabs = TRUE,
  digits = 3,
  caption = "Results using the Normal asymptotic approximation."
)


## confint-ratio-boot
library(boot)
get_estimates <- function(x, i) {
  i_u <- i[x$zone[i] == "URBAIN"]
  i_r <- i[x$zone[i] == "RURAL"]
  m_u <- mean(x$dh_ratio[i_u])
  m_r <- mean(x$dh_ratio[i_r])
  c(RURAL = m_r, URBAIN = m_u, `U-R` = m_u - m_r)
}
data_boot <- boot(dat, get_estimates, R = 1e4)


## res-boot
res_boot <-
  data_boot$t0 |>
  enframe(name = "zone", value = "Mean") |>
  left_join(
    map_dfr(
      setNames(seq_along(data_boot$t0), names(data_boot$t0)),
      ~ boot.ci(data_boot, index = ., type = "basic")$basic[1, 4:5]
    ) |>
      add_column(side = c("CI95_a", "CI95_b")) |>
      pivot_longer(
        cols = -side,
        names_to = "zone",
        values_to = "value"
      ) |>
      pivot_wider(
        names_from = "side",
        values_from = "value"
      ),
    by = "zone"
  )

kbl(
  res_boot,
  booktabs = TRUE,
  digits = 3,
  caption = "Results using the basic Bootstrap method."
)


## fm1
fm1 <- glm(
  formula = Ndog ~ zone,
  family = "poisson",
  offset = dat$Nbh,
  data = dat
)


## fm1-summary
summary(fm1)


## table-estimates-model
exp(c(coef(fm1), sum(coef(fm1)))[c(1, 3, 2)]) |>
  setNames(c("RURAL", "URBAIN", "U/R")) |>
  enframe(name = "zone", value = "Mean")


sim_pars <- tribble(
  ~param, ~value,
  "N", 334,
  "b0", -2,
  "b1", -.2
)


sim_pars |>
  kbl(
    booktabs = TRUE,
    caption = "Parameters used for simulating data."
  )

zone <- sample(c("Urban", "Rural"), size = with(sim_pars, value[param == "N"]), replace = TRUE)
log_lambda <- with(sim_pars, value[param == "b0"]) + with(sim_pars, value[param == "b1"]) * (zone == "Urban")

x <- 1 + rpois(with(sim_pars, value[param == "N"]), lambda = 3)
y <- rpois(with(sim_pars, value[param == "N"]), lambda = x * exp(log_lambda))

sim_data <- data.frame(
  id = seq.int(with(sim_pars, value[param == "N"])),
  zone = zone,
  x = x,
  y = y,
  r = y/x
)

library(hrbrthemes)
theme_set(theme_ipsum(grid = "Y"))

## Sample distributions of __simulated__ survey data (dog-human ratio (r),
## number of humans (x) and number of dogs (y)) by zone.
sim_data |>
  pivot_longer(
    x:r,
    names_to = "variable",
    values_to = "value"
  ) |>
  ggplot(aes(value)) +
  geom_histogram(bins = 15) +
  facet_grid(zone ~ variable, scales = "free")

Code that only runs in your computer

• Absolute paths

• Files (images, data files, libraries) not bundled with the code

• Removing objects (rm()) do not fully clean up the session ¹

rm(list = ls())
setwd('C:\Users\facu\My Documents\some project')

library(tidyverse)
library(readxl)
raw_data <- read_excel("HumanDogRatioData.xlsx")

1. e.g. options, loaded packages and environmental variables remain

Organise all necessary files within a coherent and intuitive directory tree

E.g.

.
├── data/
├── doc/
├── reports/
├── src/
└── Readme.md

• Start the session from the project’s root

• Sharing the bundle ensures nothing is left behind

• Use paths relative to project’s root

Code that runs only on your platform

• Platform-specific packages or functionality¹

• Encoding file or object names with special characters

1. E.g. ODBC, multi-threading, …

Good practices

• Favour cross-platform functionality if possible¹

  Otherwise, acknowledge and justify the limitation

• Favour standard encoding systems (UTF-8)

• Avoid special characters or spaces in names

  (see naming things, by J Bryan)

1. E.g. use {readxl} for importing from Excel, rather than {RODBC}

Code structure

• Structure code into sections (set up, load, prepare, analyse, results)

• Use comments for sectioning and for explanations

• Remove residual code (failed tests, interactive calls)

• Make code fit into 70-80 columns

• Meaningful object names

Final script

# Confidence intervals for the dog-human ratio in households
# Is this ratio different in urban and rural areas?
# 2022 Facundo Muñoz <facundo.munoz@cirad.fr>

# Packages -------------------------------------------------
pacman::p_load(
  "boot",
  "furrr",
  "here",
  "hrbrthemes",
  "janitor",
  "kableExtra",
  "knitr",
  "readxl",
  "tidyverse"
)


# Setup ----------------------------------------------------
knitr::opts_chunk$set(
  echo = FALSE,
  cache = FALSE,
  dev = ifelse(is_latex_output(), "cairo_pdf", "png"),
  dpi = 300
)

theme_set(theme_ipsum(grid = "Y"))


# Parameters -----------------------------------------------
sim_pars <- tribble(
  ~param, ~value,
  "N", 334,
  "b0", -2,
  "b1", -.2
)



# Data -----------------------------------------------------

## Real household dataset
household_data_file <- here::here("data/HumanDogRatioData.xlsx")

household_raw_data <- read_excel(household_data_file)

household_data <- household_raw_data |>
    select(Idmenage, zone = ru_urb, Nbh, Ndog) |>
    mutate(dh_ratio = Ndog/Nbh)

## Simulated dataset
zone <- sample(
  c("Urban", "Rural"),
  size = with(sim_pars, value[param == "N"]),
  replace = TRUE)
log_lambda <- with(sim_pars, value[param == "b0"]) + 
  with(sim_pars, value[param == "b1"]) * (zone == "Urban")

x <- 1 + rpois(
  with(sim_pars, value[param == "N"]),
  lambda = 3
)
y <- rpois(
  with(sim_pars, value[param == "N"]),
  lambda = x * exp(log_lambda)
)

sim_data <- data.frame(
  id = seq.int(with(sim_pars, value[param == "N"])),
  zone = zone,
  x = x,
  y = y,
  r = y/x
)


# Results --------------------------------------------------

## Sample distributions of survey data
## (dog-human ratio, number of humans and
## number of dogs) by zone.
household_data |>
  pivot_longer(
    Nbh:dh_ratio,
    names_to = "variable",
    values_to = "value"
  ) |>
  ggplot(aes(value)) +
  geom_histogram(bins = 15) +
  facet_grid(zone ~ variable, scales = "free")


## Classical asymptotic interval

## Confidence Interval for a mean
## 
## Compute Wang's CI for a population mean
## 
## @param x numeric vector
## @param conf_level numeric value between 0 and 1
## 
## @examples
## confint_mean(1:10, conf_level = .9)
confint_mean <- function(x, conf_level = 0.95) {
  alpha <- 1 - conf_level
  hatx <- mean(x)
  s2 <- var(x)
  n <- length(x)
  ta2 <- qt(alpha/2, n - 1, lower.tail = FALSE)
  
  return(hatx + ta2*sqrt(s2/n) * c(-1, 1))
}

## res-normal
res_normal <-
  household_data |>
  select(zone, y = dh_ratio) |>
  group_by(zone) |>
  summarise(
    Mean = mean(y),
    Variance = var(y),
    CI95_a = confint_mean(y)[1],
    CI95_b = confint_mean(y)[2],
    .groups = "drop"
  )


kbl(
  res_normal,
  booktabs = TRUE,
  digits = 3,
  caption = "Results using the Normal asymptotic approximation."
)


## Bootstrap interval

## confint-ratio-boot
get_estimates <- function(x, i) {
  i_u <- i[x$zone[i] == "URBAIN"]
  i_r <- i[x$zone[i] == "RURAL"]
  m_u <- mean(x$dh_ratio[i_u])
  m_r <- mean(x$dh_ratio[i_r])
  c(RURAL = m_r, URBAIN = m_u, `U-R` = m_u - m_r)
}
clean_data_boot <- boot(household_data, get_estimates, R = 1e4)

## res-boot
res_boot <-
  clean_data_boot$t0 |>
  enframe(name = "zone", value = "Mean") |>
  left_join(
    map_dfr(
      setNames(
        seq_along(clean_data_boot$t0),
        names(clean_data_boot$t0)
      ),
      ~ boot.ci(
        clean_data_boot,
        index = .,
        type = "basic"
      )$basic[1, 4:5]
    ) |>
      add_column(side = c("CI95_a", "CI95_b")) |>
      pivot_longer(
        cols = -side,
        names_to = "zone",
        values_to = "value"
      ) |>
      pivot_wider(
        id = "zone",
        names_from = "side",
        values_from = "value"
      ),
    by = "zone"
  )



kbl(
  res_boot,
  booktabs = TRUE,
  digits = 3,
  caption = "Results using the basic Bootstrap method."
)


## Modelling

## fm1
fm1 <- glm(
  formula = Ndog ~ zone,
  family = "poisson",
  offset = household_data$Nbh,
  data = household_data
)


## fm1-summary
summary(fm1)


## table-estimates-model
exp(c(coef(fm1), sum(coef(fm1)))[c(1, 3, 2)]) |>
  setNames(c("RURAL", "URBAIN", "U/R")) |>
  enframe(name = "zone", value = "Mean")


sim_pars |>
  kbl(
    booktabs = TRUE,
    caption = "Parameters used for simulating data."
  )


## Sample distributions of simulated survey data
## (dog-human ratio (r), number of humans (x) and
## number of dogs (y)) by zone.
sim_data |>
  pivot_longer(
    x:r,
    names_to = "variable",
    values_to = "value"
  ) |>
  ggplot(aes(value)) +
  geom_histogram(bins = 15) +
  facet_grid(zone ~ variable, scales = "free")

Use functions

• Don’t Repeat Yourself (DRY) principle

• More concise and modular code

• More rigorous documentation and testing

• More readable

res_normal <-
  clean_data |>
  select(zone, y = dh_ratio) |>
  group_by(zone) |>
  summarise(
    Mean = mean(y),
    Variance = var(y),
    CI95_a = mean(y) - qt(1 - 0.05 / 2, length(y) - 1) * sqrt(var(y) / length(y)),
    CI95_b = mean(y) + qt(1 - 0.05 / 2, length(y) - 1) * sqrt(var(y) / length(y)),
    .groups = "drop"
  )

## res-normal
res_normal <-
  clean_data |>
  select(zone, y = dh_ratio) |>
  group_by(zone) |>
  summarise(
    Mean = mean(y),
    Variance = var(y),
    CI95_a = confint_mean(y)[1],
    CI95_b = confint_mean(y)[2],
    .groups = "drop"
  )

## Confidence Interval for a mean
## 
## Compute Wang's CI for a population mean
## 
## @param x numeric vector
## @param conf_level numeric value between 0 and 1
## 
## @examples
## confint_mean(1:10, conf_level = .9)
confint_mean <- function(x, conf_level = 0.95) {
  alpha <- 1 - conf_level
  hatx <- mean(x)
  s2 <- var(x)
  n <- length(x)
  ta2 <- qt(alpha/2, n - 1, lower.tail = FALSE)
  
  return(hatx + ta2*sqrt(s2/n) * c(-1, 1))
}

Summary of basic reproducible practices in R scripts

• Avoid code that runs only in your computer

• Use relative paths within a project structure

• Restart the session to start over

• Structure your code into clearly defined sections

• Ruthlessly remove unnecessary code

• Liberally use comments

Let’s pause for quesitions

2. {renv}

Motivation

Packages and versions

• Any typical script depends on a dozen packages

• Package installation not always frictionless

• Upgrading packages can break previous code

Wokflow

• renv::init(): Initialize a new project-local environment with a private R library¹

• Business as usual, installing and removing new R packages as needed

• renv::snapshot(): Update the state of the project library to the lockfile

• renv::restore(): Revert to the previous state as encoded in the lockfile if your attempts to update packages introduced some new problems.

1. Creates a lockfile renv.lock with dependencies detected in the project

The lockfile for these slides

{
  "R": {
    "Version": "4.2.0",
    "Repositories": [
      {
        "Name": "CRAN",
        "URL": "https://cloud.r-project.org"
      },
      {
        "Name": "INLA",
        "URL": "http://inla.r-inla-download.org/R/testing"
      },
      {
        "Name": "breedR",
        "URL": "http://famuvie.github.io/breedR"
      }
    ]
  },
  "Packages": {
    "DBI": {
      "Package": "DBI",
      "Version": "1.1.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "dcd1743af4336156873e3ce3c950b8b9",
      "Requirements": []
    },
    "MASS": {
      "Package": "MASS",
      "Version": "7.3-57",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "71476c1d88d1ebdf31580e5a257d5d31",
      "Requirements": []
    },
    "Matrix": {
      "Package": "Matrix",
      "Version": "1.4-1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "699c47c606293bdfbc9fd78a93c9c8fe",
      "Requirements": [
        "lattice"
      ]
    },
    "R.methodsS3": {
      "Package": "R.methodsS3",
      "Version": "1.8.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "4bf6453323755202d5909697b6f7c109",
      "Requirements": []
    },
    "R.oo": {
      "Package": "R.oo",
      "Version": "1.24.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "5709328352717e2f0a9c012be8a97554",
      "Requirements": [
        "R.methodsS3"
      ]
    },
    "R.utils": {
      "Package": "R.utils",
      "Version": "2.11.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "a7ecb8e60815c7a18648e84cd121b23a",
      "Requirements": [
        "R.methodsS3",
        "R.oo"
      ]
    },
    "R6": {
      "Package": "R6",
      "Version": "2.5.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "470851b6d5d0ac559e9d01bb352b4021",
      "Requirements": []
    },
    "RColorBrewer": {
      "Package": "RColorBrewer",
      "Version": "1.1-3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "45f0398006e83a5b10b72a90663d8d8c",
      "Requirements": []
    },
    "Rcpp": {
      "Package": "Rcpp",
      "Version": "1.0.8.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "32e79b908fda56ee57fe518a8d37b864",
      "Requirements": []
    },
    "Rttf2pt1": {
      "Package": "Rttf2pt1",
      "Version": "1.3.10",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "8952819f94cd9fce521f9b1cb558f8a4",
      "Requirements": []
    },
    "askpass": {
      "Package": "askpass",
      "Version": "1.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "e8a22846fff485f0be3770c2da758713",
      "Requirements": [
        "sys"
      ]
    },
    "assertthat": {
      "Package": "assertthat",
      "Version": "0.2.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "50c838a310445e954bc13f26f26a6ecf",
      "Requirements": []
    },
    "astretools": {
      "Package": "astretools",
      "Version": "0.0-1",
      "Source": "Repository",
      "Repository": null,
      "Hash": "63e573218e29ef13aea92bc071c8f96c",
      "Requirements": [
        "fs",
        "rmarkdown",
        "xaringan"
      ]
    },
    "backports": {
      "Package": "backports",
      "Version": "1.4.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c39fbec8a30d23e721980b8afb31984c",
      "Requirements": []
    },
    "base64enc": {
      "Package": "base64enc",
      "Version": "0.1-3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "543776ae6848fde2f48ff3816d0628bc",
      "Requirements": []
    },
    "bib2df": {
      "Package": "bib2df",
      "Version": "1.1.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "f2cf33875b859a757ad919a4471cdc0f",
      "Requirements": [
        "dplyr",
        "httr",
        "humaniformat",
        "stringr"
      ]
    },
    "bit": {
      "Package": "bit",
      "Version": "4.0.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "f36715f14d94678eea9933af927bc15d",
      "Requirements": []
    },
    "bit64": {
      "Package": "bit64",
      "Version": "4.0.5",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "9fe98599ca456d6552421db0d6772d8f",
      "Requirements": [
        "bit"
      ]
    },
    "blob": {
      "Package": "blob",
      "Version": "1.2.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "10d231579bc9c06ab1c320618808d4ff",
      "Requirements": [
        "rlang",
        "vctrs"
      ]
    },
    "boot": {
      "Package": "boot",
      "Version": "1.3-28",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "0baa960e3b49c6176a4f42addcbacc59",
      "Requirements": []
    },
    "broom": {
      "Package": "broom",
      "Version": "0.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "fe13cb670e14da57fd7a466578db8ce5",
      "Requirements": [
        "backports",
        "dplyr",
        "ellipsis",
        "generics",
        "ggplot2",
        "glue",
        "purrr",
        "rlang",
        "stringr",
        "tibble",
        "tidyr"
      ]
    },
    "bslib": {
      "Package": "bslib",
      "Version": "0.3.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "56ae7e1987b340186a8a5a157c2ec358",
      "Requirements": [
        "htmltools",
        "jquerylib",
        "jsonlite",
        "rlang",
        "sass"
      ]
    },
    "callr": {
      "Package": "callr",
      "Version": "3.7.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "461aa75a11ce2400245190ef5d3995df",
      "Requirements": [
        "R6",
        "processx"
      ]
    },
    "cellranger": {
      "Package": "cellranger",
      "Version": "1.1.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "f61dbaec772ccd2e17705c1e872e9e7c",
      "Requirements": [
        "rematch",
        "tibble"
      ]
    },
    "cli": {
      "Package": "cli",
      "Version": "3.3.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "23abf173c2b783dcc43379ab9bba00ee",
      "Requirements": [
        "glue"
      ]
    },
    "clipr": {
      "Package": "clipr",
      "Version": "0.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "3f038e5ac7f41d4ac41ce658c85e3042",
      "Requirements": []
    },
    "codetools": {
      "Package": "codetools",
      "Version": "0.2-18",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "019388fc48e48b3da0d3a76ff94608a8",
      "Requirements": []
    },
    "colorspace": {
      "Package": "colorspace",
      "Version": "2.0-3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "bb4341986bc8b914f0f0acf2e4a3f2f7",
      "Requirements": []
    },
    "cpp11": {
      "Package": "cpp11",
      "Version": "0.4.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "fa53ce256cd280f468c080a58ea5ba8c",
      "Requirements": []
    },
    "crayon": {
      "Package": "crayon",
      "Version": "1.5.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "8dc45fd8a1ee067a92b85ef274e66d6a",
      "Requirements": []
    },
    "crosstalk": {
      "Package": "crosstalk",
      "Version": "1.2.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "6aa54f69598c32177e920eb3402e8293",
      "Requirements": [
        "R6",
        "htmltools",
        "jsonlite",
        "lazyeval"
      ]
    },
    "curl": {
      "Package": "curl",
      "Version": "4.3.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "022c42d49c28e95d69ca60446dbabf88",
      "Requirements": []
    },
    "data.table": {
      "Package": "data.table",
      "Version": "1.14.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "36b67b5adf57b292923f5659f5f0c853",
      "Requirements": []
    },
    "dbplyr": {
      "Package": "dbplyr",
      "Version": "2.1.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "1f37fa4ab2f5f7eded42f78b9a887182",
      "Requirements": [
        "DBI",
        "R6",
        "assertthat",
        "blob",
        "dplyr",
        "ellipsis",
        "glue",
        "lifecycle",
        "magrittr",
        "purrr",
        "rlang",
        "tibble",
        "tidyselect",
        "vctrs",
        "withr"
      ]
    },
    "digest": {
      "Package": "digest",
      "Version": "0.6.29",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "cf6b206a045a684728c3267ef7596190",
      "Requirements": []
    },
    "dplyr": {
      "Package": "dplyr",
      "Version": "1.0.9",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "f0bda1627a7f5d3f9a0b5add931596ac",
      "Requirements": [
        "R6",
        "generics",
        "glue",
        "lifecycle",
        "magrittr",
        "pillar",
        "rlang",
        "tibble",
        "tidyselect",
        "vctrs"
      ]
    },
    "dtplyr": {
      "Package": "dtplyr",
      "Version": "1.2.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "f5d195cd5fcc0a77499d9da698ef2ea3",
      "Requirements": [
        "crayon",
        "data.table",
        "dplyr",
        "ellipsis",
        "glue",
        "lifecycle",
        "rlang",
        "tibble",
        "tidyselect",
        "vctrs"
      ]
    },
    "ellipsis": {
      "Package": "ellipsis",
      "Version": "0.3.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "bb0eec2fe32e88d9e2836c2f73ea2077",
      "Requirements": [
        "rlang"
      ]
    },
    "evaluate": {
      "Package": "evaluate",
      "Version": "0.15",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "699a7a93d08c962d9f8950b2d7a227f1",
      "Requirements": []
    },
    "extrafont": {
      "Package": "extrafont",
      "Version": "0.18",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "07a8a50195fc77e8690fd8bc4e87e6a0",
      "Requirements": [
        "Rttf2pt1",
        "extrafontdb"
      ]
    },
    "extrafontdb": {
      "Package": "extrafontdb",
      "Version": "1.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "a861555ddec7451c653b40e713166c6f",
      "Requirements": []
    },
    "fansi": {
      "Package": "fansi",
      "Version": "1.0.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "83a8afdbe71839506baa9f90eebad7ec",
      "Requirements": []
    },
    "farver": {
      "Package": "farver",
      "Version": "2.1.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c98eb5133d9cb9e1622b8691487f11bb",
      "Requirements": []
    },
    "fastmap": {
      "Package": "fastmap",
      "Version": "1.1.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "77bd60a6157420d4ffa93b27cf6a58b8",
      "Requirements": []
    },
    "forcats": {
      "Package": "forcats",
      "Version": "0.5.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "81c3244cab67468aac4c60550832655d",
      "Requirements": [
        "ellipsis",
        "magrittr",
        "rlang",
        "tibble"
      ]
    },
    "fs": {
      "Package": "fs",
      "Version": "1.5.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "7c89603d81793f0d5486d91ab1fc6f1d",
      "Requirements": []
    },
    "furrr": {
      "Package": "furrr",
      "Version": "0.3.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c7cf394610e3aaf086fa4b66c382016e",
      "Requirements": [
        "future",
        "globals",
        "lifecycle",
        "purrr",
        "rlang",
        "vctrs"
      ]
    },
    "future": {
      "Package": "future",
      "Version": "1.25.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "877024e372cf61e41f5d13eafd8d4bac",
      "Requirements": [
        "digest",
        "globals",
        "listenv",
        "parallelly"
      ]
    },
    "gargle": {
      "Package": "gargle",
      "Version": "1.2.0",
      "Source": "CRAN",
      "Repository": "CRAN",
      "RemoteType": "standard",
      "RemotePkgRef": "gargle",
      "RemoteRef": "gargle",
      "RemoteRepos": "https://cloud.r-project.org/",
      "RemotePkgPlatform": "source",
      "RemoteSha": "1.2.0",
      "Hash": "9d234e6a87a6f8181792de6dc4a00e39",
      "Requirements": [
        "cli",
        "fs",
        "glue",
        "httr",
        "jsonlite",
        "rappdirs",
        "rlang",
        "rstudioapi",
        "withr"
      ]
    },
    "gdtools": {
      "Package": "gdtools",
      "Version": "0.2.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "bdfa7431687797edff8c9b0eb9271cc8",
      "Requirements": [
        "Rcpp",
        "systemfonts"
      ]
    },
    "generics": {
      "Package": "generics",
      "Version": "0.1.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "177475892cf4a55865868527654a7741",
      "Requirements": []
    },
    "ggplot2": {
      "Package": "ggplot2",
      "Version": "3.3.6",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "0fb26d0674c82705c6b701d1a61e02ea",
      "Requirements": [
        "MASS",
        "digest",
        "glue",
        "gtable",
        "isoband",
        "mgcv",
        "rlang",
        "scales",
        "tibble",
        "withr"
      ]
    },
    "globals": {
      "Package": "globals",
      "Version": "0.15.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "ace993e2dad5e64ed46391302b79e94f",
      "Requirements": [
        "codetools"
      ]
    },
    "glue": {
      "Package": "glue",
      "Version": "1.6.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "4f2596dfb05dac67b9dc558e5c6fba2e",
      "Requirements": []
    },
    "googledrive": {
      "Package": "googledrive",
      "Version": "2.0.0",
      "Source": "CRAN",
      "Repository": "CRAN",
      "RemoteType": "standard",
      "RemotePkgRef": "googledrive",
      "RemoteRef": "googledrive",
      "RemoteRepos": "https://cloud.r-project.org/",
      "RemotePkgPlatform": "source",
      "RemoteSha": "2.0.0",
      "Hash": "c3a25adbbfbb03f12e6f88c5fb1f3024",
      "Requirements": [
        "cli",
        "gargle",
        "glue",
        "httr",
        "jsonlite",
        "lifecycle",
        "magrittr",
        "pillar",
        "purrr",
        "rlang",
        "tibble",
        "uuid",
        "vctrs",
        "withr"
      ]
    },
    "googlesheets4": {
      "Package": "googlesheets4",
      "Version": "1.0.0",
      "Source": "CRAN",
      "Repository": "CRAN",
      "RemoteType": "standard",
      "RemotePkgRef": "googlesheets4",
      "RemoteRef": "googlesheets4",
      "RemoteRepos": "https://cloud.r-project.org/",
      "RemotePkgPlatform": "source",
      "RemoteSha": "1.0.0",
      "Hash": "9a6564184dc4a81daea4f1d7ce357c6a",
      "Requirements": [
        "cellranger",
        "cli",
        "curl",
        "gargle",
        "glue",
        "googledrive",
        "httr",
        "ids",
        "magrittr",
        "purrr",
        "rematch2",
        "rlang",
        "tibble",
        "vctrs"
      ]
    },
    "gridExtra": {
      "Package": "gridExtra",
      "Version": "2.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "7d7f283939f563670a697165b2cf5560",
      "Requirements": [
        "gtable"
      ]
    },
    "gtable": {
      "Package": "gtable",
      "Version": "0.3.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "ac5c6baf7822ce8732b343f14c072c4d",
      "Requirements": []
    },
    "haven": {
      "Package": "haven",
      "Version": "2.5.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "e3058e4ac77f4fa686f68a1838d5b715",
      "Requirements": [
        "cli",
        "cpp11",
        "forcats",
        "hms",
        "lifecycle",
        "readr",
        "rlang",
        "tibble",
        "tidyselect",
        "vctrs"
      ]
    },
    "here": {
      "Package": "here",
      "Version": "1.0.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "24b224366f9c2e7534d2344d10d59211",
      "Requirements": [
        "rprojroot"
      ]
    },
    "highr": {
      "Package": "highr",
      "Version": "0.9",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "8eb36c8125038e648e5d111c0d7b2ed4",
      "Requirements": [
        "xfun"
      ]
    },
    "hms": {
      "Package": "hms",
      "Version": "1.1.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "5b8a2dd0fdbe2ab4f6081e6c7be6dfca",
      "Requirements": [
        "ellipsis",
        "lifecycle",
        "pkgconfig",
        "rlang",
        "vctrs"
      ]
    },
    "hrbrthemes": {
      "Package": "hrbrthemes",
      "Version": "0.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "0d1257948f016840ad90320ed19c8c7d",
      "Requirements": [
        "extrafont",
        "gdtools",
        "ggplot2",
        "htmltools",
        "knitr",
        "magrittr",
        "rmarkdown",
        "scales"
      ]
    },
    "htmltools": {
      "Package": "htmltools",
      "Version": "0.5.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "526c484233f42522278ab06fb185cb26",
      "Requirements": [
        "base64enc",
        "digest",
        "fastmap",
        "rlang"
      ]
    },
    "htmlwidgets": {
      "Package": "htmlwidgets",
      "Version": "1.5.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "76147821cd3fcd8c4b04e1ef0498e7fb",
      "Requirements": [
        "htmltools",
        "jsonlite",
        "yaml"
      ]
    },
    "httpuv": {
      "Package": "httpuv",
      "Version": "1.6.5",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "97fe71f0a4a1c9890e6c2128afa04bc0",
      "Requirements": [
        "R6",
        "Rcpp",
        "later",
        "promises"
      ]
    },
    "httr": {
      "Package": "httr",
      "Version": "1.4.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "88d1b310583777edf01ccd1216fb0b2b",
      "Requirements": [
        "R6",
        "curl",
        "jsonlite",
        "mime",
        "openssl"
      ]
    },
    "humaniformat": {
      "Package": "humaniformat",
      "Version": "0.6.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "d521cf9db39ca79250a00029661fb7cd",
      "Requirements": [
        "Rcpp"
      ]
    },
    "ids": {
      "Package": "ids",
      "Version": "1.0.1",
      "Source": "CRAN",
      "Repository": "CRAN",
      "RemoteType": "standard",
      "RemotePkgRef": "ids",
      "RemoteRef": "ids",
      "RemoteRepos": "https://cloud.r-project.org/",
      "RemotePkgPlatform": "source",
      "RemoteSha": "1.0.1",
      "Hash": "99df65cfef20e525ed38c3d2577f7190",
      "Requirements": [
        "openssl",
        "uuid"
      ]
    },
    "isoband": {
      "Package": "isoband",
      "Version": "0.2.5",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "7ab57a6de7f48a8dc84910d1eca42883",
      "Requirements": []
    },
    "janitor": {
      "Package": "janitor",
      "Version": "2.1.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "6de84a8c67fb247e721166049c84695f",
      "Requirements": [
        "dplyr",
        "lifecycle",
        "lubridate",
        "magrittr",
        "purrr",
        "rlang",
        "snakecase",
        "stringi",
        "stringr",
        "tidyr",
        "tidyselect"
      ]
    },
    "jquerylib": {
      "Package": "jquerylib",
      "Version": "0.1.4",
      "Source": "CRAN",
      "Repository": "CRAN",
      "RemoteType": "standard",
      "RemotePkgRef": "jquerylib",
      "RemoteRef": "jquerylib",
      "RemoteRepos": "https://cloud.r-project.org/",
      "RemotePkgPlatform": "source",
      "RemoteSha": "0.1.4",
      "Hash": "5aab57a3bd297eee1c1d862735972182",
      "Requirements": [
        "htmltools"
      ]
    },
    "jsonlite": {
      "Package": "jsonlite",
      "Version": "1.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "d07e729b27b372429d42d24d503613a0",
      "Requirements": []
    },
    "kableExtra": {
      "Package": "kableExtra",
      "Version": "1.3.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "49b625e6aabe4c5f091f5850aba8ff78",
      "Requirements": [
        "digest",
        "glue",
        "htmltools",
        "knitr",
        "magrittr",
        "rmarkdown",
        "rstudioapi",
        "rvest",
        "scales",
        "stringr",
        "svglite",
        "viridisLite",
        "webshot",
        "xml2"
      ]
    },
    "knitr": {
      "Package": "knitr",
      "Version": "1.39",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "029ab7c4badd3cf8af69016b2ba27493",
      "Requirements": [
        "evaluate",
        "highr",
        "stringr",
        "xfun",
        "yaml"
      ]
    },
    "labeling": {
      "Package": "labeling",
      "Version": "0.4.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "3d5108641f47470611a32d0bdf357a72",
      "Requirements": []
    },
    "later": {
      "Package": "later",
      "Version": "1.3.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "7e7b457d7766bc47f2a5f21cc2984f8e",
      "Requirements": [
        "Rcpp",
        "rlang"
      ]
    },
    "lattice": {
      "Package": "lattice",
      "Version": "0.20-45",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "b64cdbb2b340437c4ee047a1f4c4377b",
      "Requirements": []
    },
    "lazyeval": {
      "Package": "lazyeval",
      "Version": "0.2.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "d908914ae53b04d4c0c0fd72ecc35370",
      "Requirements": []
    },
    "leaflet": {
      "Package": "leaflet",
      "Version": "2.1.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "97fadb60ef6eb8524b9f6e2c4a69ee93",
      "Requirements": [
        "RColorBrewer",
        "base64enc",
        "crosstalk",
        "htmltools",
        "htmlwidgets",
        "leaflet.providers",
        "magrittr",
        "markdown",
        "png",
        "raster",
        "scales",
        "sp",
        "viridis"
      ]
    },
    "leaflet.providers": {
      "Package": "leaflet.providers",
      "Version": "1.9.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "d3082a7beac4a1aeb96100ff06265d7e",
      "Requirements": []
    },
    "lifecycle": {
      "Package": "lifecycle",
      "Version": "1.0.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "a6b6d352e3ed897373ab19d8395c98d0",
      "Requirements": [
        "glue",
        "rlang"
      ]
    },
    "listenv": {
      "Package": "listenv",
      "Version": "0.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "0bde42ee282efb18c7c4e63822f5b4f7",
      "Requirements": []
    },
    "lubridate": {
      "Package": "lubridate",
      "Version": "1.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "2ff5eedb6ee38fb1b81205c73be1be5a",
      "Requirements": [
        "cpp11",
        "generics"
      ]
    },
    "magrittr": {
      "Package": "magrittr",
      "Version": "2.0.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "7ce2733a9826b3aeb1775d56fd305472",
      "Requirements": []
    },
    "markdown": {
      "Package": "markdown",
      "Version": "1.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "61e4a10781dd00d7d81dd06ca9b94e95",
      "Requirements": [
        "mime",
        "xfun"
      ]
    },
    "mgcv": {
      "Package": "mgcv",
      "Version": "1.8-40",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c6b2fdb18cf68ab613bd564363e1ba0d",
      "Requirements": [
        "Matrix",
        "nlme"
      ]
    },
    "mime": {
      "Package": "mime",
      "Version": "0.12",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "18e9c28c1d3ca1560ce30658b22ce104",
      "Requirements": []
    },
    "modelr": {
      "Package": "modelr",
      "Version": "0.1.8",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "9fd59716311ee82cba83dc2826fc5577",
      "Requirements": [
        "broom",
        "magrittr",
        "purrr",
        "rlang",
        "tibble",
        "tidyr",
        "tidyselect",
        "vctrs"
      ]
    },
    "munsell": {
      "Package": "munsell",
      "Version": "0.5.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "6dfe8bf774944bd5595785e3229d8771",
      "Requirements": [
        "colorspace"
      ]
    },
    "namedropR": {
      "Package": "namedropR",
      "Version": "2.3.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "d460ca4f2f5d67da8be4c97e4620413d",
      "Requirements": [
        "R.utils",
        "bib2df",
        "dplyr",
        "htmltools",
        "lubridate",
        "qrcode",
        "readr",
        "stringr",
        "webshot"
      ]
    },
    "nlme": {
      "Package": "nlme",
      "Version": "3.1-157",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "dbca60742be0c9eddc5205e5c7ca1f44",
      "Requirements": [
        "lattice"
      ]
    },
    "openssl": {
      "Package": "openssl",
      "Version": "2.0.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "edde731e84cf3d42aa9b7f04b8de5941",
      "Requirements": [
        "askpass"
      ]
    },
    "pacman": {
      "Package": "pacman",
      "Version": "0.5.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "b37932b67288bc62bb81686b2f337a7d",
      "Requirements": [
        "remotes"
      ]
    },
    "parallelly": {
      "Package": "parallelly",
      "Version": "1.32.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "022489ebce2ef2438db4583fae118240",
      "Requirements": []
    },
    "pillar": {
      "Package": "pillar",
      "Version": "1.7.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "51dfc97e1b7069e9f7e6f83f3589c22e",
      "Requirements": [
        "cli",
        "crayon",
        "ellipsis",
        "fansi",
        "glue",
        "lifecycle",
        "rlang",
        "utf8",
        "vctrs"
      ]
    },
    "pkgconfig": {
      "Package": "pkgconfig",
      "Version": "2.0.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "01f28d4278f15c76cddbea05899c5d6f",
      "Requirements": []
    },
    "png": {
      "Package": "png",
      "Version": "0.1-7",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "03b7076c234cb3331288919983326c55",
      "Requirements": []
    },
    "prettyunits": {
      "Package": "prettyunits",
      "Version": "1.1.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "95ef9167b75dde9d2ccc3c7528393e7e",
      "Requirements": []
    },
    "processx": {
      "Package": "processx",
      "Version": "3.5.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "8bbae1a548d0d3fdf6647bdd9d35bf6d",
      "Requirements": [
        "R6",
        "ps"
      ]
    },
    "progress": {
      "Package": "progress",
      "Version": "1.2.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "14dc9f7a3c91ebb14ec5bb9208a07061",
      "Requirements": [
        "R6",
        "crayon",
        "hms",
        "prettyunits"
      ]
    },
    "promises": {
      "Package": "promises",
      "Version": "1.2.0.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "4ab2c43adb4d4699cf3690acd378d75d",
      "Requirements": [
        "R6",
        "Rcpp",
        "later",
        "magrittr",
        "rlang"
      ]
    },
    "ps": {
      "Package": "ps",
      "Version": "1.7.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "eef74b13f32cae6bb0d495e53317c44c",
      "Requirements": []
    },
    "purrr": {
      "Package": "purrr",
      "Version": "0.3.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "97def703420c8ab10d8f0e6c72101e02",
      "Requirements": [
        "magrittr",
        "rlang"
      ]
    },
    "qrcode": {
      "Package": "qrcode",
      "Version": "0.1.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "0f4288a6cfbcb3640f8fcb0adc523c75",
      "Requirements": [
        "R.utils",
        "assertthat",
        "stringr"
      ]
    },
    "rappdirs": {
      "Package": "rappdirs",
      "Version": "0.3.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "5e3c5dc0b071b21fa128676560dbe94d",
      "Requirements": []
    },
    "raster": {
      "Package": "raster",
      "Version": "3.5-15",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "70cf51235e017702ed77427797572ef0",
      "Requirements": [
        "Rcpp",
        "sp",
        "terra"
      ]
    },
    "readr": {
      "Package": "readr",
      "Version": "2.1.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "9c59de1357dc209868b5feb5c9f0fe2f",
      "Requirements": [
        "R6",
        "cli",
        "clipr",
        "cpp11",
        "crayon",
        "hms",
        "lifecycle",
        "rlang",
        "tibble",
        "tzdb",
        "vroom"
      ]
    },
    "readxl": {
      "Package": "readxl",
      "Version": "1.4.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "170c35f745563bb307e963bde0197e4f",
      "Requirements": [
        "cellranger",
        "cpp11",
        "progress",
        "tibble"
      ]
    },
    "rematch": {
      "Package": "rematch",
      "Version": "1.0.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c66b930d20bb6d858cd18e1cebcfae5c",
      "Requirements": []
    },
    "rematch2": {
      "Package": "rematch2",
      "Version": "2.1.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "76c9e04c712a05848ae7a23d2f170a40",
      "Requirements": [
        "tibble"
      ]
    },
    "remotes": {
      "Package": "remotes",
      "Version": "2.4.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "227045be9aee47e6dda9bb38ac870d67",
      "Requirements": []
    },
    "renv": {
      "Package": "renv",
      "Version": "0.15.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c1078316e1d4f70275fc1ea60c0bc431",
      "Requirements": []
    },
    "reprex": {
      "Package": "reprex",
      "Version": "2.0.1",
      "Source": "CRAN",
      "Repository": "CRAN",
      "RemoteType": "standard",
      "RemotePkgRef": "reprex",
      "RemoteRef": "reprex",
      "RemoteRepos": "https://cloud.r-project.org/",
      "RemotePkgPlatform": "source",
      "RemoteSha": "2.0.1",
      "Hash": "911d101becedc0fde495bd910984bdc8",
      "Requirements": [
        "callr",
        "cli",
        "clipr",
        "fs",
        "glue",
        "knitr",
        "rlang",
        "rmarkdown",
        "rstudioapi",
        "withr"
      ]
    },
    "rlang": {
      "Package": "rlang",
      "Version": "1.0.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "04884d9a75d778aca22c7154b8333ec9",
      "Requirements": []
    },
    "rmarkdown": {
      "Package": "rmarkdown",
      "Version": "2.14",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "31b60a882fabfabf6785b8599ffeb8ba",
      "Requirements": [
        "bslib",
        "evaluate",
        "htmltools",
        "jquerylib",
        "jsonlite",
        "knitr",
        "stringr",
        "tinytex",
        "xfun",
        "yaml"
      ]
    },
    "rprojroot": {
      "Package": "rprojroot",
      "Version": "2.0.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "1de7ab598047a87bba48434ba35d497d",
      "Requirements": []
    },
    "rstudioapi": {
      "Package": "rstudioapi",
      "Version": "0.13",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "06c85365a03fdaf699966cc1d3cf53ea",
      "Requirements": []
    },
    "rvest": {
      "Package": "rvest",
      "Version": "1.0.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "bb099886deffecd6f9b298b7d4492943",
      "Requirements": [
        "httr",
        "lifecycle",
        "magrittr",
        "rlang",
        "selectr",
        "tibble",
        "xml2"
      ]
    },
    "sass": {
      "Package": "sass",
      "Version": "0.4.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "f37c0028d720bab3c513fd65d28c7234",
      "Requirements": [
        "R6",
        "fs",
        "htmltools",
        "rappdirs",
        "rlang"
      ]
    },
    "scales": {
      "Package": "scales",
      "Version": "1.2.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "6e8750cdd13477aa440d453da93d5cac",
      "Requirements": [
        "R6",
        "RColorBrewer",
        "farver",
        "labeling",
        "lifecycle",
        "munsell",
        "rlang",
        "viridisLite"
      ]
    },
    "selectr": {
      "Package": "selectr",
      "Version": "0.4-2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "3838071b66e0c566d55cc26bd6e27bf4",
      "Requirements": [
        "R6",
        "stringr"
      ]
    },
    "servr": {
      "Package": "servr",
      "Version": "0.24",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "e2c3e268d654becf0d78a1ec13a05b46",
      "Requirements": [
        "httpuv",
        "jsonlite",
        "mime",
        "xfun"
      ]
    },
    "snakecase": {
      "Package": "snakecase",
      "Version": "0.11.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "4079070fc210c7901c0832a3aeab894f",
      "Requirements": [
        "stringi",
        "stringr"
      ]
    },
    "sp": {
      "Package": "sp",
      "Version": "1.4-7",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c5dad1d43440f0c426a6d29b30b333fa",
      "Requirements": [
        "lattice"
      ]
    },
    "stringi": {
      "Package": "stringi",
      "Version": "1.7.6",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "bba431031d30789535745a9627ac9271",
      "Requirements": []
    },
    "stringr": {
      "Package": "stringr",
      "Version": "1.4.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "0759e6b6c0957edb1311028a49a35e76",
      "Requirements": [
        "glue",
        "magrittr",
        "stringi"
      ]
    },
    "svglite": {
      "Package": "svglite",
      "Version": "2.1.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "68dfdf211af6aa4e5f050f064f64d401",
      "Requirements": [
        "cpp11",
        "systemfonts"
      ]
    },
    "sys": {
      "Package": "sys",
      "Version": "3.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "b227d13e29222b4574486cfcbde077fa",
      "Requirements": []
    },
    "systemfonts": {
      "Package": "systemfonts",
      "Version": "1.0.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "90b28393209827327de889f49935140a",
      "Requirements": [
        "cpp11"
      ]
    },
    "terra": {
      "Package": "terra",
      "Version": "1.5-21",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "a713f0d9f9a2b68af112a991adbfe5d5",
      "Requirements": [
        "Rcpp"
      ]
    },
    "tibble": {
      "Package": "tibble",
      "Version": "3.1.7",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "08415af406e3dd75049afef9552e7355",
      "Requirements": [
        "ellipsis",
        "fansi",
        "lifecycle",
        "magrittr",
        "pillar",
        "pkgconfig",
        "rlang",
        "vctrs"
      ]
    },
    "tidyr": {
      "Package": "tidyr",
      "Version": "1.2.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "d8b95b7fee945d7da6888cf7eb71a49c",
      "Requirements": [
        "cpp11",
        "dplyr",
        "ellipsis",
        "glue",
        "lifecycle",
        "magrittr",
        "purrr",
        "rlang",
        "tibble",
        "tidyselect",
        "vctrs"
      ]
    },
    "tidyselect": {
      "Package": "tidyselect",
      "Version": "1.1.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "17f6da8cfd7002760a859915ce7eef8f",
      "Requirements": [
        "ellipsis",
        "glue",
        "purrr",
        "rlang",
        "vctrs"
      ]
    },
    "tidyverse": {
      "Package": "tidyverse",
      "Version": "1.3.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "fc4c72b6ae9bb283416bd59a3303bbab",
      "Requirements": [
        "broom",
        "cli",
        "crayon",
        "dbplyr",
        "dplyr",
        "dtplyr",
        "forcats",
        "ggplot2",
        "googledrive",
        "googlesheets4",
        "haven",
        "hms",
        "httr",
        "jsonlite",
        "lubridate",
        "magrittr",
        "modelr",
        "pillar",
        "purrr",
        "readr",
        "readxl",
        "reprex",
        "rlang",
        "rstudioapi",
        "rvest",
        "stringr",
        "tibble",
        "tidyr",
        "xml2"
      ]
    },
    "tinytex": {
      "Package": "tinytex",
      "Version": "0.38",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "759d047596ac173433985deddf313450",
      "Requirements": [
        "xfun"
      ]
    },
    "tzdb": {
      "Package": "tzdb",
      "Version": "0.3.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "b2e1cbce7c903eaf23ec05c58e59fb5e",
      "Requirements": [
        "cpp11"
      ]
    },
    "utf8": {
      "Package": "utf8",
      "Version": "1.2.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c9c462b759a5cc844ae25b5942654d13",
      "Requirements": []
    },
    "uuid": {
      "Package": "uuid",
      "Version": "1.1-0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "f1cb46c157d080b729159d407be83496",
      "Requirements": []
    },
    "vctrs": {
      "Package": "vctrs",
      "Version": "0.4.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "8b54f22e2a58c4f275479c92ce041a57",
      "Requirements": [
        "cli",
        "glue",
        "rlang"
      ]
    },
    "viridis": {
      "Package": "viridis",
      "Version": "0.6.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "ee96aee95a7a563e5496f8991e9fde4b",
      "Requirements": [
        "ggplot2",
        "gridExtra",
        "viridisLite"
      ]
    },
    "viridisLite": {
      "Package": "viridisLite",
      "Version": "0.4.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "55e157e2aa88161bdb0754218470d204",
      "Requirements": []
    },
    "vroom": {
      "Package": "vroom",
      "Version": "1.5.7",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "976507b5a105bc3bdf6a5a5f29e0684f",
      "Requirements": [
        "bit64",
        "cli",
        "cpp11",
        "crayon",
        "glue",
        "hms",
        "lifecycle",
        "progress",
        "rlang",
        "tibble",
        "tidyselect",
        "tzdb",
        "vctrs",
        "withr"
      ]
    },
    "webshot": {
      "Package": "webshot",
      "Version": "0.5.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "7261ab7f98e97c771217e6b87c085d6e",
      "Requirements": [
        "callr",
        "jsonlite",
        "magrittr"
      ]
    },
    "withr": {
      "Package": "withr",
      "Version": "2.5.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "c0e49a9760983e81e55cdd9be92e7182",
      "Requirements": []
    },
    "xaringan": {
      "Package": "xaringan",
      "Version": "0.24",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "714cfbdbc0775404c8df7894dfed0165",
      "Requirements": [
        "htmltools",
        "knitr",
        "rmarkdown",
        "servr",
        "xfun"
      ]
    },
    "xfun": {
      "Package": "xfun",
      "Version": "0.31",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "a318c6f752b8dcfe9fb74d897418ab2b",
      "Requirements": []
    },
    "xml2": {
      "Package": "xml2",
      "Version": "1.3.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "40682ed6a969ea5abfd351eb67833adc",
      "Requirements": []
    },
    "yaml": {
      "Package": "yaml",
      "Version": "2.3.5",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "458bb38374d73bf83b1bb85e353da200",
      "Requirements": []
    }
  }
}

Collaboration

• Share the lockfile with the project bundle.

• Collaborators use renv::init() to install the appropriate versions of the packages

• Supports Python’s Virtual and Conda environments via {reticulate}

3. R-Markdown

_

_

Motivation

Updating the report

• Copying-pasting results is tedious and error-prone

• Is the report up-to date with the latest fixes?

• Which results changed after this fix?

Litterate programming for R

https://rmarkdown.rstudio.com/

Rather than writing a report,

write a program that produces a report

Whenever something is fixed in your code, you can run the program and have everything automatically up to date

Demo R Markdown

• integrates R Markdown in its interface

  File > New File > R Markdown

• Without RStudio, you can compile the document with rmarkdown::render()

• Also produce presentation slides, web sites, dashboards, books and more

• Supports R, python, SQL, julia, bash, C, …

• Examples: manuscript, slides, website, book, dashboard

How it works

1. {knitr} executes all of the code chunks and creates a temporary markdown document with text and code output

2. pandoc converts into the final format

Markdown

Lightweight markup language used for writing the text

• Multiple variants and extensions¹

1. E.g. Mattermost, GitLab and many platforms support Markdown

Scientific articles

• R Markdown supports LaTeX templates for more complex documents (e.g. CV, scientific articles, …)

• provides a suite of custom R Markdown LaTeX formats and templates¹

1. Handles authors, affiliations, abstract, cross-references, bibliography, etc.

Demo - case study

Convert our script into R Markdown

• Browse the Rmd file and compile

• Demonstrate the templating options in the yaml header

References

• https://rmarkdown.rstudio.com/

• https://yihui.org/knitr/

• RMarkdown tutorial (RStudio)

• Workshop R Markdown for Scientists (N. Tierney, 2020)

On line books

https://quarto.org/

• Next-generation version of R Markdown

• Stand-alone (no need to call it from R)

• Support for Python, R, Julia, and Observable

• More straightforward scientific authoring

• Coherent implementation of features from various R Markdown extensions

• Renders Rmd files almost without modification

Break time

4. Targets

Motivation

Pipeline the analysis

• Multiple Rmd documents use the same objects

• Recomputing the same objects over and over

• Rendering time (performed dozens of times)

Principles

• Separate the computations from the reporting (without the copying-and-pasting)

• Define a list of targets (objects to compute)

• Targets (including reports) depend on other targets, files and functions

• Computed targets are stored in a local cache

• Whenever something changes, recomputes targets downstream

Case study: _targets.R file excerpt

## Define the pipeline. A pipeline is just a list of targets.
list(

  # SOURCE FILES -----------------------------
  tar_target(
    data_file,
    "data/HumanDogRatioData.xlsx",
    format = "file"
  )
  ,

  # SOURCE DATA ------------------------------
  tar_target(
    raw_data,
      read_excel(data_file)
  )
  ,

  # DERIVED DATA -----------------------------
  tar_target(
    clean_data,
    raw_data %>% cleanup()
  )

  # REPORTS -------------------------------------------------------
  tar_render(
    report_html,
    "src/confint_ratio.Rmd",
    output_dir = "public",  # https://github.com/ropensci/drake/issues/742
    output_format =
      rmdformats::readthedown(
        dev = "CairoPNG",
        toc_depth = 3,
        lightbox = T,
        gallery = T,
        use_bookdown = T,
        number_sections = T),
    output_file =
      "public/confint_ratio.html",
      quiet = FALSE
  )
)

The graph of targets

Wokflow

• targets::use_targets() creates a script _targets.R in the project root

• Write targets and run the pipeline with tar_make()

• In the report, load necessary targets with tar_load()

tar_load(
  c(
    clean_data,
    sim_data,
    sim_pars,
    raw_data
  )
)

Additional benefits

• Immediately resume a project with targets::tar_load(everytihing())

• Parallel or remote computation of targets

• Each target run in a separate R session to ensure reproducibility

• Dynamic targets, create many targets programatically

Demo - case study

Structure into a pipeline with targets

• Check the graph, run the pipeline, remove something, verify and build outdated nodes…

Questions time

5. Git

Motivation

Integrate contributions

• From multiple collaborators (including your multiple selves!)

• Keeping track of changes and past versions

• Who did what, when and why

• Both for code and manuscript¹

1. which is also code, at this point

Git stores the whole history of the project

Workflow

• Initialise a git repository in your project folder

• Take a snapshot of a file or files, by making a commit

• Add information about the changes in the commit message

• Go back to any point in history by checking out a commit

• Push your commits to a remote shared repository

• Pull contributions from collaborators and merge

Conflicts

Non-conflicting contributions are merged automatically. Otherwise git requires user input for solving the conflict.

In practice

• Git is a command-line tool

• RStudio provides a basic interface for essential operations

• Other graphical clients exist (e.g. GitAhead)

Hosting services

• Remote repositories for collaboration

• Browse projects and edit on line

• Distribute software

• Disseminate material as web pages

• Manage issues and documentation

• Test and deploy automatically using Continuous Integration

Demo

Updating these slides

• Fix something in the data, code or text

• Commit, push, and check the results on line

References

• Crash course git and Forgemia (GitLab). F. Muñoz, Cirad.

• Happy Git and GitHub for the useR. Introduction to using git, GitHub and RStudio for effective collaboration. J Bryan.

• Git for Humans. Presentation slides.

• Version Control with Git. Introductory course. Software Carpentry.

• Pro Git book. Reference book. Scott Chacon.

• J Bryan (2018). Excuse Me, Do You Have a Moment to Talk About Version Control?. The American Statistician.

Well done!! take a good breath.

(you’ll need it)

6. Docker

Motivation

• Sharing your project more widely

• R version, compilation options, Linear Algebra libraries, environment variables, system libraries, platform-dependent processes…

Reproduce the entire

computing environment

Docker images

• An entire system (OS, libraries, R, packages, project files) executed by Docker as a single process

• You can distribute a Docker image that can be run by someone else without having to even install R¹

1. They need to have Docker installed, though

Images are stackable

• No need to build everything from scratch every time

• You can start from one of the images from the Rocker project

Docker registries

• Images can be distributed as files or through docker registries

• docker hub (https://hub.docker.com/) is the official registry

• GitLab also provides a registry.

  You can push a fix and have the Continuous Integration system deploy your update automatically

Demo

• Run the latest version of R in a container docker run --rm -ti r-base

• Run the development version of R in a container docker run --rm -ti rocker/r-ver:devel

• Run RStudio in a container docker run --rm -ti -e PASSWORD=yourpassword -p 8787:8787 rocker/rstudio

• You can also run zoom or any program within a container!

References

• A Docker tutorial for reproducible research. J Stachelek .

• The Rocker Project. Docker Containers for the R Environment

7. GNU Guix

Motivation

• TBH, I have not ever used it. But we need to mention it.

• Docker images evolve and get updated with new versions of system libraries.

• The environment does not stay always the same and it is difficult to audit.

Guix

• Guix is a Linux package manager that allows to control the complete software stack down to the latest library

• For professional, sensitive software applications that require ultimate reproducibility.

• Probably a bit of overkill for Reproducible Research

• …until someone makes something that facilitates its use

References

https://guix.gnu.org/

• Reproducible computations with Guix. K Hinsen.

• Wurmus, R. et al. 2018. PiGx: reproducible genomics analysis pipelines with GNU Guix. GigaScience 7.

Conclusions

Does all this save any time at all?

• It depends…

• Saves tons of management of dependencies, copying-pasting results, integrating contributions, recomputing results…

• But also requires a lot of learning and solving issues relative to the tools

  Computers allow us to solve a lot of problems that otherwise we would not have

It’s not about saving time or effort,

it’s about the kind of problems

you want to invest your efforts in

Advantages

• Improves the quality of the work

• Instrumental tools for Open Science

• Allows scaling up

  E.g. The ASF Challenge was a 2-people project with 13k+ lines of code. Impossible otherwise.

E.g. ASF Challenge dependency graph

What do you have to adopt

It’s up to you, of course! but,

• The chapter on R-scripts and functions is a low-hanging fruit accessible and useful to almost everyone

• The rest, depend very much on your context, background and needs

Personal templates and helpers

• Setting up all these tools takes some effort

• I use a project template and a initialisation script

• This makes use of a LaTeX template with a institutional report format

• Feel free to re-use and adapt to suit your needs!

• Demo: set up a project and publish the report

Thanks!