Preface

This book is intended to be a relatively gentle introduction to carrying out Bayesian data analysis and cognitive modeling using the probabilistic programming language Stan (Carpenter et al. 2017), and the front-end to Stan called brms (Bürkner 2019). Our target audience is cognitive scientists (e.g., linguists, psychologists, and computer scientists) who carry out planned behavioral experiments, and who are interested in learning the Bayesian data analysis methodology from the ground up and in a principled manner. Our aim is to make Bayesian statistics a standard part of the data analysis toolkit for experimental linguistics, psycholinguistics, psychology, and related disciplines.

Many excellent introductory textbooks already exist for Bayesian data analysis. Why write yet another book? Our text is different from other attempts in two respects. First, our main focus is on showing how to analyze data from planned experiments involving repeated measures; this type of experimental data involves unique complexities. We provide many examples of data sets involving time measurements (e.g., self-paced reading, eye-tracking-while-reading, voice onset time), event-related potentials, pupil sizes, accuracies (e.g., recall tasks, yes-no questions), categorical answers (e.g., picture naming), choice-reaction time (e.g, Stroop task, motion detection task), etc. Second, from the very outset, we emphasize a particular workflow that has as its centerpiece simulating data; we aim to teach a philosophy that involves thinking hard about the assumed underlying generative process, even before the data are collected. The data analysis approach that we hope to teach through this book involves a cycle of prior predictive and posterior predictive checks, sensitivity analyses, and model validation using simulated data. We try to inculcate a sense of how inferences can be drawn from the posterior distribution of theoretically interesting parameters without resorting to binary decisions like “significant” or “not-significant”. We are hopeful that this will set a new standard for reporting and interpreting results of data analyses in a more nuanced manner, and lead to more measured claims in the published literature.

Please report typos, errors, or suggestions for improvement at https://github.com/vasishth/bayescogsci/issues.

Why read this book, and what is its target audience?

A commonly-held belief in psychology, psycholinguistics, and other areas is that statistical data analysis is secondary to the science, and should be quick and easy. For example, a senior mathematical psychologist once told the last author of this book: “if you need to run anything more complicated than a paired t-test, you are asking the wrong question.” We take a different perspective here: the science and the statistical modeling are one unitary thing. The statistical model should represent some reasonable approximation of the latent cognitive processes that are assumed to be in play.

The target audience for this book is students and researchers who want to treat statistics as an equal partner in their scientific work. We expect that the reader is willing to take the time to both understand and to run the computational analyses.

Any rigorous introduction to Bayesian data analysis requires at least a passive knowledge of probability theory, calculus, and linear algebra. However, we do not require that the reader has this background when they start the book. Instead, the relevant ideas are introduced informally and just in time, as soon as they are needed. The reader is never required to have an active ability to solve probability problems, to solve integrals or compute derivatives, or to carry out matrix computations (such as inverting matrices) by hand. There are a few places where the discussion becomes technical and requires some knowledge of calculus or related topics. However, the reader unfamiliar with the required mathematics can simply skip those sections, as these are not really necessary to follow the main thread of the book.

What we do expect is familiarity with arithmetic, basic set theory and elementary probability theory (e.g., sum and product rules, conditional probability), simple matrix operations like addition and multiplication, and simple algebraic operations. A quick look through chapter 1 of Gill (2006) before starting this book is highly recommended. We also presuppose that, when the need arises, the reader is willing to look up concepts that that they might have forgotten (e.g., logarithms). We also provide a self-paced course on some basic concepts in mathematics (aimed at non-mathematicians) which the reader can work through: see https://vasishth.github.io/FoM/.

We also expect that the reader already knows and/or is willing to learn enough of the programming language R (R Core Team 2019) to reproduce the examples presented and to carry out the exercises. If the reader is completely unfamiliar with R, before starting this book they should first consult books like R for data science, and Efficient R programming.

We also assume that the reader has encountered simple linear modeling, and linear mixed models (Bates, Mächler, et al. 2015 a; Baayen, Davidson, and Bates 2008). What this means in practice is that the reader should have used the lm() and lmer() functions in R. A passing acquaintance with basic statistical concepts, like the correlation between two variables, is also taken for granted.

This book is not appropriate for complete beginners to data analysis. Newcomers to data analysis should start with a freely available textbook like Kerns (2014), and then read our introduction to frequentist data analysis, which is also available freely online (Vasishth et al. 2021). This latter book will prepare the reader well for the material presented here.

Developing the right mindset for this book

One very important characteristic that the reader should bring to this book is a can-do spirit. There will be many places where the going will get tough, and the reader will have to slow down and play around with the material, or refresh their understanding of arithmetic or middle-school algebra. The basic principles of such a can-do spirit are nicely summarized in the book by Burger and Starbird (2012); also see Levy (2021). Although we cannot summarize all the insights from these books in a few words, inspired by the Burger and Starbird (2012) book, here is a short enumeration of the kind of mindset the reader will need to cultivate:

Spend time on the basic, apparently easy material; make sure you understand it deeply. Look for gaps in your understanding. Reading different presentations of the same material (in different books or articles) can yield new insights.
Let mistakes and errors be your teacher. We instinctively recoil from our mistakes, but errors are ultimately our friends; they have the potential to teach us more than our correct answers can. In this sense, a correct solution can be less interesting than an incorrect one.
When you are intimidated by some exercise or problem, give up and admit defeat immediately. This relaxes the mind; you’ve already given up, there’s nothing more to do. Then, after a while, try to solve a simpler version of the problem. Sometimes, it is useful to break the problem down to smaller parts, each of which may be easier to solve.
Create your own questions. Don’t wait to be asked questions; develop your own problems and then try to solve them.
Don’t expect to understand everything in the first pass. Just mentally note the gaps in your understanding, and return to them later and work on these gaps.
Step back periodically to try to sketch out a broader picture of what you are learning. Writing down what you know, without looking up anything, is one helpful way to achieve this. Don’t wait for the teacher to give you bullet-point summaries of what you should have learned; develop such summaries yourself.
Develop the art of finding information. When confronted with something you don’t know, or with some obscure error message, use google to find some answers.
Do not hesitate to re-read a chapter; often, one only understands a topic after one revisits the material.

As instructors, we have noticed over the years that students with such a mindset generally do very well. Some students already have that spirit, but others need to explicitly develop it. We firmly believe that everyone can develop such a mindset, but one may have to work on acquiring it. In any case, such an attitude is necessary for a book of this sort.

How to read this book

The chapters in this book are intended to be read in sequence, but during the first pass through the book, the reader should feel free to completely skip the boxes. These boxes provide a more formal development (useful to transition to more advanced textbooks like Gelman et al. 2014), or deal with tangential aspects of the topics presented in the chapter.

Here are some suggested paths through this book, depending on the reader’s goals:

For a short course for complete beginners, read chapters 1 to 5. We usually cover these five chapters in a five-day summer school course that we teach annually. Most of the material in this chapter is also covered in a free four-week course available online: https://open.hpi.de/courses/bayesian-statistics2023.
For a course that focuses on regression models with the R package brms, read chapters 1 to 9 and, optionally, 15.
For an advanced course that focuses on complex models involving Stan, read chapters 10 to 20.

Some conventions used in this book

We adopt the following conventions:

All distribution names are lower-case unless they are also a proper name (e.g., Poisson, Bernoulli).
The univariate normal distribution is parameterized by the mean and standard deviation (not variance).
The code for figures is provided only in some cases, where we consider it to be pedagogically useful. In other cases, the code remains hidden, but it can be found in the web version of the book. Notice that all the R code from the book can be extracted from the Rmd source files for each chapter, which are released with the book.

Online materials

The entire book, including all data and source code, is available online for free on https://vasishth.github.io/bayescogsci/book. The solutions to exercises will be made available on request.

Software needed

Before you start, please install

R and RStudio, or any other Integrated Development Environment that you prefer, such as Visual Studio Code and Emacs Speaks Statistics.
The R package rstan. At the time of writing this book, the CRAN version of rstan lags behind the latest developments in Stan so it is recommended to install rstan from https://mc-stan.org/r-packages/ as indicated in https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started
The R packages dplyr, purrr, tidyr, extraDistr, brms, hypr and lme4 are used in many chapters of the book and can be installed the usual way: install.packages(c("dplyr","purrr","tidyr", "extraDistr", "brms","hypr","lme4")). We use ggplot2 for figures; please consult the relevant documentation if you are unfamiliar with ggplot2 (e.g., Wickham, Chang, et al. 2019).
The following R packages are optional: tictoc, rootSolve, SHELF, cmdstanr, and SBC.
Some packages and their dependencies, such as intoo, barsurf, bivariate, and SIN could require manual installation from archived or github versions.
The data and Stan models used in this book can be installed using remotes::install_github("bnicenboim/bcogsci"). This command uses the function install_github from the package remotes. (Thus this package should be in the system as well.)

In every R session, load these packages, and set the options shown below for Stan.

library(MASS)
## be careful to load dplyr after MASS
library(dplyr)
library(tidyr)
library(purrr)
library(extraDistr)
library(ggplot2)
library(loo)
library(bridgesampling)
library(brms)
library(bayesplot)
library(tictoc)
library(hypr)
library(bcogsci)
library(lme4)
library(rstan)
# This package is optional, see https://mc-stan.org/cmdstanr/:
library(cmdstanr)
# This package is optional, see https://hyunjimoon.github.io/SBC/:
library(SBC)
library(SHELF)
library(rootSolve)

## Save compiled models:
rstan_options(auto_write = FALSE)
## Parallelize the chains using all the cores:
options(mc.cores = parallel::detectCores())
# To solve some conflicts between packages:
select <- dplyr::select
extract <- rstan::extract

Acknowledgments

We are grateful to the many generations of students at the University of Potsdam, various summer schools at ESSLLI, the LOT winter school, the Open HPI MOOC on Bayesian statistics (https://open.hpi.de/courses/bayesian-statistics2023), other short courses we have taught at various institutions, and the annual summer school on Statistical Methods for Linguistics and Psychology (SMLP) held annually at Potsdam, Germany. The participants in these courses helped us considerably in improving the material presented here. A special thanks to Anna Laurinavichyute, Paula Lissón, and Himanshu Yadav for co-teaching the the Bayesian courses at SMLP. We are also grateful to members of Vasishth lab, especially Dorothea Pregla, for comments on earlier drafts of this book. We would also like to thank Douglas Bates, Ben Bolker, Christian Robert (otherwise known as Xi’an), Robin Ryder, Nicolas Chopin, Michael Betancourt, Andrew Gelman, and the Stan developers (especially Bob Carpenter and Paul-Christian Bürkner) for their comments and advice; to Pavel Logačev for his feedback, and Athanassios Protopapas, Patricia Mirabile, Masataka Ogawa, Alex Swiderski, Andrew Ellis, Jakub Szewczyk, Chi Hou Pau, Alec Shaw, Patrick Wen, Riccardo Fusaroli, Abdulrahman Dallak, Elizabeth Pankratz, João Veríssimo, Jean-Pierre Haeberly, Chris Hammill, Florian Wickelmaier, Ole Seeth, Jules Bouton, Siqi Zheng, Michael Gaunt, Benjamin Senst, Chris Moreh, Richard Hatcher, Noelia Stetie, Robert Lew, Leonardo Cerliani, Stefan Riedel, Noelia Stetie, Raluca Rilla, Arne Schernich, Sven Koch, Joy Sarow, Iñigo Urrestarazu-Porta, Jan Winkowski, Adrian Staub, Brian Dillon, Job Schepens, Katja Politt, Cui Ding, Marc Tortorello, and Michael Vrazitulis for catching typos, unclear passages, and errors in the book. Our apologies to anyone that we have forgotten to mention. Thanks also go to Jeremy Oakley and other statisticians at the School of Mathematics and Statistics, University of Sheffield, UK, for helpful discussions, and ideas for exercises that were inspired from the MSc program taught online at Sheffield.

This book would have been impossible to write without the following software: R (Version 4.3.2; R Core Team 2019) and the R-packages afex (Singmann et al. 2020), barsurf (Version 0.7.0; Spurdle 2020 a), bayesplot (Version 1.11.1; Gabry and Mahr 2019), bcogsci (Version 0.0.0.9000; Nicenboim, Schad, and Vasishth 2020), bibtex (Francois 2017), bivariate (Version 0.7.0; Spurdle 2020 b), bookdown (Version 0.38; Xie 2019 a), bridgesampling (Version 1.1.2; Gronau, Singmann, and Wagenmakers 2020), brms (Version 2.20.4; Bürkner 2019), citr (Aust 2019), cmdstanr (Version 0.7.1; Gabry and Češnovar 2021), cowplot (Version 1.1.3; Wilke 2020), digest (Version 0.6.34; Antoine Lucas et al. 2021), dplyr (Version 1.1.4; Wickham, François, et al. 2019), DT (Version 0.32; Xie, Cheng, and Tan 2019), extraDistr (Version 1.10.0; Wolodzko 2019), forcats (Wickham 2019 a), gdtools (Gohel et al. 2019), ggplot2 (Version 3.5.0; Wickham, Chang, et al. 2019), gridExtra (Version 2.3; Auguie 2017), htmlwidgets (Version 1.6.4; Vaidyanathan et al. 2018), hypr (Version 0.2.8; Schad, Vasishth, et al. 2019; Rabe et al. 2020 b), intoo (Version 0.4.0; Spurdle and Bode 2020), kableExtra (Version 1.4.0; Zhu 2019), knitr (Version 1.45; Xie 2019 b), lme4 (Version 1.1.35.1; Bates, Mächler, et al. 2015 b), loo (Version 2.7.0; Vehtari, Gelman, and Gabry 2017 a; Yao et al. 2017), MASS (Version 7.3.60; Ripley 2019), Matrix (Version 1.6.1.1; Bates and Maechler 2019), miniUI (Version 0.1.1.1; Cheng 2018), papaja (Version 0.1.2; Aust and Barth 2020), pdftools (Version 3.4.0; Ooms 2021), purrr (Version 1.0.2; Henry and Wickham 2019), Rcpp (Version 1.0.12; Eddelbuettel et al. 2019), readr (Wickham, Hester, and Francois 2018), RefManageR (McLean 2017), remotes (Version 2.4.2.1; Hester et al. 2021), rethinking (Version 2.40; McElreath 2021), rmarkdown (Version 2.26; Allaire et al. 2019), rootSolve (Version 1.8.2.4; Soetaert and Herman 2009), rstan (Version 2.32.6; Guo, Gabry, and Goodrich 2019), SBC (Version 0.2.0.9000; Kim et al. 2022), servr (Xie 2019 c), SHELF (Version 1.9.0; Oakley 2021), SIN (Version 0.6; Drton 2013), StanHeaders (Version 2.32.2; Goodrich et al. 2019), stringr (Version 1.5.1; Wickham 2019 b), texPreview (Sidi and Polhamus 2020), tibble (Version 3.2.1; Müller and Wickham 2020), tictoc (Version 1.2; Izrailev 2014), tidyr (Version 1.3.1; Wickham and Henry 2019), tidyverse (Wickham, Averick, et al. 2019), tinylabels (Version 0.2.4; Barth 2022), and webshot (Chang 2018).

Bruno Nicenboim (Tilburg, The Netherlands), Daniel Schad (Potsdam, Germany), Shravan Vasishth (Potsdam, Germany)

References

Allaire, JJ, Yihui Xie, Jonathan McPherson, Javier Luraschi, Kevin Ushey, Aron Atkins, Hadley Wickham, Joe Cheng, Winston Chang, and Richard Iannone. 2019. rmarkdown: Dynamic Documents for R. https://CRAN.R-project.org/package=rmarkdown.

Antoine Lucas, Dirk Eddelbuettel with contributions by, Jarek Tuszynski, Henrik Bengtsson, Simon Urbanek, Mario Frasca, Bryan Lewis, Murray Stokely, et al. 2021. Digest: Create Compact Hash Digests of R Objects. https://CRAN.R-project.org/package=digest.

Auguie, Baptiste. 2017. GridExtra: Miscellaneous Functions for "Grid" Graphics. https://CRAN.R-project.org/package=gridExtra.

Aust, Frederik. 2019. citr: RStudio Add-in to Insert Markdown Citations. https://CRAN.R-project.org/package=citr.

Aust, Frederik, and Marius Barth. 2020. papaja: Create APA Manuscripts with R Markdown. https://github.com/crsh/papaja.

Baayen, R. Harald, Douglas J. Davidson, and Douglas M. Bates. 2008. “Mixed-Effects Modeling with Crossed Random Effects for Subjects and Items.” Journal of Memory and Language 59 (4): 390–412.

Barth, Marius. 2022. tinylabels: Lightweight Variable Labels. https://cran.r-project.org/package=tinylabels.

Bates, Douglas M., Martin Mächler, Ben Bolker, and Steve Walker. 2015a. “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67 (1): 1–48. https://doi.org/10.18637/jss.v067.i01.

2015b. “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67 (1): 1–48. https://doi.org/10.18637/jss.v067.i01.

Bates, Douglas M., and Martin Maechler. 2019. Matrix: Sparse and Dense Matrix Classes and Methods. https://CRAN.R-project.org/package=Matrix.

Burger, Edward B., and Michael Starbird. 2012. The 5 Elements of Effective Thinking. Princeton University Press.

Bürkner, Paul-Christian. 2019. brms: Bayesian Regression Models Using “Stan”. https://CRAN.R-project.org/package=brms.

Carpenter, Bob, Andrew Gelman, Matthew D. Hoffman, Daniel Lee, Ben Goodrich, Michael J. Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. “Stan: A Probabilistic Programming Language.” Journal of Statistical Software 76 (1).

Chang, Winston. 2018. webshot: Take Screenshots of Web Pages. https://CRAN.R-project.org/package=webshot.

Cheng, Joe. 2018. miniUI: Shiny Ui Widgets for Small Screens. https://CRAN.R-project.org/package=miniUI.

Drton, Mathias. 2013. SIN: A Sinful Approach to Selection of Gaussian Graphical Markov Models. https://CRAN.R-project.org/package=SIN.

Eddelbuettel, Dirk, Romain Francois, J. J. Allaire, Kevin Ushey, Qiang Kou, Nathan Russell, Douglas M. Bates, and John Chambers. 2019. Rcpp: Seamless R and C++ Integration. https://CRAN.R-project.org/package=Rcpp.

Francois, Romain. 2017. Bibtex: Bibtex Parser. https://CRAN.R-project.org/package=bibtex.

Gabry, Jonah, and Rok Češnovar. 2021. cmdstanr: R Interface to “CmdStan”.

Gabry, Jonah, and Tristan Mahr. 2019. bayesplot: Plotting for Bayesian Models. https://CRAN.R-project.org/package=bayesplot.

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2014. Bayesian Data Analysis. Third Edition. Boca Raton, FL: Chapman; Hall/CRC Press.

Gill, Jeff. 2006. Essential Mathematics for Political and Social Research. Cambridge University Press Cambridge.

Gohel, David, Hadley Wickham, Lionel Henry, and Jeroen Ooms. 2019. gdtools: Utilities for Graphical Rendering. https://CRAN.R-project.org/package=gdtools.

Goodrich, Ben, Andrew Gelman, Bob Carpenter, Matthew D. Hoffman, Daniel Lee, Michael J. Betancourt, Marcus Brubaker, et al. 2019. StanHeaders: C++ Header Files for Stan. https://CRAN.R-project.org/package=StanHeaders.

Gronau, Quentin F., Henrik Singmann, and Eric-Jan Wagenmakers. 2017. “Bridgesampling: An R Package for Estimating Normalizing Constants.” Arxiv. http://arxiv.org/abs/1710.08162.

2020. “bridgesampling: An R Package for Estimating Normalizing Constants.” Journal of Statistical Software 92 (10): 1–29. https://doi.org/10.18637/jss.v092.i10.

Guo, Jiqiang, Jonah Gabry, and Ben Goodrich. 2019. rstan: R Interface to Stan. https://CRAN.R-project.org/package=rstan.

Henry, Lionel, and Hadley Wickham. 2019. purrr: Functional Programming Tools. https://CRAN.R-project.org/package=purrr.

Hester, Jim, Gábor Csárdi, Hadley Wickham, Winston Chang, Martin Morgan, and Dan Tenenbaum. 2021. Remotes: R Package Installation from Remote Repositories, Including ’Github’. https://CRAN.R-project.org/package=remotes.

Izrailev, Sergei. 2014. Tictoc: Functions for Timing R Scripts, as Well as Implementations of Stack and List Structures. https://CRAN.R-project.org/package=tictoc.

Kerns, G. J. 2014. Introduction to Probability and Statistics Using R. Second Edition.

Kim, Shinyoung, Hyunji Moon, Martin Modrák, and Teemu Säilynoja. 2022. SBC: Simulation Based Calibration for Rstan/Cmdstanr Models. https://github.com/hyunjimoon/SBC.

Levy, Dan. 2021. Maxims for Thinking Analytically: The Wisdom of Legendary Harvard Professor Richard Zeckhauser. Dan Levy.

McElreath, Richard. 2021. Rethinking: Statistical Rethinking Book Package.

McLean, Mathew William. 2017. “RefManageR: Import and Manage Bibtex and Biblatex References in R.” The Journal of Open Source Software. https://doi.org/10.21105/joss.00338.

Müller, Kirill, and Hadley Wickham. 2020. Tibble: Simple Data Frames. https://CRAN.R-project.org/package=tibble.

Nicenboim, Bruno, Daniel J. Schad, and Shravan Vasishth. 2020. bcogsci: Data and Models for the Book “An Introduction to Bayesian Data Analysis for Cognitive Science”.

Oakley, Jeremy E. 2021. SHELF: Tools to Support the Sheffield Elicitation Framework. https://CRAN.R-project.org/package=SHELF.

Ooms, Jeroen. 2021. pdftools: Text Extraction, Rendering and Converting of Pdf Documents. https://CRAN.R-project.org/package=pdftools.

Rabe, Maximilian M., Shravan Vasishth, Sven Hohenstein, Reinhold Kliegl, and Daniel J. Schad. 2020a. “hypr: An R Package for Hypothesis-Driven Contrast Coding.” Journal of Open Source Software 5 (48): 2134.

2020b. “Hypr: An R Package for Hypothesis-Driven Contrast Coding.” The Journal of Open Source Software. https://doi.org/10.21105/joss.02134.

R Core Team. 2019. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Ripley, Brian D. 2019. MASS: Support Functions and Datasets for Venables and Ripley’s MASS. https://CRAN.R-project.org/package=MASS.

Schad, Daniel J., Shravan Vasishth, Sven Hohenstein, and Reinhold Kliegl. 2019. “How to Capitalize on a Priori Contrasts in Linear (Mixed) Models: A Tutorial.” Journal of Memory and Language 110. https://doi.org/10.1016/j.jml.2019.104038.

Sidi, Jonathan, and Daniel Polhamus. 2020. TexPreview: Compile and Preview Snippets of “Latex”. https://CRAN.R-project.org/package=texPreview.

Singmann, Henrik, Ben Bolker, Jake Westfall, Frederik Aust, and Mattan S. Ben-Shachar. 2020. Afex: Analysis of Factorial Experiments. https://CRAN.R-project.org/package=afex.

Soetaert, Karline, and Peter M. J. Herman. 2009. A Practical Guide to Ecological Modelling. Using R as a Simulation Platform. Springer.

Spurdle, Abby. 2020a. Barsurf: Heatmap-Related Plots and Smooth Multiband Color Interpolation. https://CRAN.R-project.org/package=barsurf.

Spurdle, Abby. 2020b. Bivariate: Bivariate Probability Distributions. https://CRAN.R-project.org/package=bivariate.

Spurdle, Abby, and Emil Bode. 2020. Intoo: Minimal Language-Like Extensions. https://CRAN.R-project.org/package=intoo.

Vaidyanathan, Ramnath, Yihui Xie, J. J. Allaire, Joe Cheng, and Kenton Russell. 2018. htmlwidgets: HTML Widgets for R. https://CRAN.R-project.org/package=htmlwidgets.

Vasishth, Shravan, Daniel J. Schad, Audrey Bürki, and Reinhold Kliegl. 2021. “Linear Mixed Models for Linguistics and Psychology: A Comprehensive Introduction.” https://vasishth.github.io/Freq_CogSci/.

Vehtari, Aki, Andrew Gelman, and Jonah Gabry. 2017a. “Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and Waic.” Statistics and Computing 27 (5): 1413–32. https://doi.org/10.1007/s11222-016-9696-4.

Wickham, Hadley. 2019a. forcats: Tools for Working with Categorical Variables (Factors). https://CRAN.R-project.org/package=forcats.

Wickham, Hadley. 2019b. stringr: Simple, Consistent Wrappers for Common String Operations. https://CRAN.R-project.org/package=stringr.

Wickham, Hadley, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the tidyverse.” Journal of Open Source Software 4 (43): 1686. https://doi.org/10.21105/joss.01686.

Wickham, Hadley, Winston Chang, Lionel Henry, Thomas Lin Pedersen, Kohske Takahashi, Claus O. Wilke, Kara Woo, and Hiroaki Yutani. 2019. ggplot2: Create Elegant Data Visualisations Using the Grammar of Graphics. https://CRAN.R-project.org/package=ggplot2.

Wickham, Hadley, Romain François, Lionel Henry, and Kirill Müller. 2019. dplyr: A Grammar of Data Manipulation. https://CRAN.R-project.org/package=dplyr.

Wickham, Hadley, and Lionel Henry. 2019. Tidyr: Tidy Messy Data. https://CRAN.R-project.org/package=tidyr.

Wickham, Hadley, Jim Hester, and Romain Francois. 2018. readr: Read Rectangular Text Data. https://CRAN.R-project.org/package=readr.

Wilke, Claus O. 2020. cowplot: Streamlined Plot Theme and Plot Annotations for ’Ggplot2’. https://CRAN.R-project.org/package=cowplot.

Wolodzko, Tymoteusz. 2019. extraDistr: Additional Univariate and Multivariate Distributions. https://CRAN.R-project.org/package=extraDistr.

Xie, Yihui. 2019a. bookdown: Authoring Books and Technical Documents with R Markdown. https://CRAN.R-project.org/package=bookdown.

Xie, Yihui. 2019b. knitr: A General-Purpose Package for Dynamic Report Generation in R. https://CRAN.R-project.org/package=knitr.

Xie, Yihui. 2019c. servr: A Simple Http Server to Serve Static Files or Dynamic Documents. https://CRAN.R-project.org/package=servr.

Xie, Yihui, Joe Cheng, and Xianying Tan. 2019. DT: A Wrapper of the Javascript Library ’Datatables’. https://CRAN.R-project.org/package=DT.

Yao, Yuling, Aki Vehtari, Daniel P. Simpson, and Andrew Gelman. 2017. “Using Stacking to Average Bayesian Predictive Distributions.” Bayesian Analysis. https://doi.org/10.1214/17-BA1091.

Zhu, Hao. 2019. KableExtra: Construct Complex Table with ’Kable’ and Pipe Syntax. https://CRAN.R-project.org/package=kableExtra.