(This publication is still a work in progress)

“What is to be sought in designs for the display of information is the clear portrayal of complexity. Not the complication of the simple; rather … the revelation of the complex.”
- Edward R. Tufte

Introduction

The ggstatsplot package is an opinionated collection of plots made with ggplot2 and is designed for exploratory data analysis or for producing publication-ready statistical graphics. All plots share an underlying principle of displaying information-rich plots with all necessary statistical details included in the plots themselves. Although the plots produced by ggstatsplot are still ggplot objects and can thus be further modified using ggplot2 commands, there is a limit to how many such modifications can be made. That is, it is less flexible than ggplot2, but that’s a feature and not a bug. The original intent behind this package is to offload struggles associated with constructing the plot and focus more on the interpretation of that data displayed in the plot.

Graphical perception

Graphical perception involves visual decoding of the encoded information in graphs. ggstatsplot incorporates the paradigm proposed in Cleveland (1985, Chapter 4) to facilitate making visual judgments about quantitative information effortless and almost instantaneous. Based on experiments, Cleveland proposes that there are ten elementary graphical-perception tasks that we perform to visually decode quantitative information in graphs (organized from most to least accurate; Cleveland, 1985, p.254)-

  • Position along a common scale
  • Position along identical, non-aligned scales
  • Length
  • Angle (Slope)
  • Area
  • Volume
  • Color hue - Color saturation - Density

So the key principle of Cleveland’s paradigm for data display is-

“We should encode data on a graph so that the visual decoding involves [graphical-perception] tasks as high in the ordering as possible.”

For example, decoding the data point values in ggbetweenstats requires position judgments along a common scale (Figure-1):

**Figure-1**. Note that assessing differences in mean values between groups has been made easier with the help of *position* of data points along a common scale (the Y-axis) and labels.

Figure-1. Note that assessing differences in mean values between groups has been made easier with the help of position of data points along a common scale (the Y-axis) and labels.

There are few instances where ggstatsplot diverges from recommendations made in Cleveland’s paradigm:

  • For the categorical/nominal data, ggstatsplot uses pie charts (see Figure-2) which rely on angle judgments, which are less accurate (as compared to bar graphs, e.g., which require position judgments). This shortcoming is assuaged to some degree by using plenty of labels that describe percentages for all slices. This makes angle judgment unnecessary and pre-vacates any concerns about inaccurate judgments about percentages.
**Figure-2**. Pie charts don't follow Cleveland's paradigm to data display because they rely on less accurate angle judgments. `ggstatsplot` sidesteps this issue by always labelling percentages for pie slices, which makes angle judgments unnecessary.

Figure-2. Pie charts don’t follow Cleveland’s paradigm to data display because they rely on less accurate angle judgments. ggstatsplot sidesteps this issue by always labelling percentages for pie slices, which makes angle judgments unnecessary.

  • Cleveland’s paradigm also emphasizes that superposition of data is better than juxtaposition (Cleveland, 1985, p.201) because this allows for a more incisive comparison of the values from different parts of the dataset. This recommendation is violated in all grouped_ variants of the function (see Figure-3). Note that the range for Y-axes are no longer the same across juxtaposed subplots and so visually comparing the data becomes difficult. On the other hand, in the superposed plot, all data have the same range and coloring different parts makes the visual discrimination of different components of the data, and their comparison, easier. But the goal of grouped_ variants of functions is to not only show different aspects of the data but also to run statistical tests and showing detailed results for all aspects of the data in a superposed plot is difficult. Therefore, this is a compromise ggstatsplot is comfortable with, at least to produce plots for quick exploration of different aspects of the data.
**Figure-3**. Comparing different aspects of data is much more accurate in (a) a *superposed* plot, which is recommended in Cleveland's paradigm, than in (b) a *juxtaposed* plot, which is how it is implemented in `ggstatsplot` package. This is because displaying detailed results from statistical tests would be difficult in a superposed plot.

Figure-3. Comparing different aspects of data is much more accurate in (a) a superposed plot, which is recommended in Cleveland’s paradigm, than in (b) a juxtaposed plot, which is how it is implemented in ggstatsplot package. This is because displaying detailed results from statistical tests would be difficult in a superposed plot.

The grouped_ plots follow the Shrink Principle (Tufte, 2001, p.166-7) for high-information graphics, which dictates that the data density and the size of the data matrix can be maximized to exploit maximum resolution of the available data-display technology. Given the large maximum resolution afforded by most computer monitors today, saving grouped_ plots with appropriate resolution ensures no loss in legibility with reduced graphics area.

Graphical integrity (and clean design)

Graphical excellence consists of communicating complex ideas with clarity and in a way that the viewer understands the greatest number of ideas in a short amount of time all the while not quoting the data out of context. The package follows the principles for graphical integrity (as outlined in Tufte, 2001):

  • The physical representation of numbers is proportional to the numerical quantities they represent (e.g., Figure-1 and Figure-2 show how means (in ggbetweenstats) or percentages (ggpiestats) are proportional to the vertical distance or the area, respectively).

  • All important events in the data have clear, detailed, and thorough labeling (e.g., Figure-1 plot shows how ggbetweenstats labels means, sample size information, outliers, and pairwise comparisons; same can be appreciated for ggpiestats in Figure-2 and gghistostats in Figure-5). Note that data labels in the data region are designed in a way that they don’t interfere with our ability to assess the overall pattern of the data (Cleveland, 1985; p.44-45). This is achieved by using ggrepel package to place labels in a way that reduces their visual prominence.

  • None of the plots have design variation (e.g., abrupt change in scales) over the surface of a same graphic because this can lead to a false impression about variation in data.

  • The number of information-carrying dimensions never exceed the number of dimensions in the data (e.g., using area to show one-dimensional data).

  • All plots are designed to have no chartjunk (like moiré vibrations, fake perspective, dark grid lines, etc.) (Tufte, 2001, Chapter 5).

There are some instances where ggstatsplot graphs don’t follow principles of clean graphics, as formulated in the Tufte theory of data graphics (Tufte, 2001, Chapter 4). The theory has four key principles:

  1. Above all else show the data.
  2. Maximize the data-ink ratio.
  3. Erase non-data-ink.
  4. Erase redundant data-ink, within reason.

In particular, default plots in ggstatsplot can sometimes violate one of the principles from 2-4. According to these principles, every bit of ink should have reason for its inclusion in the graphic and should convey some new information to the viewer. If not, such ink should be removed. One instance of this is bilateral symmetry of data measures. For example, in Figure-1, we can see that both the box and violin plots are mirrored, which consumes twice the space in the graphic without adding any new information. But this redundancy is tolerated for the sake of beauty that such symmetrical shapes can bring to the graphic. Even Tufte admits that efficiency is but one consideration in the design of statistical graphics (Tufte, 2001, p. 137). Additionally, these principles were formulated in an era in which computer graphics had yet to revolutionize the ease with which graphics could be produced and thus some of the concerns about minimizing data-ink for easier production of graphics are not as relevant as they were.

Statistical analysis

As an extension of ggplot2, ggstatsplot has the same expectations about the structure of the data. More specifically,

  • The data should be an object of class data.frame (a tibble dataframe will also work).

  • The data should be organized following the principles of tidy data, which specify how statistical structure of a data frame (variables and observations) should be mapped to physical structure (columns and rows). More specifically, tidy data means all variables have their own columns and each row corresponds to a unique observation (Wickham, 2014).

  • All ggstatsplot functions remove NAs from variables of interest (similar to ggplot2; Wickham, 2016, p.207) in the data and display total sample size (n) in the subtitle to inform the user/reader about the number of observations included for both the statistical analysis and the visualization. But, when sample sizes differ across tests in the same function, ggstatsplot makes an effort to inform the user of this aspect (Figure-4).

**Figure-4**. `ggstatsplot` functions remove `NA`s from variables of interest and display total sample size *n*, but they can give more nuanced information about sample sizes when *n* differs across tests. For example, `ggcorrmat` will display (a) only one total sample size once when no `NA`s present, but (b) will instead show minimum, median, and maximum sample sizes across all correlation tests when `NA`s are present across correlation variables.

Figure-4. ggstatsplot functions remove NAs from variables of interest and display total sample size n, but they can give more nuanced information about sample sizes when n differs across tests. For example, ggcorrmat will display (a) only one total sample size once when no NAs present, but (b) will instead show minimum, median, and maximum sample sizes across all correlation tests when NAs are present across correlation variables.

Types of statistics supported

Functions Description Parametric Non-parametric Robust Bayes Factor
ggbetweenstats Between group/condition comparisons \(\checkmark\) \(\checkmark\) \(\checkmark\) \(\checkmark\)
gghistostats Distribution of a numeric variable \(\checkmark\) \(\checkmark\) \(\checkmark\) \(\checkmark\)
ggcorrmat Correlation matrix \(\checkmark\) \(\checkmark\) \(\checkmark\) \(\times\)
ggscatterstats Correlation between two variables \(\checkmark\) \(\checkmark\) \(\checkmark\) \(\checkmark\)
ggpiestats Association between categorical variables \(\checkmark\) \(\times\) \(\times\) \(\checkmark\)
ggcoefstats Regression model coefficients \(\checkmark\) \(\times\) \(\checkmark\) \(\times\)

Types of statistical tests supported

Functions Type Test Effect size 95% CI available?
ggbetweenstats Parametric Student’s and Welch’s t-test Cohen’s d, Hedge’s g \(\checkmark\)
ggbetweenstats Parametric Fisher’s and Welch’s one-way ANOVA \[\eta^2_p, \omega^2_p\] \(\checkmark\)
ggbetweenstats Non-parametric Mann-Whitney U-test r \(\checkmark\)
ggbetweenstats Non-parametric Kruskal-Wallis Rank Sum Test \[\eta^2_H\] \(\checkmark\)
ggbetweenstats Robust Yuen’s test for trimmed means \[\xi\] \(\checkmark\)
ggbetweenstats Robust Heteroscedastic one-way ANOVA for trimmed means \[\xi\] \(\checkmark\)
ggpiestats Parametric \[\text{Pearson's}~ \chi^2 ~\text{test}\] Cramer’s V \(\checkmark\)
ggpiestats Parametric McNemar’s test odds ratio (OR) \(\checkmark\)
ggscatterstats/ggcorrmat Parametric Pearson’s r r \(\checkmark\)
ggscatterstats/ggcorrmat Non-parametric \[\text{Spearman's}~ \rho\] \[\rho\] \(\checkmark\)
ggscatterstats/ggcorrmat Robust Percentage bend correlation r \(\checkmark\)
gghistostats/ggdotplotstats Parametric One-sample t-test Cohen’s d \(\checkmark\)
gghistostats Non-parametric One-sample Wilcoxon signed rank test Cohen’s d \(\times\)
gghistostats/ggdotplotstats Robust One-sample percentile bootstrap robust estimator \(\checkmark\)
gghistostats/ggdotplotstats Parametric Regression models \[\beta\] \(\checkmark\)

For the ggbetweenstats function, the following post-hoc tests are available for (adjusted) pairwise multiple comparisons:

Type Design Equal variance assumed? Pairwise comparison test p-value adjustment?
Parametric between-subjects No Games-Howell test \(\checkmark\)
Parametric between-subjects Yes Student’s t-test \(\checkmark\)
Parametric within-subjects NA Student’s t-test \(\checkmark\)
Non-parametric between-subjects No Dwass-Steel-Crichtlow-Fligner test \(\checkmark\)
Non-parametric within-subjects No Durbin-Conover test \(\checkmark\)
Robust between-subjects No Yuen’s trimmed means test \(\checkmark\)
Robust within-subjects NA Yuen’s trimmed means test \(\checkmark\)
Bayes Factor between-subjects No \(\times\) \(\times\)
Bayes Factor between-subjects Yes \(\times\) \(\times\)
Bayes Factor within-subjects NA \(\times\) \(\times\)

Note-

  • NA: not applicable
  • available methods for p-value adjustment: “holm”, “hochberg”, “hommel”, “bonferroni”, “BH”, “BY”, “fdr”, “none”

Statistical variation

One of the important functions of a plot is to show the variation in the data, which comes in two forms:

  • Measurement noise: In ggstatsplot, the actual variation in measurements is shown by plotting a combination of (jittered) raw data points with a boxplot laid on top (Figure-1) or a histogram (Figure-5). None of the plots, where empirical distribution of the data is concerned, show the sample standard deviation because they are poor at conveying information about limits of the sample and presence of outliers (Cleveland, 1985, p.220).
**Figure-5**. Distribution of a variable shown using `gghistostats`.

Figure-5. Distribution of a variable shown using gghistostats.

  • Sample-to-sample statistic variation: Although, traditionally, this variation has been shown using the standard error of the mean (SEM) of the statistic, ggstatsplot plots instead use 95% confidence intervals (e.g., Figure-6). This is because the interval formed by error bars correspond to a 68% confidence interval, which is not a particularly interesting interval (Cleveland, 1985, p.222-225).
**Figure-6**. Sample-to-sample variation in regression estimates is displayed using confidence intervals in `ggcoefstats`.

Figure-6. Sample-to-sample variation in regression estimates is displayed using confidence intervals in ggcoefstats.

Reporting results

The default setting in ggstatsplot is to produce plots with statistical details included. Most often than not, the results are displayed as a subtitle in the plot. Great care has been taken into which details are included in statistical reporting and why.

  1. APA guidelines (APA, 2009) are followed (for the most part) by default:
  • Percentages are displayed with no decimal places (Figure-2).
  • Correlations, t-tests, and chi-squared tests are reported with the degrees of freedom in parentheses and the significance level (Figure-2, Figure-3b, Figure-5).
  • ANOVAs are reported with two degrees of freedom and the significance level (Figure-1).
  • Regression results are presented with the unstandardized or standardized estimate (beta), whichever was specified by the user, along with the statistic (depending on the model, this can be a t or z statistic) and the corresponding significance level (Figure-6).
  • With the exception of p-values, most statistics are rounded to two decimal places.
  1. Default statistical tests:

  2. Dealing with null results:

  3. Avoiding the “p-value error”:
    The p-value indexes the probability that the researchers have falsely rejected a true null hypothesis (Type I error, i.e.) and can rarely be exactly 0. And yet over 97,000 manuscripts on Google Scholar report the p-value to be p = 0.000 (Lilienfeld et al., 2015), putatively due to relying on default computer outputs. All p-values displayed in ggstatsplot plots avoid this mistake. Anything less than p < 0.001 is displayed as such (e.g, Figure-1). The package deems it unimportant how infinitesimally small the p-values are and, instead, puts emphasis on the effect size magnitudes and their 95% CIs.

Conclusion

References

Appendix

Appendix A: Documentation

There are three main documents one can rely on to learn how to use ggstatsplot:

Appendix B: Suggestions

If you find any bugs or have any suggestions/remarks, please file an issue on GitHub repository for this package: https://github.com/IndrajeetPatil/ggstatsplot/issues

Appendix C: Session information

Summarizing session information for reproducibility.

options(width = 200)
devtools::session_info()
#> - Session info ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#>  setting  value                                             
#>  version  R Under development (unstable) (2018-11-30 r75724)
#>  os       Windows 10 x64                                    
#>  system   x86_64, mingw32                                   
#>  ui       RTerm                                             
#>  language (EN)                                              
#>  collate  English_United States.1252                        
#>  ctype    English_United States.1252                        
#>  tz       Asia/Calcutta                                     
#>  date     2018-12-16                                        
#> 
#> - Packages -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#>  package      * version     date       lib source                              
#>  assertthat     0.2.0       2017-04-11 [1] CRAN (R 3.5.1)                      
#>  backports      1.1.3       2018-12-14 [1] CRAN (R 3.6.0)                      
#>  BayesFactor    0.9.12-4.2  2018-05-19 [1] CRAN (R 3.5.1)                      
#>  bayesplot      1.6.0       2018-08-02 [1] CRAN (R 3.5.1)                      
#>  bindr          0.1.1       2018-03-13 [1] CRAN (R 3.5.1)                      
#>  bindrcpp     * 0.2.2       2018-03-29 [1] CRAN (R 3.5.1)                      
#>  boot           1.3-20      2017-08-06 [2] CRAN (R 3.6.0)                      
#>  broom          0.5.1.9000  2018-12-09 [1] Github (tidymodels/broom@9a85f75)   
#>  broom.mixed    0.2.3.9000  2018-11-28 [1] Github (bbolker/broom.mixed@b3ad950)
#>  ca             0.71        2018-10-10 [1] CRAN (R 3.5.1)                      
#>  callr          3.1.0       2018-12-10 [1] CRAN (R 3.6.0)                      
#>  cellranger     1.1.0       2016-07-27 [1] CRAN (R 3.5.1)                      
#>  cli            1.0.1.9000  2018-10-30 [1] Github (r-lib/cli@56538e3)          
#>  cluster        2.0.7-1     2018-04-13 [2] CRAN (R 3.6.0)                      
#>  coda           0.19-2      2018-10-08 [1] CRAN (R 3.5.1)                      
#>  codetools      0.2-15      2016-10-05 [2] CRAN (R 3.6.0)                      
#>  coin           1.2-2       2017-11-28 [1] CRAN (R 3.5.1)                      
#>  colorspace     1.3-2       2016-12-14 [1] CRAN (R 3.5.1)                      
#>  commonmark     1.7         2018-12-01 [1] CRAN (R 3.6.0)                      
#>  cowplot        0.9.99      2018-10-29 [1] Github (wilkelab/cowplot@a4ccd77)   
#>  crayon         1.3.4       2017-09-16 [1] CRAN (R 3.5.1)                      
#>  data.table     1.11.8      2018-09-30 [1] CRAN (R 3.5.1)                      
#>  DEoptimR       1.0-8       2016-11-19 [1] CRAN (R 3.5.0)                      
#>  desc           1.2.0       2018-10-30 [1] Github (r-lib/desc@7c12d36)         
#>  devtools       2.0.1       2018-10-26 [1] CRAN (R 3.5.1)                      
#>  digest         0.6.18      2018-10-10 [1] CRAN (R 3.5.1)                      
#>  dplyr        * 0.7.8       2018-11-10 [1] CRAN (R 3.5.1)                      
#>  effsize        0.7.1       2017-03-21 [1] CRAN (R 3.5.1)                      
#>  emmeans        1.3.1       2018-12-13 [1] CRAN (R 3.6.0)                      
#>  estimability   1.3         2018-02-11 [1] CRAN (R 3.5.0)                      
#>  evaluate       0.12        2018-10-09 [1] CRAN (R 3.5.1)                      
#>  exact2x2       1.6.3       2018-07-27 [1] CRAN (R 3.5.1)                      
#>  exactci        1.3-3       2017-10-02 [1] CRAN (R 3.5.0)                      
#>  extrafont      0.17        2014-12-08 [1] CRAN (R 3.5.0)                      
#>  extrafontdb    1.0         2012-06-11 [1] CRAN (R 3.5.0)                      
#>  fit.models     0.5-14      2017-04-06 [1] CRAN (R 3.5.1)                      
#>  forcats      * 0.3.0       2018-02-19 [1] CRAN (R 3.5.1)                      
#>  foreign        0.8-71      2018-07-20 [2] CRAN (R 3.6.0)                      
#>  fs             1.2.6       2018-08-23 [1] CRAN (R 3.5.1)                      
#>  generics       0.0.2       2018-11-29 [1] CRAN (R 3.6.0)                      
#>  ggcorrplot     0.1.2.9000  2018-10-28 [1] local                               
#>  ggExtra        0.8         2018-11-08 [1] Github (daattali/ggExtra@f2a04df)   
#>  ggplot2      * 3.1.0.9000  2018-12-15 [1] Github (tidyverse/ggplot2@9292832)  
#>  ggrepel        0.8.0       2018-05-09 [1] CRAN (R 3.5.1)                      
#>  ggridges       0.5.1       2018-09-27 [1] CRAN (R 3.5.1)                      
#>  ggsignif       0.4.0       2017-08-03 [1] CRAN (R 3.5.1)                      
#>  ggstatsplot  * 0.0.7.9000  2018-12-16 [1] local                               
#>  glmmTMB        0.2.2.0     2018-07-03 [1] CRAN (R 3.5.1)                      
#>  glue           1.3.0       2018-07-17 [1] CRAN (R 3.5.1)                      
#>  gnm            1.1-0       2018-06-21 [1] CRAN (R 3.5.1)                      
#>  groupedstats   0.0.4.9000  2018-12-12 [1] local                               
#>  gtable         0.2.0       2016-02-26 [1] CRAN (R 3.5.1)                      
#>  gtools         3.8.1       2018-06-26 [1] CRAN (R 3.5.0)                      
#>  haven          2.0.0       2018-11-22 [1] CRAN (R 3.6.0)                      
#>  highr          0.7         2018-06-09 [1] CRAN (R 3.5.1)                      
#>  hms            0.4.2       2018-03-10 [1] CRAN (R 3.5.1)                      
#>  hrbrthemes     0.5.0.1     2018-08-19 [1] CRAN (R 3.5.1)                      
#>  htmltools      0.3.6       2017-04-28 [1] CRAN (R 3.5.1)                      
#>  httpuv         1.4.5       2018-07-19 [1] CRAN (R 3.5.1)                      
#>  httr           1.4.0       2018-12-11 [1] CRAN (R 3.6.0)                      
#>  jmv            0.9.6       2018-12-11 [1] CRAN (R 3.6.0)                      
#>  jmvcore        0.9.5.2     2018-12-10 [1] CRAN (R 3.6.0)                      
#>  jsonlite       1.6         2018-12-07 [1] CRAN (R 3.6.0)                      
#>  knitr          1.21        2018-12-10 [1] CRAN (R 3.6.0)                      
#>  labeling       0.3         2014-08-23 [1] CRAN (R 3.5.0)                      
#>  later          0.7.5       2018-09-18 [1] CRAN (R 3.5.1)                      
#>  lattice        0.20-38     2018-11-04 [2] CRAN (R 3.6.0)                      
#>  lazyeval       0.2.1       2017-10-29 [1] CRAN (R 3.5.1)                      
#>  lme4           1.1-19      2018-11-10 [1] CRAN (R 3.6.0)                      
#>  lmtest         0.9-36      2018-04-04 [1] CRAN (R 3.5.1)                      
#>  lubridate      1.7.4       2018-04-11 [1] CRAN (R 3.5.1)                      
#>  magrittr       1.5         2014-11-22 [1] CRAN (R 3.5.1)                      
#>  MASS           7.3-51.1    2018-11-01 [2] CRAN (R 3.6.0)                      
#>  Matrix         1.2-15      2018-11-01 [1] CRAN (R 3.5.1)                      
#>  MatrixModels   0.4-1       2015-08-22 [1] CRAN (R 3.5.1)                      
#>  mc2d           0.1-18      2017-03-06 [1] CRAN (R 3.5.1)                      
#>  memoise        1.1.0       2017-04-21 [1] CRAN (R 3.5.1)                      
#>  metafor        2.0-0       2017-06-22 [1] CRAN (R 3.5.1)                      
#>  mime           0.6         2018-10-05 [1] CRAN (R 3.5.1)                      
#>  miniUI         0.1.1.1     2018-05-18 [1] CRAN (R 3.5.1)                      
#>  minqa          1.2.4       2014-10-09 [1] CRAN (R 3.5.1)                      
#>  mnormt         1.5-5       2016-10-15 [1] CRAN (R 3.5.0)                      
#>  modelr         0.1.2       2018-05-11 [1] CRAN (R 3.5.1)                      
#>  modeltools     0.2-22      2018-07-16 [1] CRAN (R 3.5.1)                      
#>  multcomp       1.4-8       2017-11-08 [1] CRAN (R 3.5.1)                      
#>  munsell        0.5.0       2018-06-12 [1] CRAN (R 3.5.1)                      
#>  mvtnorm        1.0-8       2018-05-31 [1] CRAN (R 3.5.0)                      
#>  nlme           3.1-137     2018-04-07 [2] CRAN (R 3.6.0)                      
#>  nloptr         1.2.1       2018-10-03 [1] CRAN (R 3.5.1)                      
#>  nnet           7.3-12      2016-02-02 [2] CRAN (R 3.6.0)                      
#>  paletteer      0.1.0       2018-07-10 [1] CRAN (R 3.5.1)                      
#>  pbapply        1.3-4       2018-01-10 [1] CRAN (R 3.5.0)                      
#>  pcaPP          1.9-73      2018-01-14 [1] CRAN (R 3.5.0)                      
#>  pillar         1.3.1       2018-12-15 [1] CRAN (R 3.6.0)                      
#>  pkgbuild       1.0.2       2018-10-16 [1] CRAN (R 3.5.1)                      
#>  pkgconfig      2.0.2       2018-08-16 [1] CRAN (R 3.5.1)                      
#>  pkgdown        1.3.0       2018-12-07 [1] CRAN (R 3.6.0)                      
#>  pkgload        1.0.2       2018-10-29 [1] CRAN (R 3.6.0)                      
#>  plyr           1.8.4       2016-06-08 [1] CRAN (R 3.5.1)                      
#>  prediction     0.3.6.1     2018-12-04 [1] CRAN (R 3.6.0)                      
#>  prettyunits    1.0.2       2015-07-13 [1] CRAN (R 3.5.1)                      
#>  processx       3.2.1       2018-12-05 [1] CRAN (R 3.6.0)                      
#>  promises       1.0.1       2018-04-13 [1] CRAN (R 3.5.1)                      
#>  ps             1.2.1       2018-11-06 [1] CRAN (R 3.6.0)                      
#>  psych          1.8.11.15   2018-11-16 [1] local                               
#>  purrr        * 0.2.5.9000  2018-11-28 [1] Github (tidyverse/purrr@b4ae036)    
#>  purrrlyr       0.0.3.9000  2018-11-17 [1] Github (hadley/purrrlyr@3feae44)    
#>  pwr            1.2-2       2018-03-03 [1] CRAN (R 3.5.1)                      
#>  qvcalc         0.9-1       2017-09-19 [1] CRAN (R 3.5.1)                      
#>  R6             2.3.0       2018-10-04 [1] CRAN (R 3.5.1)                      
#>  Rcpp           1.0.0       2018-11-07 [1] CRAN (R 3.6.0)                      
#>  readr        * 1.3.0.9000  2018-12-12 [1] Github (tidyverse/readr@27da43e)    
#>  readxl         1.1.0       2018-04-20 [1] CRAN (R 3.5.1)                      
#>  relimp         1.0-5       2016-03-30 [1] CRAN (R 3.5.1)                      
#>  remotes        2.0.2       2018-10-30 [1] CRAN (R 3.6.0)                      
#>  reshape        0.8.8       2018-10-23 [1] CRAN (R 3.6.0)                      
#>  reshape2       1.4.3       2017-12-11 [1] CRAN (R 3.5.1)                      
#>  rjson          0.2.20      2018-06-08 [1] CRAN (R 3.5.0)                      
#>  rlang        * 0.3.0.1     2018-10-25 [1] CRAN (R 3.5.1)                      
#>  rmarkdown      1.11        2018-12-08 [1] CRAN (R 3.6.0)                      
#>  robust         0.4-18      2017-04-27 [1] CRAN (R 3.5.1)                      
#>  robustbase     0.93-3      2018-09-21 [1] CRAN (R 3.5.1)                      
#>  roxygen2       6.1.1       2018-11-07 [1] CRAN (R 3.6.0)                      
#>  rprojroot      1.3-2       2018-01-03 [1] CRAN (R 3.5.1)                      
#>  rrcov          1.4-7       2018-11-15 [1] CRAN (R 3.6.0)                      
#>  rstudioapi     0.8         2018-10-02 [1] CRAN (R 3.5.1)                      
#>  Rttf2pt1       1.3.7       2018-06-29 [1] CRAN (R 3.5.0)                      
#>  rvest          0.3.2       2016-06-17 [1] CRAN (R 3.5.1)                      
#>  sandwich       2.5-0       2018-08-17 [1] CRAN (R 3.5.1)                      
#>  scales         1.0.0       2018-08-09 [1] CRAN (R 3.5.1)                      
#>  sessioninfo    1.1.1       2018-11-05 [1] CRAN (R 3.6.0)                      
#>  shiny          1.2.0       2018-11-02 [1] CRAN (R 3.5.1)                      
#>  sjlabelled     1.0.15      2018-11-22 [1] CRAN (R 3.6.0)                      
#>  sjmisc         2.7.6       2018-11-06 [1] CRAN (R 3.6.0)                      
#>  sjstats        0.17.2      2018-11-15 [1] CRAN (R 3.6.0)                      
#>  skimr          1.0.3       2018-06-07 [1] CRAN (R 3.5.1)                      
#>  snakecase      0.9.2       2018-08-14 [1] CRAN (R 3.5.1)                      
#>  ssanv          1.1         2015-06-23 [1] CRAN (R 3.5.0)                      
#>  stringdist     0.9.5.1     2018-06-08 [1] CRAN (R 3.5.1)                      
#>  stringi        1.2.4       2018-07-20 [1] CRAN (R 3.6.0)                      
#>  stringr      * 1.3.1       2018-05-10 [1] CRAN (R 3.5.1)                      
#>  survival       2.43-3      2018-11-26 [2] CRAN (R 3.6.0)                      
#>  testthat       2.0.1       2018-10-13 [1] CRAN (R 3.5.1)                      
#>  TH.data        1.0-9       2018-07-10 [1] CRAN (R 3.5.1)                      
#>  tibble       * 1.4.99.9006 2018-11-29 [1] Github (tidyverse/tibble@a248cf1)   
#>  tidyr        * 0.8.2       2018-10-28 [1] CRAN (R 3.6.0)                      
#>  tidyselect     0.2.5       2018-10-11 [1] CRAN (R 3.5.1)                      
#>  tidyverse    * 1.2.1       2017-11-14 [1] CRAN (R 3.5.1)                      
#>  TMB            1.7.15      2018-11-09 [1] CRAN (R 3.6.0)                      
#>  usethis        1.4.0.9000  2018-12-12 [1] Github (r-lib/usethis@923dd75)      
#>  vcd            1.4-4       2017-12-06 [1] CRAN (R 3.5.1)                      
#>  vcdExtra       0.7-1       2017-09-29 [1] CRAN (R 3.5.1)                      
#>  withr          2.1.2       2018-03-15 [1] CRAN (R 3.5.1)                      
#>  WRS2           0.10-0      2018-06-15 [1] CRAN (R 3.5.1)                      
#>  xfun           0.4         2018-10-23 [1] CRAN (R 3.6.0)                      
#>  xml2           1.2.0       2018-01-24 [1] CRAN (R 3.5.1)                      
#>  xtable         1.8-3       2018-08-29 [1] CRAN (R 3.5.1)                      
#>  yaml           2.2.0       2018-07-25 [1] CRAN (R 3.5.1)                      
#>  zoo            1.8-4       2018-09-19 [1] CRAN (R 3.5.1)                      
#> 
#> [1] C:/Users/inp099/Documents/R/win-library/3.6
#> [2] C:/Program Files/R/R-devel/library