Hi folks,

We have a HCI group meeting on this Tuesday at 11:45 @DGP Seminar room. This meeting, Fanny will introduce her current project , 'explorable multiverse analyses' , and have a discussion about the project. Please join the meeting and have a great discussion. Details are below.

Thanks,

Seyong

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

When performing statistical analysis on empirical data, researchers inevitably make a series of arbitrary choices among several options for processing data (i.e. exclusions of participants for various reasonable reasons, aggregation of participants based on one data dimension, data transformations, etc.) and analysing it (i.e. frequentist vs. Bayesian inferential models, corrections, inclusion or exclusion of different variables, etc.). Due to space restrictions (and possibly other reasons), researchers typically practice selective reporting of their results, that is, only one particular possibility among the larger landscape of reasonable options, even if they have experimented with multiple paths in their research. But these arbitrary choices may be misleading, in that some choices may result in fluctuations in conclusions. Revealing the outcome of the multiple (reasonable) analysis paths for a data set would inform the reader on the fragility or robustness of the presented results.

Together with colleagues I am working on a project that introduces explorable multiverse analyses, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper itself. This approach draws from two recent ideas: multiverse analysis http://www.stat.columbia.edu/~gelman/research/published/multiverse_publishe d.pdf a new philosophy of statistical reporting where the researcher conducts many different statistical analyses and summarizes all outcomes in their paper in order to show how fragile or robust their findings are; and explorable explanations http://worrydream.com/ExplorableExplanations/ textual or illustrated explanations that can be read as normal explanations, but where the reader can also become active by interacting with some elements of the explanation.

I will show early prototypes and elicit ideas and thoughts from you on this idea from your experience as an author, reader, and (possibly) reviewer of academic papers.