Our new paper in Cortex with Sam Cason, “No Evidential Value in Samples of tDCS Studies of Cognition and Working Memory in Healthy Populations“, uses a p-curve analysis to examine the strength of evidence in the tDCS literature. Briefly, a p-curve analysis is used to examine the distribution of significant p-values (p < .05) in a literature.  If there is a real effect in the literature, the p-value distribution should be right-skewed, such that there are more p-values ranging from .00 to .01 compared to a bin ranging from .04 to .05.  Interestingly, if there is no real underlying effect, the p-curve should be flat (i.e. an equal distribution of p-values in these bins).  Furthermore, if someone is engaging in “questionable research practices”, this distribution may be left-skewed (more p-values from .04 to .05 vs. .00 to .01).

We used the p-curve analysis for two sets of studies in the tDCS literature.  One was a random selection of tDCS studies on cognition, and the second was a set of tDCS studies on working memory taken from a recent meta-analysis.  The p-value distribution from our sample of cognitive studies was not right-skewed, and our best estimate is that 8-16% of the studies from this sample would replicate.  For the sample of working memory papers, our estimate is that 5% of the studies would replicate.  We note that, given an α of .05, this is the same number of studies that would be expected to replicate if the data were randomly generated.

In the discussion, we note that these results do not mean that tDCS does not, or can not, influence cognitive processes. However, we are concerned that current practices in the general psychological literature (including small sample sizes, hypothesizing after the results are known, and multiple data analyses (the “garden of forking paths“) may lead to a literature with a low signal-to-noise ratio. Therefore, we strongly support preregistration of future tDCS studies, and replication of past tDCS studies, to better understand the relationship between brain stimulation and cognitive processing.