Meta-analysis course: part 4: Publication bias.

class: center, middle, inverse, title-slide

# Meta-analysis course: part 4: Publication bias.
### Thomas Pollet (@tvpollet), Northumbria University
### 2019-09-16 | <a href="http://tvpollet.github.io/disclaimer">disclaimer</a>

---

## Outline this section.

* Principles of systematic reviews / meta-analysis

* Effect sizes

* Fixed vs. random effects meta-analyses --> effect size distributions.

* **Publication bias**

* Moderators and metaregression

* Advanced 'stuff'.

---
## Overview

Past 2 sections:

- Effect sizes

- How to **pool effects** in meta-analysis, choose the **right pooling model** (fixed/random), assess the **heterogeneity** of your effect estimate.

--> These have implicitly assumed a 'perfect world'.

**Important: this builds on previous sections so if you are coding along make sure the data and objects are loaded.**

---
## Publication bias

Even the most thoroughly conducted meta-analysis can only work with the **available study material**.

**File-drawer** or **Publication bias** problem, which states that a study with a significant effect **is more likely to be published** than a study without a significant (/strong) effect. Moreover, studies with strong effects more likely to make it into the literature ([Ioannidis, 2008](http://sci-hub.tw/https://insights.ovid.com/crossref?an=00001648-200809000-00002)).
 
Such **missing studies** with low effect sizes, thus might never get published and therefore cannot be integrated into our meta-analysis. This leads to **publication bias**, as the pooled effect we estimate in our meta-analysis might be higher than the **true effect size** because we did not consider the missing studies with lower effects due to the simple fact that they were never published.

Although this practice is gradually changing (e.g., registered reports), whether a study is published or not heavily depends on the **statistical significance** ( `$p<0.05$` ) of its results. For any sample size, a result is more likely to become **statistically significant** if its **effect size is high**. This is particularly true for **small studies**, for which very large effect sizes (but unreliable).

---
## Publication bias II

Formal definition: _the tendency toward preparation, submission and publication of research findings based on the nature and direction of the research results_ (Dickersin, 2005: 13)

---
## What is your 'favourite' example of publication bias.

**Thomas opens gosoapbox**

---
## Ongoing debate.

Traditional methods assume that publication bias is primarily driven by **effect size** and because researchers **immediately put every study in the file drawer once the results are not significant**.

Recently: **assumptions may not be true**, and that publication bias is mostly caused by **significance levels** and **p-hacking** (Simonsohn et al., 2014). An alternative method called **P-Curve** has therefore been suggested to examine **publication bias**.

**P-curve** was developed with **full-blown experimental psychological research in mind**, in which researchers often have **high degrees of "researcher freedom"** (Simmons et al., 2011) in deleting outliers and performing statistical test on their data.

In some other fields (e.g., evolutionary biology, epidemiology), the **effect size** has always been more of a focus than significance.

One solution: conducting both analyses and reporting them, perhaps  most adequate approach until meta-scientific research gives us more certainty about which method is best.

???
Again I have not followed this through yet entirely in my own research.

---
## Publication bias: Small-study effect methods.

The **small-study effect methods** are conventional for many years. Some of these have (well-documented) biases, but you might still be asked to perform them... . We will only focus on the most important ones here.

**The model behind small-study effects methods**

According to Borenstein et al. (2009): 3 core **assumptions**:
1.  Because they involve large commitment of resources and time, **large studies are most likely to get published**, regardless of significant or not.
2.  Moderately sized studies are at **greater risk of missing**, but with a moderate sample size: moderately sized effects are likely to become significant --> only some studies will be missing,... .
3.  Small studies are **at greatest risk** for being non-significant, and thus being missing. Only small studies with a very large effect size become significant, and will be found in the published literature.

In accordance with these assumptions, the methods we present here particularly focus **on small studies with small effect sizes, and whether they are missing**.

---
## Funnel plots

The best way is to visualize through **funnel plots**.

---
## Funnel plot: Example.
Funnel plot for our `model_hksj` (fom meta-analysis slides 3) meta-analysis output using the `funnel()` function in the `meta` package.

```r
funnel(model_hksj,xlab = "Hedges' g") # adds label on X
```

![](Meta-analysis_4_files/figure-html/funnel-1.svg)

---
## Funnel plot: What does it show?

The **funnel plot** basically consists of a **funnel** and two **axes**: the y-axis showing the **Standard Error** `$SE$` (larger studies (which thus have a smaller `$SE$`) plotted **on top of the y-axis**; and the x-axis showing the **effect size** of each study.

When there is **no publication bias**: all studies would lie **symmetrically around our pooled effect size (the striped line)** within the shape of the funnel.

When **publication bias is present**: the funnel would look asymmetrical, because only the small studies with a large effect size very published, **while small studies without a significant, large effect would be missing**.

In the case of our meta-anlysis: publication bias present. Plot is highly asymmetrical, and no data the bottom-left corner of our plot.

We can also display the name of each study using the `studlab` parameter.

---
## Adding labels.

```r
funnel(model_hksj,xlab = "g",studlab = TRUE)
```

![](Meta-analysis_4_files/figure-html/funnel labels-1.svg)

???
3 outlying studies driving the asymmetry, large effect size but small sample.

---
## Other Y-axes (`yaxis` command)

![](Meta-analysis_4_files/figure-html/different funnels-1.svg)

---
## Contour enhanced funnelplot

```r
meta::funnel(model_hksj, xlab="Hedges' g",level = 0.95, contour = c(0.95, 0.975, 0.99),
 col.contour = c("darkgray", "gray", "lightgray"),
 lwd = 2, cex = 2, pch = 16)
legend(1.25, 0.10,
 c("0.05 > p > 0.025", "0.025 > p > 0.01", "< 0.01"),
 fill = c("darkgray", "gray", "lightgray"), bg=c("white"))
```

---
## Result

![](Meta-analysis_4_files/figure-html/unnamed-chunk-2-1.svg)

???
Some studies have statistically significant effect sizes (grey background), others do not (white background). We also see a trend that while the moderately sized studies are partly significant and non-significant, with slightly more significant studies, the asymmetry is much larger for small studies. This gives us a hint that publication bias might indeed be present in our analysis.

---
## Funnel plots: Evidence for publication bias?

* Although funnel plots are a helpful visualization technique, their interpretation is subjective and can be misleading.

* The presence of asymmetry does not prove the existence of publication bias.

* Egger, Smith, Schneider, and Minder (1997:632): number of ways in which asymmetries can be created in funnel plots, including true heterogeneity, data irregularities (e.g., methodologically poorly designed small studies), or fraud.

--> Funnel plots can be helpful in identifying possible publication bias; however, recommended that they are combined with (a) statistical test(s).

---
## Formal statistical tests of publication bias.

* Regression tests (associated with funnel plot) (Egger's test)

* non-parametric tests using rank-correlation methods, introduced and influenced by Begg and Mazumdar (1994)

--> some other methods also exist.

---
## Egger's test.

**Egger's test of the intercept** quantifies funnel plot asymmetry and performs a statistical test (Egger et al., 1997).

```r
metabias(model_hksj)
```

```
## 
## 	Linear regression test of funnel plot asymmetry
## 
## data:  model_hksj
## t = 4.677, df = 16, p-value = 0.0002525
## alternative hypothesis: asymmetry in funnel plot
## sample estimates:
##       bias    se.bias      slope 
##  4.1111350  0.8790029 -0.3407464
```

---
## More about `metabias`.

* Needs minimum k=10 (Sterne et al., 2011)
* Further variants for binary outcomes (see manual)

---
## Duval & Tweedie: Trim and fill.

**Duval & Tweedie's trim-and-fill procedure** also based the funnel plot and its symmetry/asymmetry.

Five steps (Schwarzer et al., 2015):

1. Estimating the number of studies in the outlying (right) part of the funnel plot
2. Removing (trimming) these effect sizes and pooling the results with the remaining effect sizes
3. This pooled effect is then taken as the center of all effect sizes
4. For each trimmed/removed study, an additional study is imputed, mirroring the effect of the study on the left side of the funnel plot
5. Pooling the results with the imputed studies and the trimmed studies included

---
## Trim and fill in `meta`

```r
trimfill(model_hksj)
```

```
## SMD 95%-CI %W(random)
## Call et al. 0.7091 [ 0.1979; 1.2203] 3.8
## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 4.1
## DanitzOrsillo 1.7912 [ 1.1139; 2.4685] 3.3
## de Vibe et al. 0.1825 [-0.0484; 0.4133] 4.4
## Frazier et al. 0.4219 [ 0.1380; 0.7057] 4.3
## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 4.1
## Gallego et al. 0.7249 [ 0.2846; 1.1652] 4.0
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 4.0
## Hintz et al. 0.2840 [-0.0453; 0.6133] 4.2
## Kang et al. 1.2751 [ 0.6142; 1.9360] 3.4
## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 4.1
## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 3.9
## Phang et al. 0.5407 [ 0.0619; 1.0196] 3.9
## Rasanen et al. 0.4262 [-0.0794; 0.9317] 3.8
## Ratanasiripong 0.5154 [-0.1731; 1.2039] 3.3
## Shapiro et al. 1.4797 [ 0.8618; 2.0977] 3.5
## SongLindquist 0.6126 [ 0.1683; 1.0569] 4.0
## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 3.9
## Filled: Warnecke et al. 0.0520 [-0.4360; 0.5401] 3.9
## Filled: SongLindquist 0.0395 [-0.4048; 0.4837] 4.0
## Filled: Frogeli et al. 0.0220 [-0.3621; 0.4062] 4.1
## Filled: Call et al. -0.0571 [-0.5683; 0.4541] 3.8
## Filled: Gallego et al. -0.0729 [-0.5132; 0.3675] 4.0
## Filled: Kang et al. -0.6230 [-1.2839; 0.0379] 3.4
## Filled: Shapiro et al. -0.8277 [-1.4456; -0.2098] 3.5
## Filled: DanitzOrsillo -1.1391 [-1.8164; -0.4618] 3.3
## 
## Number of studies combined: k = 26 (with 8 added studies)
## 
## SMD 95%-CI t p-value
## Random effects model 0.3431 [ 0.0994; 0.5868] 2.90 0.0077
## Prediction interval [-0.8463; 1.5326] 
## 
## Quantifying heterogeneity:
## tau^2 = 0.3181; H = 2.05 [1.70; 2.47]; I^2 = 76.2% [65.4%; 83.7%]
## 
## Test of heterogeneity:
## Q d.f. p-value
## 105.15 25 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
## - Trim-and-fill method to adjust for funnel plot asymmetry
```

---
## Let's put in a sink()

```r
sink("trim_model_hksj.txt")
trimfill(model_hksj)
```

```r
sink()
```

???
We see that the procedure identified and trimmed **eight studies** `(with 8 added studies)`). The overall effect estimated by the procedure is `$g = 0.34$`.

---
## Interpretation.

The initial pooled effect size was `$g = 0.59$`, which is substantially larger than the bias-corrected effect.

If we assume that **publication bias** was a problem in the analyses, the **trim-and-fill procedure** allows us to assume that our initial results were **overestimated** due to publication bias, and the "true" effect when controlling for selective publication might be `$g = 0.34$`, rather than `$g = 0.59$`.

Note that in both cases the 95% CI does not overlap with 0.

---
## Funnel plot with trim and fill.

```r
model_hksj.trimfill<-trimfill(model_hksj)
funnel(model_hksj.trimfill,xlab = "Hedges' g")
```

![](Meta-analysis_4_files/figure-html/trim and fill plot-1.svg)

???
Can be further modified.

---
## Issues with trim and fill.

* Positive = Trim and fill method requires no assumptions about the mechanism leading to publication bias, provides an estimate of the number of missing studies, and also provides an estimated effect size   ‘adjusted’ for the publication bias (based on the filled studies).

* Negative = Strong assumption: a symmetric funnel plot. No guarantee that the adjusted intervention effect matches what would have been observed in the absence of publication bias, since we cannot know the true mechanism for publication bias. Equally importantly, the trim and fill method **does not take into account** reasons for funnel plot asymmetry other than publication bias.

* Note: Corrects asymmetry, regardless of whether it is bias! If small N studies used better research designs, they might show larger effects and this does not constitute bias.

* The method is known to perform poorly in the presence of substantial between-study heterogeneity (e.g., [Peters et al., 2007](https://sci-hub.tw/https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.2889)).

* --> Therefore, ‘corrected’ effect estimates should be interpreted with great caution. (More [here](http://datacolada.org/30))

---
## Radial plot: an alternative to a funnel plot.

* Galbraith proposed a different plot.

* Preferred if many studies exist (funnel plot will be cluttered).

* No small-study effects, individual study results should scatter randomly around the regression line through the origin corresponding to the fixed effect estimate.

* Read more about how it is done in `metafor` [here](http://www.metafor-project.org/doku.php/plots:radial_plot)

---
## Radial plot

```r
radial(model_hksj)
```

![](Meta-analysis_4_files/figure-html/radial-1.svg)

???
Results for smaller studies (inverse of standard error less than 2) do not scatter randomly around the regression line, here. Conforming

---
## Begg and Mazumdar test (1994)

* This test is non-parametric and uses the correlation between the ranks of effect sizes and the ranks of their variance (Kendall's `$\tau$`).

* This test has low statistical power (Schwarzer et al., 2015: 116), i.e. you'll need a lot of studies in order to detect publication bias.

```r
metabias(model_hksj,method.bias = "rank")
```

```
## 
## 	Rank correlation test of funnel plot asymmetry
## 
## data:  model_hksj
## z = 3.0681, p-value = 0.002154
## alternative hypothesis: asymmetry in funnel plot
## sample estimates:
##       ks    se.ks 
## 81.00000 26.40076
```

---
## Not covered today: binary outcomes.

Tests discussed above were designed when one has **continuous outcomes**.

The largest problem is that these tests assume symmetry in the funnel plots, which is less likely when dealing with binary outcomes (e.g., odds ratios). Standard tests then tend to be anti-conservative when applied to binary outcome measures. (i.e. suggesting evidence for publication bias when there is limited evidence).

There are numerous tests which adapt tests for binary outcomes, like Egger test. You can read about them in Schwarzer et al. (2015: 120-ff) or find them in the `meta` manual and package.

---
## Not covered today, PET-PEESE / TES / Caliper test, ... .

* Newer method: PET-PEESE popular in Psychology , PET: precision-effect test, PEESE: precision‐effect estimate with standard error. [Stanley and Doucouliagos,2014](https://sci-hub.tw/https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1095). There are lots of debates on how well this method performs (e.g., [Alinaghi & Reed, 2018](https://sci-hub.tw/https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1298)), especially when faced with heterogeneity. No R package currenly implements this, as far as I can tell. There are some helper functions, [here](https://github.com/Joe-Hilgard/PETPEESE) but these seem to build on CMA.

* [TES](https://sci-hub.tw/https://journals.sagepub.com/doi/abs/10.1177/1740774507079441) : Tests of Excess Significance (Ioannidis & Trikalinos, 2007). If interested look [here](https://cran.r-project.org/web/packages/PubBias/index.html) and look at the discussion [here](http://datacolada.org/24) and [this blog post](https://alexanderetz.com/tag/test-of-excess-significance/). Note that there is controversy around this.

* 'Caliper test', within a window _p_-values should fall symmetrically (Gelber & Malhotra, 2008)

* Recent review on correcting for [publication biases](https://journals.sagepub.com/doi/full/10.1177/2515245919847196) (Carter et al., 2019)

---
## Fail-safe N's

I'll tell you about these, even though not optimal, as you might still be asked by reviewers/editors to report them... .

'Older method' suggested that we could estimate how many null studies that needed to be in the file drawer ("file drawer number").

---
## Fail-safe N's (Rosenthal, 1979)

Formal definition: Number of additional ‘negative’ studies (studies in which the effect was 0) that would be needed to increase the _p_-value of the meta-analysis to >0.05.

```r
fsn(TE,se=seTE, data=madata)
```

```
## 
## Fail-safe N Calculation Using the Rosenthal Approach
## 
## Observed Significance Level: <.0001
## Target Significance Level: 0.05
## 
## Fail-safe N: 763
```

???
Example I saw has se= but the manual states sei= rather than se=

---
## Orwin Fail-safe N

Default is dividing effect size.

```r
fsn(TE,se=seTE, data=madata, type='Orwin')
```

```
## 
## Fail-safe N Calculation Using the Orwin Approach
## 
## Average Effect Size: 0.6427
## Target Effect Size:  0.3214
## 
## Fail-safe N: 18
```

```r
fsn(TE,se=seTE, data=madata, type='Orwin', target=.5) # reduce to medium effect.
```

```
## 
## Fail-safe N Calculation Using the Orwin Approach
## 
## Average Effect Size: 0.6427
## Target Effect Size:  0.5000
## 
## Fail-safe N: 6
```

???
Note that Orwin's method produces a larger estimate (?!) of the effect size.

---
## Rosenberg Fail-safe N

Number of studies averaging null results that would have to be added to the given set of observed outcomes to reduce _p_-value of the (weighted) average effect size (based on a fixed-effects model) to a target `$\alpha$` level (e.g., .05).

```r
fsn(TE,se=seTE, data=madata, type='Rosenberg')
```

```
## 
## Fail-safe N Calculation Using the Rosenberg Approach
## 
## Average Effect Size: 0.4805
## Observed Significance Level: <.0001
## Target Significance Level: 0.05
## 
## Fail-safe N: 428
```

???
Note that this is using the estimate.

---
## Issues with fail-safe N's.

* Dependent on what we assume the unpublished studies to be like.

* Widely varying estimates possible dependent on analytical choices (examplel: Orwin: 6 , Rosenthal: 428).

* Focuses on _p_ values rather than effect sizes and its (but see Orwin's Method which focuses on effect size)

Therefore, typically not recommended, but who knows you might still be asked to calculate them... .

---
## P-curves.

**Thomas opens gosoapbox question**

Let's watch a brief [video](https://www.youtube.com/watch?v=V7pvYLZkcK4).

---
## P-curves.

Did authors _p_-hack? If so, did it our affect meta-analysis?

Ideally, provide a **detailed table in which the reported results of each outcome data used in the p-curve is documented.**

---
## P-Curves: Controversial...

[McShane et al. (2016)](http://sci-hub.tw/https://doi.org/10.1177/1745691616662243) and this [blogpost](https://statmodeling.stat.columbia.edu/2018/02/26/p-curve-p-uniform-hedges-1984-methods-meta-analysis-selection-bias-exchange-blake-mcshane-uri-simosohn/) by Andrew Gelman.

* Debate revolves on whether we expect heterogeneity in effect sizes or not. If heterogeneity, then _p_-curve is not a good approach... .

* Reinventing the wheel... . ([Hedges, 1984](http://sci-hub.tw/https://doi.org/10.3102/10769986009001061))

---
## P-Curves: Some recommendations ([Van Aert et al., 2016](https://journals.sagepub.com/doi/10.1177/1745691616650874)).

"First, we recommend that researchers be reluctant to interpret the results of any meta-analytic technique if there are indicators of p-hacking in the primary studies ... because p-hacking may bias the effect-size estimates of meta-analysis in any direction."

"Second, we recommend applying fixed-effect and random-effects meta-analysis and p-uniform or p-curve."

"Next, we recommend checking for direct and indirect evidence of publication bias"

"We recommend setting the estimates of p-uniform and p-curve to 0 if the average p value of statistically significant studies is larger than .025"

**"Interpreting the estimates of p-uniform and p-curve as the average population effect size estimate is discouraged if the effect-size heterogeneity is large `$I^2>.5$`"**

---
## What to do?

* If you have heterogeneity and still want to pursue _p_-curves, screen if heterogeneity is driven by outliers? Perhaps possible to do _p_-curve analysis after removal.

* **Subgroup analysis**: Next session. Suppose that heterogeneity is explained due to subgroups could run analyses on each.

---
## Data preparation...

We need data in this type of format. Means and SD and N. lower case _e_: experimental, lower case _c_ : control

<table>
 <thead>
 <tr>
 <th style="text-align:left;"> Author </th>
 <th style="text-align:left;"> Ne </th>
 <th style="text-align:left;"> Me </th>
 <th style="text-align:left;"> Se </th>
 <th style="text-align:right;"> Nc </th>
 <th style="text-align:left;"> Mc </th>
 <th style="text-align:left;"> Sc </th>
 </tr>
 </thead>
<tbody>
 <tr>
 <td style="text-align:left;"> Cavanagh </td>
 <td style="text-align:left;"> 50 </td>
 <td style="text-align:left;"> 4.5 </td>
 <td style="text-align:left;"> 2.7 </td>
 <td style="text-align:right;"> 50 </td>
 <td style="text-align:left;"> 5.6 </td>
 <td style="text-align:left;"> 2.6 </td>
 </tr>
 <tr>
 <td style="text-align:left;"> Day </td>
 <td style="text-align:left;"> 64 </td>
 <td style="text-align:left;"> 18.3 </td>
 <td style="text-align:left;"> 6.4 </td>
 <td style="text-align:right;"> 65 </td>
 <td style="text-align:left;"> 20.2 </td>
 <td style="text-align:left;"> 7.6 </td>
 </tr>
 <tr>
 <td style="text-align:left;"> Frazier </td>
 <td style="text-align:left;"> 90 </td>
 <td style="text-align:left;"> 12.5 </td>
 <td style="text-align:left;"> 3.2 </td>
 <td style="text-align:right;"> 95 </td>
 <td style="text-align:left;"> 15.5 </td>
 <td style="text-align:left;"> 4.4 </td>
 </tr>
 <tr>
 <td style="text-align:left;"> Gaffney </td>
 <td style="text-align:left;"> 30 </td>
 <td style="text-align:left;"> 2.34 </td>
 <td style="text-align:left;"> 0.87 </td>
 <td style="text-align:right;"> 30 </td>
 <td style="text-align:left;"> 3.13 </td>
 <td style="text-align:left;"> 1.234 </td>
 </tr>
 <tr>
 <td style="text-align:left;"> Greer </td>
 <td style="text-align:left;"> 77 </td>
 <td style="text-align:left;"> 15.212 </td>
 <td style="text-align:left;"> 5.35 </td>
 <td style="text-align:right;"> 69 </td>
 <td style="text-align:left;"> 20.13 </td>
 <td style="text-align:left;"> 7.43 </td>
 </tr>
 <tr>
 <td style="text-align:left;"> Harrer </td>
 <td style="text-align:left;"> 60 </td>
 <td style="text-align:left;"> 3.153 </td>
 <td style="text-align:left;"> 1.256 </td>
 <td style="text-align:right;"> 60 </td>
 <td style="text-align:left;"> 3.4213 </td>
 <td style="text-align:left;"> 1.878 </td>
 </tr>
</tbody>
</table>

---
## Function to prepare data when in this format (Harrer et al. 2019)

```r
pcurve_dataprep<-function(data){
data<-data
Me<-as.numeric(data$Me)
Se<-as.numeric(data$Se)
Ne<-as.numeric(data$Ne)
Mc<-as.numeric(data$Mc)
Sc<-as.numeric(data$Sc)
Nc<-as.numeric(data$Nc)
esc<-esc_mean_sd(grp1m=Me, 
 grp1sd=Se, 
 grp1n=Ne, 
 grp2m=Mc, 
 grp2sd=Sc, 
 grp2n=Nc, 
 es.type = "d")
output<-des(d=esc$es,n.1=Ne,n.2=Nc, verbose = FALSE) # calculate d
output$r<-abs(output$r) # Abs of r
tot<-data.frame(paste("r(",output$N.total-2,")=",output$r)) # df and r
colnames(tot)<-c("output")
tot$output<-gsub(" ", "", tot$output, fixed = TRUE)
totoutput<-as.character(tot$output)
print(tot, row.names = FALSE)
write(totoutput,ncolumns=1, file="input.txt") # format
}
```

???
Unreadable so show function.

---
## Inspect Function: code overflow

Inspect function...

**Thomas clicks function**

Incidentally, recently put [into a package](https://dmetar.protectlab.org/)

???
needs > R 3.5.2
---
## Apply function.

```r
pcurve_dataprep(data=metacont)
```

```
##       output
##    r(98)=0.2
##  r(127)=0.13
##  r(183)=0.36
##   r(58)=0.35
##  r(144)=0.36
##  r(118)=0.08
```

---
## Source the P-curve app function

We take a function from [here](http://p-curve.com/app4/pcurve_app4.06.r), put it in a new file and make it available to R. Make sure it lives where your .rmd lives,... .

```r
source("pcurve_app4.06.r")
```

Will need to find location where our input file lives!

```r
getwd()
```

```
## [1] "/Users/tvpollet/Dropbox/teaching/Meta_analysis_course_Brazil/Meta-analysis_4"
```

---
## P Curve

Will store lots of files in your working directory... .

```r
pcurve_app("input.txt", "~/Dropbox/teaching/Meta_analysis_course_Brazil/Meta-analysis_4" )
```

```
## quartz_off_screen 
##                 2
```

???
Note I use tilde to truncate as office PC will have different username. Note Windows might have different annotation.

---
## Figure 1: "input.png"

???
The bottom, you can also find the number of effect sizes with `$p<0.05$` which were included in the analysis. There are two tests displayed in the plot, right skewness/flatness.

---
## Interpretation of two tests.

Evidential value: One can statistically rule out selective reporting was solely responsible for the statistically significant results that have been observed. (Simmons & Simonsohn, 2017). (i.e. there is evidence for an effect)

**The test for right-skewness**

If there is evidential value behind our data, the p-curve should be **right-skewed**. Through eyeballing, we see that this is pretty much the case here, and the **tests for the half and full p-curve** are also both **significant** ( `$p_{Full}<0.001, p_{Half}<0.001$` ). This means that the p-curve is heavily right-skewed, indicating that **evidential value is present in our data**

**The test for flatness**

If there is evidential value behind our data, the p-curve should also **not be flat**. Through eyeballing, we see that this is pretty much the case here. The **tests for flatness** are both not significant ( `$p_{Full}=0.9887, p_{Binomial}=0.7459$` ).

???
half uses p<.025

---
## Figure 2: "input_fit.png"

???
This plot estimates the **power** behind our data: do we have sufficient studies with sufficient participants to find a true effect if it exists? A conventional threshold for optimal power is **80%**, but P-curve can even assess evidential value if studies are **underpowered**. In our case, the the power estimate is **90%**. It looks relatively smooth... sort of like a V?

---
## Figure 3: "input_cumulative.png"

Robustness check: What happens when we drop the highest and the lowest _p_ value?

???
Here we find that it is robust. Red line is .05.

---
## Estimate effect when 'correcting' for p-hacking.

Similar to trim and fill procedure which we covered.

Estimate what the effect size would be if we adjust for publication bias.

---
## Customised function

Code from [https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/true_effect_estimation.R](https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/true_effect_estimation.R)

Copy and paste to 'console' or make a file and host in your working folder.

```r
source("true_effect_estimation.r")
```

???
Read through functions. Basically takes _t_-values and then applies p curve to it.

---
##  Plot

The function provides an estimate of `$d=0.642$`.

```r
plotloss(data=metacont,dmin=0,dmax=1)
```

```
## [1] 0.6418174
```

---
## A variant of the P-curve: P uniform(\*)

Original method described in [here](https://www.researchgate.net/profile/Jelte_Wicherts/publication/268985081_Meta-Analysis_Using_Effect_Size_Distributions_of_Only_Statistically_Significant_Studies/links/548024e00cf2ccc7f8bb0ca6.pdf). Recently improved upon: P uniform\* , see [here](https://osf.io/preprints/metaarxiv/zqjr9/).

Very basic example using 'meta-analysis attraction data' from our [preprint](https://psyarxiv.com/28yvp/) on whether attractive rivals are more likely to elicit jealousy in women than in men.

```r
require(readxl)
require(puniform)
ma_attract<-read_xlsx('meta-analysis_attract-N_marelich.xlsx') 
sink("punistar.txt")
puni_star(ri = ma_attract$r, ni=ma_attract$N, side='right')
```

```
## 
## Method: ML (k = 22; ksig = 16)
## 
## Estimating effect size p-uniform*
## 
## est ci.lb ci.ub L.0 pval
## 0.1954 0.0763 0.3057 4.8575 0.0275
## 
## ===
## 
## Estimating between-study variance p-uniform*
## 
## tau2 tau2.lb tau2.ub L.het pval
## 0.0261 0.0131 0.056 100.0676 <.001
## 
## ===
## 
## Publication bias test p-uniform*
## 
## L.pb pval
## 0.2123 0.8993
```

```r
sink()
```

???
Read .2 .txt as it prints blanks... . **Original**: r= 0.22, 95%CI [0.15,0.3] , this:  r=.1954 [.08,.31]. Also no evidence for publication bias.

---
## Remember: critiques exist.

[Richard Morey on p-curve](http://richarddmorey.org/content/Psynom17/pcurve/#/):

1) Incorrect test

2) justification of grouping (remember: heterogeneity!)

---
## Exercise.

Please see the exercise posted [here](https://tvpollet.github.io/Meta-analysis_4/Exercise_4_questions.html)

---
## Any Questions?

[http://tvpollet.github.io](http://tvpollet.github.io)

Twitter: @tvpollet

---
## Acknowledgments

* Numerous students and colleagues. Any mistakes are my own.

* My colleagues who helped me with regards to meta-analysis Nexhmedin Morina, Stijn Peperkoorn, Gert Stulp, Mirre Simons, Johannes Honekopp.

* [HBES](www.hbes.com) and [LECH](https://www.lechufrn.com/) for funding this workshop. Those who have funded me (not these studies per se): [NWO](www.nwo.nl), [Templeton](www.templeton.org), [NIAS](http://nias.knaw.nl).

* You for listening!

---
## References and further reading (errors = blame RefManageR)

<cite>Aert, R. C. M. van, J. M. Wicherts, and M. A. L. M. van
Assen
(2016).
&ldquo;Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p-Uniform and p-Curve&rdquo;.
In: Perspectives on Psychological Science 11.5, pp. 713-729.
DOI: <a href="https://doi.org/10.1177/1745691616650874">10.1177/1745691616650874</a>.
eprint: https://doi.org/10.1177/1745691616650874.</cite>

<cite>Aloe, A. M. and C. G. Thompson
(2013).
&ldquo;The Synthesis of Partial Effect Sizes&rdquo;.
In: Journal of the Society for Social Work and Research 4.4, pp. 390-405.
DOI: <a href="https://doi.org/10.5243/jsswr.2013.24">10.5243/jsswr.2013.24</a>.
eprint: https://doi.org/10.5243/jsswr.2013.24.</cite>

<cite>Assink, M. and C. J. Wibbelink
(2016).
&ldquo;Fitting Three-Level Meta-Analytic Models in R: A Step-by-Step Tutorial&rdquo;.
In: The Quantitative Methods for Psychology 12.3, pp. 154-174.
ISSN: 2292-1354.</cite>

<cite>Barendregt, J. J, S. A. Doi, Y. Y. Lee, et al.
(2013).
&ldquo;Meta-Analysis of Prevalence&rdquo;.
In: Journal of Epidemiology and Community Health 67.11, pp. 974-978.
ISSN: 0143-005X.
DOI: <a href="https://doi.org/10.1136/jech-2013-203104">10.1136/jech-2013-203104</a>.</cite>

<cite>Becker, B. J. and M. Wu
(2007).
&ldquo;The Synthesis of Regression Slopes in Meta-Analysis&rdquo;.
In: Statistical science 22.3, pp. 414-429.
ISSN: 0883-4237.</cite>
---
## More refs 1.

<cite>Borenstein, M, L. V. Hedges, J. P. Higgins, et al.
(2009).
Introduction to Meta-Analysis.
John Wiley &amp; Sons.
ISBN: 1-119-96437-7.</cite>

<cite>Burnham, K. P. and D. R. Anderson
(2002).
Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach.
New York, NY: Springer.
ISBN: 0-387-95364-7.</cite>

<cite>Burnham, K. P. and D. R. Anderson
(2004).
&ldquo;Multimodel Inference: Understanding AIC and BIC in Model Selection&rdquo;.
In: Sociological Methods &amp; Research 33.2, pp. 261-304.
ISSN: 0049-1241.
DOI: <a href="https://doi.org/10.1177/0049124104268644">10.1177/0049124104268644</a>.</cite>

<cite>Carter, E. C, F. D. Schönbrodt, W. M. Gervais, et al.
(2019).
&ldquo;Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods&rdquo;.
In: Advances in Methods and Practices in Psychological Science 2.2, pp. 115-144.
DOI: <a href="https://doi.org/10.1177/2515245919847196">10.1177/2515245919847196</a>.</cite>

<cite>Chen, D. D. and K. E. Peace
(2013).
Applied Meta-Analysis with R.
Chapman and Hall/CRC.
ISBN: 1-4665-0600-8.</cite>

---
## More refs 2.

<cite>Cheung, M. W.
(2015a).
&ldquo;metaSEM: An R Package for Meta-Analysis Using Structural Equation Modeling&rdquo;.
In: Frontiers in Psychology 5, p. 1521.
ISSN: 1664-1078.
DOI: <a href="https://doi.org/10.3389/fpsyg.2014.01521">10.3389/fpsyg.2014.01521</a>.</cite>

<cite>Cheung, M. W.
(2015b).
Meta-Analysis: A Structural Equation Modeling Approach.
New York, NY: John Wiley &amp; Sons.
ISBN: 1-119-99343-1.</cite>

<cite>Cooper, H.
(2010).
Research Synthesis and Meta-Analysis: A Step-by-Step Approach.
4th.
Sage publications.
ISBN: 1-4833-4704-4.</cite>

<cite>Cooper, H, L. V. Hedges, and J. C. Valentine
(2009).
The Handbook of Research Synthesis and Meta-Analysis.
New York: Russell Sage Foundation.
ISBN: 1-61044-138-9.</cite>

<cite>Cooper, H. and E. A. Patall
(2009).
&ldquo;The Relative Benefits of Meta-Analysis Conducted with Individual Participant Data versus Aggregated Data.&rdquo;
In: Psychological Methods 14.2, pp. 165-176.
ISSN: 1433806886.
DOI: <a href="https://doi.org/10.1037/a0015565">10.1037/a0015565</a>.</cite>

---
## More refs 3.

<cite>Crawley, M. J.
(2013).
The R Book: Second Edition.
New York, NY: John Wiley &amp; Sons.
ISBN: 1-118-44896-0.</cite>

<cite>Cumming, G.
(2014).
&ldquo;The New Statistics&rdquo;.
In: Psychological Science 25.1, pp. 7-29.
ISSN: 0956-7976.
DOI: <a href="https://doi.org/10.1177/0956797613504966">10.1177/0956797613504966</a>.</cite>

<cite>Dickersin, K.
(2005).
&ldquo;Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm&rdquo;.
In: 
Publication Bias in Meta-Analysis Prevention, Assessment and Adjustments.
Ed. by H. R. Rothstein, A. J. Sutton and M. Borenstein.
Chichester, UK: John Wiley.</cite>

<cite>Fisher, R. A.
(1946).
Statistical Methods for Research Workers.
10th ed.
Edinburgh, UK: Oliver and Boyd.</cite>

<cite>Flore, P. C. and J. M. Wicherts
(2015).
&ldquo;Does Stereotype Threat Influence Performance of Girls in Stereotyped Domains? A Meta-Analysis&rdquo;.
In: Journal of School Psychology 53.1, pp. 25-44.
ISSN: 0022-4405.
DOI: <a href="https://doi.org/10.1016/j.jsp.2014.10.002">10.1016/j.jsp.2014.10.002</a>.</cite>

---
## More refs 4.

<cite>Galbraith, R. F.
(1994).
&ldquo;Some Applications of Radial Plots&rdquo;.
In: Journal of the American Statistical Association 89.428, pp. 1232-1242.
ISSN: 0162-1459.
DOI: <a href="https://doi.org/10.1080/01621459.1994.10476864">10.1080/01621459.1994.10476864</a>.</cite>

<cite>Glass, G. V.
(1976).
&ldquo;Primary, Secondary, and Meta-Analysis of Research&rdquo;.
In: Educational researcher 5.10, pp. 3-8.
ISSN: 0013-189X.
DOI: <a href="https://doi.org/10.3102/0013189X005010003">10.3102/0013189X005010003</a>.</cite>

<cite>Goh, J. X, J. A. Hall, and R. Rosenthal
(2016).
&ldquo;Mini Meta-Analysis of Your Own Studies: Some Arguments on Why and a Primer on How&rdquo;.
In: Social and Personality Psychology Compass 10.10, pp. 535-549.
ISSN: 1751-9004.
DOI: <a href="https://doi.org/10.1111/spc3.12267">10.1111/spc3.12267</a>.</cite>

<cite>Harrell, F. E.
(2015).
Regression Modeling Strategies.
2nd.
Springer Series in Statistics.
New York, NY: Springer New York.
ISBN: 978-1-4419-2918-1.
DOI: <a href="https://doi.org/10.1007/978-1-4757-3462-1">10.1007/978-1-4757-3462-1</a>.</cite>

<cite>Harrer, M., P. Cuijpers, and D. D. Ebert
(2019).
Doing Meta-Analysis in R: A Hands-on Guide.
https://bookdown.org/MathiasHarrer/Doing\_ Meta\_ Analysis\_ in\_ R/.</cite>

---
## More refs 5.

<cite>Hartung, J. and G. Knapp
(2001).
&ldquo;On Tests of the Overall Treatment Effect in Meta-Analysis with Normally Distributed Responses&rdquo;.
In: Statistics in Medicine 20.12, pp. 1771-1782.
DOI: <a href="https://doi.org/10.1002/sim.791">10.1002/sim.791</a>.</cite>

<cite>Hayes, A. F. and K. Krippendorff
(2007).
&ldquo;Answering the Call for a Standard Reliability Measure for Coding Data&rdquo;.
In: Communication Methods and Measures 1.1, pp. 77-89.
ISSN: 1931-2458.
DOI: <a href="https://doi.org/10.1080/19312450709336664">10.1080/19312450709336664</a>.</cite>

<cite>Hedges, L. V.
(1981).
&ldquo;Distribution Theory for Glass's Estimator of Effect Size and Related Estimators&rdquo;.
In: Journal of Educational Statistics 6.2, pp. 107-128.
DOI: <a href="https://doi.org/10.3102/10769986006002107">10.3102/10769986006002107</a>.</cite>

<cite>Hedges, L. V.
(1984).
&ldquo;Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences&rdquo;.
In: Journal of Educational Statistics 9.1, pp. 61-85.
ISSN: 0362-9791.
DOI: <a href="https://doi.org/10.3102/10769986009001061">10.3102/10769986009001061</a>.</cite>

<cite>Hedges, L. V. and I. Olkin
(1980).
&ldquo;Vote-Counting Methods in Research Synthesis.&rdquo;
In: Psychological bulletin 88.2, pp. 359-369.
ISSN: 1939-1455.
DOI: <a href="https://doi.org/10.1037/0033-2909.88.2.359">10.1037/0033-2909.88.2.359</a>.</cite>

---
## More refs 6.

<cite>Higgins, J. P. T. and S. G. Thompson
(2002).
&ldquo;Quantifying Heterogeneity in a Meta-Analysis&rdquo;.
In: Statistics in Medicine 21.11, pp. 1539-1558.
DOI: <a href="https://doi.org/10.1002/sim.1186">10.1002/sim.1186</a>.</cite>

<cite>Higgins, J. P. T, S. G. Thompson, J. J. Deeks, et al.
(2003).
&ldquo;Measuring Inconsistency in Meta-Analyses&rdquo;.
In: BMJ 327.7414, pp. 557-560.
ISSN: 0959-8138.
DOI: <a href="https://doi.org/10.1136/bmj.327.7414.557">10.1136/bmj.327.7414.557</a>.</cite>

<cite>Higgins, J, S. Thompson, J. Deeks, et al.
(2002).
&ldquo;Statistical Heterogeneity in Systematic Reviews of Clinical Trials: A Critical Appraisal of Guidelines and Practice&rdquo;.
In: Journal of Health Services Research &amp; Policy 7.1, pp. 51-61.
DOI: <a href="https://doi.org/10.1258/1355819021927674">10.1258/1355819021927674</a>.</cite>

<cite>Hirschenhauser, K. and R. F. Oliveira
(2006).
&ldquo;Social Modulation of Androgens in Male Vertebrates: Meta-Analyses of the Challenge Hypothesis&rdquo;.
In: Animal Behaviour 71.2, pp. 265-277.
ISSN: 0003-3472.
DOI: <a href="https://doi.org/10.1016/j.anbehav.2005.04.014">10.1016/j.anbehav.2005.04.014</a>.</cite>

<cite>Ioannidis, J. P.
(2008).
&ldquo;Why Most Discovered True Associations Are Inflated&rdquo;.
In: Epidemiology 19.5, pp. 640-648.
ISSN: 1044-3983.</cite>
---
## More refs 7.

<cite>Jackson, D, M. Law, G. Rücker, et al.
(2017).
&ldquo;The Hartung-Knapp Modification for Random-Effects Meta-Analysis: A Useful Refinement but Are There Any Residual Concerns?&rdquo;
In: Statistics in Medicine 36.25, pp. 3923-3934.
DOI: <a href="https://doi.org/10.1002/sim.7411">10.1002/sim.7411</a>.
eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.7411.</cite>

<cite>Jacobs, P. and W. Viechtbauer
(2016).
&ldquo;Estimation of the Biserial Correlation and Its Sampling Variance for Use in Meta-Analysis&rdquo;.
In: Research Synthesis Methods 8.2, pp. 161-180.
DOI: <a href="https://doi.org/10.1002/jrsm.1218">10.1002/jrsm.1218</a>.</cite>

<cite>Koricheva, J, J. Gurevitch, and K. Mengersen
(2013).
Handbook of Meta-Analysis in Ecology and Evolution.
Princeton, NJ: Princeton University Press.
ISBN: 0-691-13729-3.</cite>

<cite>Kovalchik, S.
(2013).
Tutorial On Meta-Analysis In R - R useR! Conference 2013.</cite>

<cite>Lipsey, M. W. and D. B. Wilson
(2001).
Practical Meta-Analysis.
London: SAGE publications, Inc.
ISBN: 0-7619-2167-2.</cite>

---
## More refs 8.

<cite>Littell, J. H, J. Corcoran, and V. Pillai
(2008).
Systematic Reviews and Meta-Analysis.
Oxford, UK: Oxford University Press.
ISBN: 0-19-532654-7.</cite>

<cite>McShane, B. B, U. Böckenholt, and K. T. Hansen
(2016).
&ldquo;Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes&rdquo;.
In: Perspectives on Psychological Science 11.5, pp. 730-749.
DOI: <a href="https://doi.org/10.1177/1745691616662243">10.1177/1745691616662243</a>.
eprint: https://doi.org/10.1177/1745691616662243.</cite>

<cite>Mengersen, K, C. Schmidt, M. Jennions, et al.
(2013).
&ldquo;Statistical Models and Approaches to Inference&rdquo;.
In: 
Handbook of Meta-Analysis in Ecology and Evolution.
Ed. by Koricheva, J, J. Gurevitch and Mengersen, Kerrie.
Princeton, NJ: Princeton University Press, pp. 89-107.</cite>

<cite>Methley, A. M, S. Campbell, C. Chew-Graham, et al.
(2014).
&ldquo;PICO, PICOS and SPIDER: A Comparison Study of Specificity and Sensitivity in Three Search Tools for Qualitative Systematic Reviews&rdquo;.
Eng.
In: BMC health services research 14, pp. 579-579.
ISSN: 1472-6963.
DOI: <a href="https://doi.org/10.1186/s12913-014-0579-0">10.1186/s12913-014-0579-0</a>.</cite>

<cite>Morina, N, K. Stam, T. V. Pollet, et al.
(2018).
&ldquo;Prevalence of Depression and Posttraumatic Stress Disorder in Adult Civilian Survivors of War Who Stay in War-Afflicted Regions. A Systematic Review and Meta-Analysis of Epidemiological Studies&rdquo;.
In: Journal of Affective Disorders 239, pp. 328-338.
ISSN: 0165-0327.
DOI: <a href="https://doi.org/10.1016/j.jad.2018.07.027">10.1016/j.jad.2018.07.027</a>.</cite>

---
## More refs 9.

<cite>Nakagawa, S, D. W. A. Noble, A. M. Senior, et al.
(2017).
&ldquo;Meta-Evaluation of Meta-Analysis: Ten Appraisal Questions for Biologists&rdquo;.
In: BMC Biology 15.1, p. 18.
ISSN: 1741-7007.
DOI: <a href="https://doi.org/10.1186/s12915-017-0357-7">10.1186/s12915-017-0357-7</a>.</cite>

<cite>Pastor, D. A. and R. A. Lazowski
(2018).
&ldquo;On the Multilevel Nature of Meta-Analysis: A Tutorial, Comparison of Software Programs, and Discussion of Analytic Choices&rdquo;.
In: Multivariate Behavioral Research 53.1, pp. 74-89.
DOI: <a href="https://doi.org/10.1080/00273171.2017.1365684">10.1080/00273171.2017.1365684</a>.</cite>

<cite>Poole, C. and S. Greenland
(1999).
&ldquo;Random-Effects Meta-Analyses Are Not Always Conservative&rdquo;.
In: American Journal of Epidemiology 150.5, pp. 469-475.
ISSN: 0002-9262.
DOI: <a href="https://doi.org/10.1093/oxfordjournals.aje.a010035">10.1093/oxfordjournals.aje.a010035</a>.
eprint: http://oup.prod.sis.lan/aje/article-pdf/150/5/469/286690/150-5-469.pdf.</cite>

<cite>Popper, K.
(1959).
The Logic of Scientific Discovery.
London, UK: Hutchinson.
ISBN: 1-134-47002-9.</cite>

<cite>Roberts, P. D, G. B. Stewart, and A. S. Pullin
(2006).
&ldquo;Are Review Articles a Reliable Source of Evidence to Support Conservation and Environmental Management? A Comparison with Medicine&rdquo;.
In: Biological conservation 132.4, pp. 409-423.
ISSN: 0006-3207.</cite>

---
## More refs 10.

<cite>Rosenberg, M. S, H. R. Rothstein, and J. Gurevitch
(2013).
&ldquo;Effect Sizes: Conventional Choices and Calculations&rdquo;.
In: Handbook of Meta-analysis in Ecology and Evolution, pp. 61-71.</cite>

<cite>Röver, C, G. Knapp, and T. Friede
(2015).
&ldquo;Hartung-Knapp-Sidik-Jonkman Approach and Its Modification for Random-Effects Meta-Analysis with Few Studies&rdquo;.
In: BMC Medical Research Methodology 15.1, p. 99.
ISSN: 1471-2288.
DOI: <a href="https://doi.org/10.1186/s12874-015-0091-1">10.1186/s12874-015-0091-1</a>.</cite>

<cite>Schwarzer, G, J. R. Carpenter, and G. Rücker
(2015).
Meta-Analysis with R.
New York, NY: Springer.
ISBN: 3-319-21415-2.</cite>

<cite>Schwarzer, G, H. Chemaitelly, L. J. Abu-Raddad, et al.
&ldquo;Seriously Misleading Results Using Inverse of Freeman-Tukey Double Arcsine Transformation in Meta-Analysis of Single Proportions&rdquo;.
In: Research Synthesis Methods 0.0.
DOI: <a href="https://doi.org/10.1002/jrsm.1348">10.1002/jrsm.1348</a>.
eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/jrsm.1348.</cite>

<cite>Simmons, J. P, L. D. Nelson, and U. Simonsohn
(2011).
&ldquo;False-Positive Psychology&rdquo;.
In: Psychological Science 22.11, pp. 1359-1366.
ISSN: 0956-7976.
DOI: <a href="https://doi.org/10.1177/0956797611417632">10.1177/0956797611417632</a>.</cite>

---
## More refs 11.

<cite>Simonsohn, U, L. D. Nelson, and J. P. Simmons
(2014).
&ldquo;P-Curve: A Key to the File-Drawer.&rdquo;
In: Journal of Experimental Psychology: General 143.2, pp. 534-547.
ISSN: 1939-2222.
DOI: <a href="https://doi.org/10.1037/a0033242">10.1037/a0033242</a>.</cite>

<cite>Sterne, J. A. C, A. J. Sutton, J. P. A. Ioannidis, et al.
(2011).
&ldquo;Recommendations for Examining and Interpreting Funnel Plot Asymmetry in Meta-Analyses of Randomised Controlled Trials&rdquo;.
In: BMJ 343.jul22 1, pp. d4002-d4002.
ISSN: 0959-8138.
DOI: <a href="https://doi.org/10.1136/bmj.d4002">10.1136/bmj.d4002</a>.</cite>

<cite>Veroniki, A. A, D. Jackson, W. Viechtbauer, et al.
(2016).
&ldquo;Methods to Estimate the Between-Study Variance and Its Uncertainty in Meta-Analysis&rdquo;.
Eng.
In: Research synthesis methods 7.1, pp. 55-79.
ISSN: 1759-2887.
DOI: <a href="https://doi.org/10.1002/jrsm.1164">10.1002/jrsm.1164</a>.</cite>

<cite>Viechtbauer, W.
(2015).
&ldquo;Package &lsquo;metafor&rsquo;: Meta-Analysis Package for R&rdquo;.
</cite>

<cite>Weiss, B. and J. Daikeler
(2017).
Syllabus for Course: &ldquo;Meta-Analysis in Survey Methodology&quot;, 6th Summer Workshop (GESIS).</cite>

---
## More refs 12.

<cite>Wickham, H. and G. Grolemund
(2016).
R for Data Science.
Sebastopol, CA: O'Reilly..</cite>

<cite>Wiernik, B.
(2015).
A Brief Introduction to Meta-Analysis.</cite>

<cite>Wiksten, A, G. Rücker, and G. Schwarzer
(2016).
&ldquo;Hartung-Knapp Method Is Not Always Conservative Compared with Fixed-Effect Meta-Analysis&rdquo;.
In: Statistics in Medicine 35.15, pp. 2503-2515.
DOI: <a href="https://doi.org/10.1002/sim.6879">10.1002/sim.6879</a>.</cite>

<cite>Wingfield, J. C, R. E. Hegner, A. M. Dufty Jr, et al.
(1990).
&ldquo;The&quot; Challenge Hypothesis&quot;: Theoretical Implications for Patterns of Testosterone Secretion, Mating Systems, and Breeding Strategies&rdquo;.
In: American Naturalist 136, pp. 829-846.
ISSN: 0003-0147.</cite>

<cite>Yeaton, W. H. and P. M. Wortman
(1993).
&ldquo;On the Reliability of Meta-Analytic Reviews: The Role of Intercoder Agreement&rdquo;.
In: Evaluation Review 17.3, pp. 292-309.
ISSN: 0193-841X.
DOI: <a href="https://doi.org/10.1177/0193841X9301700303">10.1177/0193841X9301700303</a>.</cite>

---
## More refs 13.

---
## More refs 14.