Conservativeness of the Simes inequality under positive dependence

P. Neuvial

2022-11-11

Source: vignettes/Simes_equi-correlation.Rmd

The Simes inequality

Let \(\mathbf{q}=(q_i)_{1 \leq i \leq m}\) be a vector of random variables such that:

  • for each \(i=1 \ldots m\), \(q_i \sim \mathcal{U}[0,1]\)
  • \(\mathbf{q}\) satisfies positive regression dependency on a subset (PRDS)

We refer to Sarkar (1998) for a formal definition of this form of positive dependence. Here, we simply note that it holds in particular of Gaussian equi-correlated random variables.

Then, letting \((q_{(1)}, \ldots q_{(m)} )\) be the corresponding order statistics, we have:

\[ \mathbb{P} \left( \exists i \in \{1, \ldots m\} \::\: q_{(i)} \leq \frac{\alpha i}{m}\right) \leq \alpha \] This inequality is due to Simes (1986). It is sharp under independence of the \(q_i\) (that is, the above inequality is an equality). How sharp is it under positive dependence?

Estimating the size of the Simes test

We write a small function to estimate the size of the Simes test, that is, the level actually achieved by the left-hand side in the above inequality.

size <- function(rho, m, B, alpha=0.05) {
    X0 <- gaussianTestStatistics(m, B, dep = "equi", param = rho)$X0
    p0 <- pnorm(X0, lower.tail = FALSE)
    thr <- t_linear(alpha, 1:m, m)
    p0s <- sanssouci:::colSort(p0);
    isAbove <- sweep(p0s, 1, thr,  "<")
    nAbove <- colSums(isAbove)
    mean(nAbove > 0)
}

Our simulation parameters are set as follows:

m <- 1e3
rhos <- c(0, 0.1, 0.2, 0.4, 0.8)
alpha <- 0.2
B <- 1000

That is:

  • \(1000\) hypotheses
  • equi-correlation level between \(0\) and \(0.8\)
  • target level of the Simes test: \(0.2\)
  • \(1000\) replications are used to estimate the size of the test
ahat <- sapply(rhos, size, m, B, alpha=alpha)
ses <- sqrt(ahat*(1-ahat)/B)

We estimate the size of the test as the average size \(\hat{\alpha}\) achieved across replications, and the corresponding standard error as \(\sqrt{\hat{\alpha}(1-\hat{\alpha})/B}\).

tb <- rbind(ahat/alpha, ses/alpha)
colnames(tb) <- rhos
rownames(tb) <- c("Achieved level/target level", "Standard error")
knitr::kable(tb, digits = 2)
0 0.1 0.2 0.4 0.8
Achieved level/target level 1.02 0.86 0.86 0.44 0.32
Standard error 0.06 0.06 0.06 0.04 0.04

This table illustrates the sharpness of the Simes test under independence (\(\rho=0\)), and the conservativeness of this test under positive dependence (\(\rho>0\)). For example, when \(\rho=0.4\), the size of the Simes test is less than half the target level.

This conservativeness is one of the motivations of the development of post hoc inference methods that use randomization to adapt to dependence. See the following vignettes:

Session information 

## R version 4.2.2 (2022-10-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.5 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitr_1.40         matrixStats_0.62.0 sanssouci_0.12.8  
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.9          magrittr_2.0.3      lattice_0.20-45    
##  [4] R6_2.5.1            ragg_1.2.4          rlang_1.0.6        
##  [7] fastmap_1.1.0       highr_0.9           stringr_1.4.1      
## [10] tools_4.2.2         grid_4.2.2          xfun_0.34          
## [13] cli_3.4.1           jquerylib_0.1.4     matrixTests_0.1.9.1
## [16] htmltools_0.5.3     systemfonts_1.0.4   yaml_2.3.6         
## [19] digest_0.6.30       rprojroot_2.0.3     pkgdown_2.0.6      
## [22] textshaping_0.3.6   Matrix_1.5-3        purrr_0.3.5        
## [25] sass_0.4.2          fs_1.5.2            memoise_2.0.1      
## [28] cachem_1.0.6        evaluate_0.18       rmarkdown_2.18     
## [31] stringi_1.7.8       compiler_4.2.2      bslib_0.4.1        
## [34] generics_0.1.3      desc_1.4.2          jsonlite_1.8.3

References

Sarkar, Sanat K. 1998. “Some Probability Inequalities for Ordered Mtp2 Random Variables: A Proof of the Simes Conjecture.” Annals of Statistics, 494–504.

Simes, R J. 1986. “An improved Bonferroni procedure for multiple tests of significance.” Biometrika 73 (3): 751–54.