MultipleTesting

The MultipleTesting package offers common algorithms for p-value adjustment and combination as well as the estimation of the proportion π₀ of true null hypotheses.

adjust(pvalues, <:PValueAdjustmentMethod)

The adjustment can also be performed on the k smallest out of n p-values:

adjust(pvalues, n, <:PValueAdjustmentMethod)

adjust(pvalues, Bonferroni())

Bonferroni, C.E. (1936). Teoria statistica delle classi e calcolo delle probabilita (Libreria internazionale Seeber).

adjust(pvalues, BenjaminiHochberg())

Adaptive Benjamini-Hochberg with known π₀ or π₀ estimation method.

adjust(pvalues, BenjaminiHochbergAdaptive(π₀))
adjust(pvalues, BenjaminiHochbergAdaptive(<:PValueAdjustmentMethod))

Benjamini, Y., and Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological) 57, 289–300.

Benjamini, Y., Krieger, A. M. & Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93, 491–507.

adjust(pvalues, BenjaminiYekutieli())

Benjamini, Y., and Yekutieli, D. (2001). The Control of the False Discovery Rate in Multiple Testing under Dependency. The Annals of Statistics 29, 1165–1188.

adjust(pvalues, BenjaminiLiu())

Benjamini, Y., and Liu, W. (1999). A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. Journal of Statistical Planning and Inference 82, 163–170.

adjust(pvalues, Hochberg())

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–802.

adjust(pvalues, Holm())

Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6, 65–70.

adjust(pvalues, Hommel())

Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75, 383–386.

adjust(pvalues, Sidak())

Šidák, Z. (1967). Rectangular Confidence Regions for the Means of Multivariate Normal Distributions. Journal of the American Statistical Association 62, 626–633.

adjust(pvalues, ForwardStop())

G’Sell, M.G., Wager, S., Chouldechova, A., and Tibshirani, R. (2016). Sequential selection procedures and false discovery rate control. J. R. Stat. Soc. B 78, 423–444.

adjust(pvalues, BarberCandes())

Barber, R.F., and Candès, E.J. (2015). Controlling the false discovery rate via knockoffs. Ann. Statist. 43, 2055–2085.

Arias-Castro, E., and Chen, S. (2017). Distribution-free multiple testing. Electron. J. Statist. 11, 1983–2001.

estimate(pvalues, <:Pi0Estimator)

estimate(pvalues, Storey())

Storey, J.D., Taylor, J.E., and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 66, 187–205.

estimate(pvalues, StoreyBootstrap())

Robinson, D. (2016). Original Procedure for Choosing λ. http://varianceexplained.org/files/pi0boot.pdf

estimate(pvalues, LeastSlope())

Benjamini, Y., and Hochberg, Y. (2000). On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics. Journal of Educational and Behavioral Statistics 25, 60–83.

estimate(pvalues, TwoStep())
estimate(pvalues, TwoStep(α))
estimate(pvalues, TwoStep(α, <:PValueAdjustmentMethod)

Benjamini, Y., Krieger, A.M., and Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93, 491–507.

Storey's estimate with dynamically chosen λ

estimate(pvalues, RightBoundary())

Liang, K., and Nettleton, D. (2012). Adaptive and dynamic adaptive procedures for false discovery rate control and estimation. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 74, 163–182.

estimate(pvalues, BUM())

Pounds, S., and Morris, S.W. (2003). Estimating the occurrence of false positives and false negatives in microarray studies by approximating and partitioning the empirical distribution of p-values. Bioinformatics 19, 1236–1242.

estimate(pvalues, CensoredBUM())

Markitsis, A., and Lai, Y. (2010). A censored beta mixture model for the estimation of the proportion of non-differentially expressed genes. Bioinformatics 26, 640–646.

estimate(pvalues, FlatGrenander())

Langaas, M., Lindqvist, B.H., and Ferkingstad, E. (2005). Estimating the proportion of true null hypotheses, with application to DNA microarray data. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67, 555–572.

estimate(pvalues, ConvexDecreasing())
fit(ConvexDecreasing(), pvalues)

Langaas, M., Lindqvist, B.H., and Ferkingstad, E. (2005). Estimating the proportion of true null hypotheses, with application to DNA microarray data. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67, 555–572.

estimate(pvalues, Oracle(π₀))

combine(pvalues, <:PValueCombination)

combine(pvalues, Fisher())

Fisher, R.A. (1925). Statistical methods for research workers (Genesis Publishing Pvt Ltd).

Optionally with weights

combine(pvalues, Stouffer())
combine(pvalues, weights, Stouffer())

Stouffer, S.A. (1949). The American soldier. Vol. 1: Adjustment during army life (Princeton University Press).

Liptak, T. (1958). On the combination of independent tests. Magyar Tud Akad Mat Kutato Int Kozl 3, 171–197.

combine(pvalues, Logit())

Mudholkar, G.S., and George, E.O. (1977). The Logit Statistic for Combining Probabilities - An Overview (Rochester University NY, Dept of Statistics).

combine(pvalues, Tippett())

Tippett, L.H.C. (1931). The Methods of Statistics. An introduction mainly for workers in the biological sciences.

combine(pvalues, Simes())

Simes, R.J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73, 751–754.

combine(pvalues, Wilkinson(rank))

Wilkinson, B. (1951). A statistical consideration in psychological research. Psychological Bulletin 48, 156.

combine(pvalues, Minimum(PValueAdjustment()))

Higher criticism scores and threshold

estimate(pvalues, HigherCriticismScores())
estimate(pvalues, HigherCriticismThreshold())

Donoho, D., and Jin, J. (2008). Higher criticism thresholding: Optimal feature selection when useful features are rare and weak. PNAS 105, 14790–14795.

Klaus, B., and Strimmer, K. (2013). Signal identification for rare and weak features: higher criticism or false discovery rates? Biostatistics 14, 129–143.

BetaUniformMixtureModel(π₀, α, β)

Pounds, S., and Morris, S.W. (2003). Estimating the occurrence of false positives and false negatives in microarray studies by approximating and partitioning the empirical distribution of p-values. Bioinformatics 19, 1236–1242.

The MultipleTesting package is part of the Julia ecosphere and the latest release version can be installed with

pkg> add MultipleTesting

More details on packages and how to manage them can be found in the package section of the Julia documentation.

Contributions of any kind are very welcome. Please feel free to open pull requests or issues with your questions or ideas.

The package uses semantic versioning.