Rules or sayings passed down through generations that, to quote Wikipedia, contain “unverified claims with exaggerated and/or inaccurate details.”

• Example from my mother: “Don’t pull faces, if the wind changes then your face will stay that way.”

• Recently debunked example: “Don’t swim after eating - you can get stomach cramps.”

Often they are created with good reasons, like parents wanting to rest after eating and not watch their kids in the pool.

And often those reasons are lost over time, leaving only a rule without context. Then when the context changes the rule stops being valid without people realising it.

In statistics we also have many rules that are sometimes taught as hard rules, but were actually never meant to be more than temporary rules of thumb.

Or rules that were actually twisted over time from good intentions, through misunderstandings, to end up being quite wrong or bad practice.

And of course rules that were created in times before computing power and Big Data where the authors never conceived of the idea that I’d have a personal computer at home that can perform 100,000 t-tests per second.

• Assume the world is boring
• See something possibly interesting
• Work out how likely it is to see something at least that interesting under the assumption that the world is boring
  • This is called a p-value
• If the p-value is large (say more than 0.05) then we keep assuming the world is boring
• If the p-value is small (say less than 0.05) then we reject our assumption that the world is boring and conclude there’s at least one interesting thing going on

• The Central Limit Theorem says that averages follow a Gaussian (Normal) distribution
  • IF your sample size is large enough
• So if we care about the average then we can make judgements using this theorem

• The t-test seems to be the most popular tool in statistics after summary statistics and graphs
• Assuming you are interested in testing some hypothesized average \((\mu)\)
  • Usually “No difference”, implying an average difference of zero \((\mu=0)\)
• IF your sample size is large enough then
  • \(t=\)(The average minus \(\mu\))*\(\sqrt{n}\)/standard deviation
  • Follows a \(t_{n-1}\) distribution in a boring world, where \(n\) is the number of observations
• So if \(|t|\) is large (>2 is a good rule of thumb) then we reject the boring hypothesis

• It is standard practice in many places to do a test for equality of variances before doing a two sample t-test.
• If the variance test fails to reject then people do an equal variance t-test, and if it rejects then people do an unequal variance t-test.
• I also engaged in and taught this practice on occasion in the past.

But I will show you now that it is a waste of time and effort.

Just do the unequal variance test every time.

No power is lost when doing the unequal variance test on equal variance data, but accuracy is gained with unequal variances.

If the sample sizes are not the same, specially we double the sample size of the sample with the smallest variance, then we get that the unequal variance test is much better.

• A very similar, but more complex argument is made by proponents of a bootstrap version of the t-test.
• They say that you lose little power in the case where the data does meet the t-test assumptions, but gain a lot when it doesn’t.
• Not quite as clear cut though:
  • Power loss for small samples is not negligible.
  • t-test assumptions usually hold if done right thanks to the Central Limit Theorem and Slutsky’s Theorem.

• Fallacy: the observations must follow a normal distribution in order to do a two sample t-test.
• Truth: it helps if the observations within each group (or residuals) vaguely resemble a normal distribution but is not strictly necessary.
• A t-test assumes that the test statistic follows a t density, but is robust to many kinds of deviations.
• If the data in each sample does actually follow normal distributions then the assumptions are guaranteed to hold.
• If a sample is very skew (e.g. exponential) then it may help to do a transformation first and do the test on the transformed scale, but it needs to be justified (e.g. obvious outliers on a box plot).

Tests for normality don’t test for normality.

They assume normality and test for evidence to reject it.

• In small samples you often won’t find evidence even if the data structurally can’t be normal.
• In large samples you will usually find evidence, even if the departures from normality are tiny and irrelevant.
• But large samples enjoy the Central Limit Theorem; while small samples are where we really care about the distribution 😒
• This is why so many experts advocate for visual inspection over testing.

• If you keep doing tests you will ‘find’ something sooner or later.
• We expect coincidences to pop up 5% of the time even when nothing is happening.
• People keep doing tests and experiments until they find something significant and then try to explain it retroactively.
• They throw out the boring stuff (file-drawer problem)

End result: a lot of published research is not reproducible (over 40% in some psychology journals at one stage).

• Of course this is still miles better than popular media. I’m just saying that there’s room for improvement.

If you do 100 hypothesis tests (I’ve seen more than 1000 in a single PhD) then the probability of at least 1 false positive is

\[1-\left[(1-0.05)^{100}\right]>99.4\%\]

People are out there talking a lot of nonsense as a result.

Bonferroni said you could decrease your significance level, but I like the Holm-Bonferroni method more, where you inflate your p-values.

• It’s strictly more powerful and just as effective.
• It’s easy to implement, e.g. see ?p.adjust in R.
• Not as powerful as a good multivariate test though - assuming the multivariate assumptions hold 😉

• Practical significance is what really matters.
• Statistical significance should come first, to help avoid analysing coincidences;
• But once you find something statistically significant you have to ask what the practical meaning is.
• The key is the effect size, which is the measure of the practical effect of one thing on another.
• The best effect sizes are so big that anyone can see the impact, and the fancy stats becomes irrelevant!

We start with a proper regression, then we add a noise variable, then we add another 18 noise variables. In theory the regression should get worse right?

set.seed(92340)
x <- rnorm(30)
y <- 2 * x + 30 + rnorm(30)
m <- s <- vector("list", 3)
s[[1]] <- summary(m[[1]] <- lm(y ~ x))

X <- cbind(x, rnorm(30))
s[[2]] <- summary(m[[2]] <- lm(y ~ X))

X <- cbind(X, matrix(rnorm(30 * 18), 30, 18))
s[[3]] <- summary(m[[3]] <- lm(y ~ X))

knitr::kable(round(data.frame(R2 = sapply(s, with, r.squared), 
    adjR2 = sapply(s, with, adj.r.squared), AIC = sapply(m, AIC), 
    BIC = sapply(m, BIC)), 2))

We see \(R^2\) go up a lot. Further, we see \(Adj.R^2\) and AIC improve, with 2 of the noise variables significant at 5%.

• Most researchers work in such a way that nobody else understands what they are doing most of the time.
  • Most researchers coming back to their own work after a short time don’t understand what their own past selves were doing.
  • Just think of the average number of revisions on a published paper.
  • There is almost always a disconnect between what a researcher thinks, does, and eventually puts down in writing.

The result is that a shocking amount of published research is wrong, or just not reproducible.

Remember to understand rules and where they come from, don’t just memorise!

• Watch out for the worst tale in statistics: that it’s fine to build your model based on the data and then use that model to judge what relationships are significant based on the model fit to that data.
  • If you use observed relationships in the data to build a model for that data then you can’t test those relationships using that model on that data without quickly starting to talk a lot of nonsense.
  • You need new data that a model hasn’t seen to test a model. The best way to do that is via controlled experiments where the model is created in advance of collecting the data.