Closely related: How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples?

Do you know how the reference lines were drawn in your software? Usually the line goes through the 25th & 75th percentile, but that doesn't seem to be the case here.

Thank you for the link above - very good read. I'm using SPSS, but I am not 100% sure how the software is choosing the line.

The idea that you can have "a hard and fast normal vs non-normal" is illusory. If you aren't in a position to say (e.g. from some a priori knowledge of the distribution shape, the sample size or both) "the t-test should be fine", you should not assume it. Deciding which test to do on the basis of the sample leaves neither test having its nominal properties, and the resulting overall properties are often substantially less useful than simply doing Mann-Whitney; if you have reason (a priori) to think that the tails are not too heavy & distribution not very skew, the t-test should be fine

If those plots are typical of your distribution shapes, it will matter little which test you choose. You have mild skewness, it won't harm the properties of the t very much (though I'd lean toward MW or a permutation test of means myself). If you use a t, make sure you allow for unequal variance (that is, do Welch or something along those lines --- and don't test for equal variances, just assume they're different).

Updated links: One can use the Shapiro-Wilk test to test for equivalence to normality, rather than difference from normality using the tost package for Stata. Doing so and combining with the standard test for different from normality will permit one to distinguish between trivial and relevant difference from normality, as well as to conclude equivalence to normality.

@Alexis Can you clarify in what way an omnibus test like the Shapiro-Wilk would be considered one sided (since TOST is "two one-sided tests")? With say a t-test or a Mann Whitney or even a comparison of two variances, the alternatives are "bigger" or "smaller" in a way that makes "one side" clear. But there's an infinity of ways to be non-normal. I'm having trouble visualizing what you mean here.

1/3 Hi @Glen_b ! Thank you for the question. Omnibus tests for equivalence (TOST or UMP) can be expressed ($H_{0}:$ all samples differ from all samples by at least some amount; $H_{A}:$ At least one sample differs from at least one other sample by less than some amount). See chapter 7 of Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority. Chapman and Hall/CRC Press, second edition for more on omnibus tests for equivalence.

2/3 @Glen_b My logic for the Shapiro-Wilk and Shapiro-Francia test results from (a) the fact that the $W$ and $W'$ test statistics are approximated by the $Z$ distribution, and (b) TOST tests for equivalence and their hypotheses may be expressed purely in terms of the test-statistic distribution (which is better for standardizing interpretation of equivalence, and worse for expressing in with respect to the units of the measured variable).

3/3 @Glen_b If you would like to critique this idea more in depth with me I would be delighted (i.e. mayhap I am gravely mistaken, or mayhap this idea could be improved, etc.).

@Alexis [feel free to ignore this bit - it's basically just me reminding myself of how the test works] If xᵢ, i=1,...,n are the data and yᵢ are the corresponding order statistics, and mᵢ are a set of corresponding S-W scores, the Shapiro-Wilk test consists of a numerator which is the square of m'y and whose denominator is the sums of squared deviations of the x's from their sample mean (n-1 times the usual variance of x and perforce, n-1 times the variance of y).

Algebraically, the ratio is always ≤1. When the order statistics are for a sample from the normal distribution, the numerator is generally very close in size to the denominator. It's akin to a squared correlation between order statistics and their expected value (except that the Shapiro Wilk scores incorporates the variance-covariance matrix of standard normal order statistics, in a manner akin to a standardized Mahalanobis distance); in small samples this will improve the power,

I think its impact reduces in very large samples though (it's a while since I looked at that stuff). . . . . . . . . [end of bit to ignore]

When you have a sample that is from a distribution which is different from the normal, the yᵢ values (the order statistics) are far from their expected values, reducing the correlation with the S-W scores and the typical size of the numerator decreases in relative to the variance on the denominator.

That is, you reject only for small values of W.

It doesn't matter what you do with W after that, up to monotonic transformation; all the values in the rejection region will be up one end of the distribution.

It converts deviations from normality, in all their multifarious dimensions, into a single direction of deviation from it, rather like an F statistic in one-way ANOVA converts all the various patterns of deviations in means from equality into a single statistic that will tend to be larger than it is under equality, whatever that pattern may be.

As a result, the test in either case (F in ANOVA, or the Shapiro Wilk) is inherently "one sided".

(Similar comments apply to the Shapiro-Francia test)

It doesn't really matter if you convert W into some Z-score, your rejection region (corresponding to values of W inconsistent with normality) will still only be in one tail.

Values in the other tail are the ones most consistent with normality, in the same way that a very small F-value in an ANOVA should not lead you to conclude that the means differ.

Analogously, I could readily convert (via monotonic transformation) F to a Z-score, but I should still only conclude the means differ for the tail of that Z which corresponded to a large F.

I don't think there's any way to do two-one-sided tests for this

Though it's possible I misunderstand the intent of what you're doing (i.e. if TOST is more of a misnomer for your procedure)

That is, it might be reasonable to do an equivalence test based in one tail, but I really wouldn't refer to it as TOST, since "TOST" is literally using two tails.

You could have one-OST, though setting some boundary for equivalence may be a problem. There's no inherent "scale" on which one could reasonably argue "well, below that value of [log(1-W)-µ]/σ it should be considered equivalent to normal"; you're just choosing a different alpha on a hypothesis test then. If I am comparing heights or driving speeds or tolerances on the size of a bolt, I can see ways to say "5mm in height makes no odds because individual heights vary that much across a day" or

"half a km/h won't matter because this speedometer is only calibrated to that accuracy" or "2 thousandths of an inch makes no difference; that's inside the specified tolerance for these bolts". . . . In all those cases the line of equivalence isn't just some arbitrary thing; you can really make some argument about why that "close" really is close enough.

What makes a value inside an equivalence statement about W "a trivial difference from normal" in a way that doesn't correspond to just a shift in significance level?

I don't presently see any reasonable way to do an equivalence test with a Shapiro-Wilk approach. It's not even like you're dealing with standard scores -- it's not distance of one distribution from another in standard units but distance in terms of the statistic's own distribution (or of a monotonic transformation of it), so the criterion is sample-size dependent, which is completely at odds with what I see the notion of equivalence being about.

[More broadly I think that even in the case of equivalence tests where the boundary of equivalence is based on some line determined purely from some rule applied to standardized scores (which I don't think you have here) without reference to what that actually means in terms of the thing being measured is generally a recipe for bad statistical practice, but that's kind of beside the point for the present discussion, which is mostly about whether it even makes sense to look in both tails.]

I can see that there could be some situations where it might just about make sense for (say) a Lilliefors test. You're talking about deviations in cdf, and if you want to say "if my calculated probabilities assuming normality are within 0.005 I'll call it equivalent", so that sort of corresponds to bounding |Ĝ-Gₒ| (the difference of the ecdf from the fitted normal cdf)

I don't think there's a similar sense where we can talk about equivalence for Shapiro-Wilk. Maybe there is one but I don't think that will be doable in this way.

Hi @Glen_b! Thanks for the insight. Very briefly: I really appreciate yer input. I did cotton to W being an asymmetric statistic. There are omnibus equivalence tests (see the Wellek reference).

The TOST-specific approach comes from the Z approximation for the Shapiro-Wilk, using the equivalence boundary is expressed in standardized scores (which in TOST are simply non-standardized scores divided by the standard error used to calculate, e.g., Z), and I think being very considered about using such scores is warranted (although I am not convinced that they are "bad practice" as long as you are being explicit.).

I would like to chat about all this more. But it's past my bedtime. :) Catch you tomorrow.

PS: Standardized equivalence can be interpreted as something like "how far past the nominal rejection boundary does a test for difference need to be to be relevant"

Sure, but we've basically just replaced "equivalent" with its complement ("relevant difference"); we're still left to explain how exactly a particular deviation comes to be called relevant/equivalent.

Our library has Welleck but I won't be able to look at it until tomorrow at the earliest.

One could certainly make omnibus equivalence tests in some cases. For example, if I decided that two groups whose population means were say 0.1 units apart were equivalent, then it would be perfectly possible to set a population equivalence at a given sum of squared deviations so that mean-equivalence was certain to hold. In equal-sample size one way ANOVA this would be really straightforward.

So in a case where I can construct a meaningful measure of what equivalence is and have a way to relate that to the test statistic, there's no inherent difficulty. Moving outside 'standard' tests, I could work with a test statistic based on sums of pairwise absolute deviations in mean, say and construct a permutation test; that would be trivial to build a one-OST for because you can just set overall equivalence at the single-mean-pair equivalence already established and again get a guarantee.

But in the case of the Shapiro-Wilk I see no way to do anything like this that carries a sense of an actually "relevant difference"; the dimensionality of possible ways of distributions be different is much larger, and the relationship to practical effects on some task we're using normality for is considerably less direct/clear.

"relevant difference" really only merits being called relevant when we can show it has a practical importance for something. In my bolt example, obviously a bolt that won't fit in the thread in the nut matters -- if the bolt won't fit, that certainly makes a difference in a clearly practical sense (if you can't build it, they won't come)

but now consider something differing by a standard deviation. I'd say ... "so?". Why would standard deviations matter? If the standard deviation on my manufacturing process is only say 12% of the actual allowable tolerance on the bolt, being out by 1sd or even 2, or 3, matters not in the least, I will still not go remotely close to having a problem with the bolt.

On the other hand if my standard deviation is much larger, I might exceed the actual tolerance at a fraction of a standard deviation; in general Z scores don't tell me a thing about what's relevant for some actual purpose, even when I'm simply standardizing a difference in a pair of means.

That's of course a different sort of issue to the practical one above of defining equivalence and relating equivalence to the test statistic

It's an interesting conversation for me; clearly you've read more about equivalence tests than I have, so I expect I'll be learning a few things along the way.

> we're still left to explain how exactly a particular deviation comes to be called relevant/equivalent.

Yes. But equivalence thresholds are always an explicit researcher choice, except when articulated by regulation as with the FDA in the US for bioequivalence.

> In equal-sample size one way ANOVA this would be really straightforward.

Chapter 7 of Wellek! ;)

"At least I don't presently see any good way to do this directly in terms of W." I suppose that's fair. I could restrict the program to only permit in terms of (Royston's) Z approximation and a stdandardized equivalence region.

> but now consider something differing by a standard deviation. I'd say ... "so?". Why would standard deviations matter? If the standard deviation on my manufacturing process is only say 12% of the actual allowable tolerance on the bolt, being out by 1sd or even 2, or 3, matters not in the least, I will still not go remotely close to having a problem with the bolt.

I really like this bolt example! Of course, tolerance sometimes is measured in standards in the sciences... as in the "5-sigma" or "8-sigma" discoveries in particle physics and astrophysics. I think a "cookie cutter" equivalence boundary—e.g., 2.146 is just what one always uses—is a problem. In my software documentation and courses I explicitly caution against precisely this approach.

"TOST" is indeed a misnomer for omnibus tests such as F or W, but is not a misnomer when the omnibus test uses a Z-distributed variable. As implemented in my software for the non-standardized equivalence threshold (what I term specifying equivalence in terms of Delta), I translate a user-specified units of equivalence of the W distribution into a measure of equivalence in the Z distribution (following Royston), which is then used to conduct the actual test for equivalence and relevance test.

"but is not a misnomer when the omnibus test uses a Z-distributed variable." is predicated on the notion that the test can deviate from a mean of 0 in both positive and negative directions.

> I'd say ... "so?". Why would standard deviations matter?

And you would be welcome to do so... in more or less precisely the same way you would be welcome to ask "Why does a 1% Type I error rate matter?"

Whoops... i neglected to put an ' @Glen_b ' in my replies.

I think you're largely missing the issues I am trying to convey. Using Z instead of W does not help you get around the problem with using W in any way; if anything it serves to simply obscure that you're not dealing with any reasonable sense of the word 'equivalent' in this case.

(sorry about the deletion there; I decided not to pursue anything but the main issues and so removed comments that can be left aside to focus on the most important parts. In fact right now I will focus on only one of the two main ones - one vs two tails. I should probably delete my first comment above, since that's the other issue)

You say "following Royston" but you're not following Royston. Royston uses only the upper tail of the distribution of log(1-W), corresponding to the lower tail of the distribution of W. He does a one tailed Z test because W is one tailed. If you consider two tails when dealing with any monotonic transformation of W you'll be doing it wrong; the other tail has the cases suggesting the least deviation from normality.

BTW my lack of replies wasn't due to a missing "@Glen_b". I was asleep. There's no harm in

@-pinging, but I should normally see a notification if you reply under my comment