Descriptive Statistics Reconciler

A descriptive statistic is a summary: one number that stands for a whole collection of observations. That is its power and its trap. The number arrives without its recipe, and for most of the familiar summaries there is more than one recipe in honest use. Two careful programs can read the same ten values and report different quartiles, different skewness and different standard deviations, and neither has made a mistake.

This matters most at the moment of comparison. A colleague’s report says the third quartile was 24.25. Your spreadsheet says 25.75. Somebody starts checking the data for typos. The difference was a rule, chosen silently by software, and nobody wrote it down.

A summary is a number plus a convention

Start with the most familiar case. The variance of a set of values is the average squared distance from the mean, but “average” can divide by n or by n minus 1. Dividing by n gives the variance of the values as a complete population. Dividing by n minus 1 is Bessel’s correction, which removes the bias that appears when the sample mean stands in for an unknown population mean. Both are correct answers to different questions. A program that prints “variance” without saying which is asking you to guess.

The same split runs through the standard deviation, which is just the square root of one variance or the other. For the ten values 12, 15, 18, 22, 15, 30, 25, 15, 20 and 28, the population variance is exactly 33.6 and the sample variance is exactly 112/3, about 37.333. Their roots are 5.79655069848 and 6.11010092661. A reported deviation of 6.1101 came from the sample rule; 5.7966 came from the population rule. The figures differ by more than five percent, which is easily enough to make two reports look inconsistent.

Single values make the point sharper. With one observation the sample variance divides zero by zero and is not defined at all. A program that shows 0 there is reporting a convention of its own invention, and a reader may take it as evidence of no spread.

Nine ways to find a quartile

Quantiles are where the conventions multiply. To estimate the value below which a fraction p of a sample falls, a program has to decide where in the sorted list that fraction lands and what to do when it lands between two values. Hyndman and Fan catalogued nine rules used by statistical packages, and the Quantile article tabulates them as types 1 to 9. The first three jump from one observed value to the next. Types 4 to 9 interpolate along a straight line between neighbours and differ only in where they place the position, written h.

For the ten values above, every rule agrees that the first quartile is 15, because three of the values are 15 and they straddle every candidate position. The third quartile is another story. Types 1, 2, 3 and 5 give 25. Type 4 gives 23.5, type 6 gives 25.75, type 7 gives 24.25, type 8 gives 25.25 and type 9 gives 25.1875. Six different answers from one list of ten numbers. Hyndman and Fan recommended type 8, the article notes, while most packages chose type 6 or type 7 as their default. Recommendation and practice parted ways long ago.

That spread is the whole reason for a reconciler. Type 24.25 into one, ask for the quantile at p = 0.75, and only type 7 lands within half a unit of the last digit shown, that is from 24.245 up to but not including 24.255. The figure 25.00 matches types 1, 2, 3 and 5. A bare 25, with no decimals, claims less precision, so its window runs from 24.5 to 25.5 and types 8 and 9 also fit. The decimals a report prints are part of what it says.

Skewness repeats the pattern with three formulas. The Skewness article lists g1, the third central moment over the second raised to the power 3/2; b1, which uses the sample standard deviation instead; and G1, an adjusted form multiplied by the square root of n(n minus 1) over n minus 2, which the article says several statistical packages report. For the example data they come out at about 0.3851, 0.3288 and 0.4567. A reported skewness of 0.4567 names G1 and nothing else.

Kurtosis has the same three relatives, and one more twist: some sources subtract 3 so that a normal distribution scores zero, and some do not. The NIST/SEMATECH handbook keeps the original definition, without the 3, and warns readers to check which convention their software follows.

Resistance to outliers has conventions too

The mean and the standard deviation move a long way when one value goes wrong. Robust measures are built not to. The median absolute deviation is the median distance from the median, and multiplying it by about 1.4826, the reciprocal of the normal quantile at 3/4, makes it estimate the standard deviation when the data are normal. Some programs apply the factor by default and some do not, so a MAD of 4 and a MAD of 5.93 can describe the same ten values.

Trimmed and winsorized means discard or clamp a fraction at each end. The Truncated mean article says that 5 to 25 percent is usual, and that when the percentage does not give a whole number of points, some definitions interpolate between the two nearest whole counts. Others simply cut the count down. For the example, cutting one value from each end gives a 10 percent trimmed mean of 19.75, and replacing those two values by their nearest neighbours gives a winsorized mean of 20.1.

The Hodges-Lehmann estimator is the median of the means of every pair of values. The article on it insists that the pairs include each value with itself and remarks that many secondary sources omit that detail, so even this estimator comes in two versions in practice. Rousseeuw and Croux proposed two scale estimators built from pairwise distances, Sn and Qn. The article on robust measures of scale gives their efficiency for normal data as about 58 and 82 percent, against 37 percent for the median absolute deviation, and notes that Qn needs small-sample correction factors at modest n. A figure that quotes Qn without saying whether such a factor was applied cannot be reproduced exactly.

Weights add a final fork. A frequency weight says a value occurred that many times, so the Weighted arithmetic mean article shows that values 2, 4 and 5 with weights 2, 1 and 3 give exactly the same results as the raw list 2, 2, 4, 5, 5, 5, with a sample variance of 13/6. A reliability weight says how much to trust a value, and the unbiased variance then divides by V1 minus V2 over V1, where V1 and V2 are the sums of the weights and of their squares. Those same three rows treated as reliability weights give 65/22 instead. The rows did not change. The meaning of the weights did.

When arithmetic itself drifts

The second source of disagreement is quieter, and it has nothing to do with choosing a formula.

Computers usually store numbers in binary floating point, which cannot hold 0.1 or 0.01 exactly. Most of the time the error is far below anything printed. The textbook variance formula is the famous exception: subtract the squared sum over n from the sum of squares, and when the data sit far from zero both terms are huge and nearly equal. Their difference is a small number computed from two large rounded ones. Numerical analysts call this catastrophic cancellation, the loss of accuracy when good approximations to two nearby numbers are subtracted.

The Algorithms for calculating variance article gives the demonstration that is worth reproducing. Take 4, 7, 13 and 16 and add one thousand million to each. The sample variance is exactly 30, as it is without the offset. The naive formula in double precision returns -170.66666666666666, or -170.67 to two decimals. A variance cannot be negative, and yet a careful program printed one. With an offset of one hundred million it returns 29.333333333333332, which is wrong in a way that looks plausible. The two-pass method, subtracting the mean first, gets 30 in both cases.

Exact arithmetic removes the question entirely. A decimal typed with k digits after the point is a rational number, its digits over a power of ten, and sums, means and central moments of rationals are rationals. Holding them as fractions of large integers means 0.1 is exactly one tenth and the variance of the shifted values is exactly 30. Square roots are not rational, but they can be rounded once, correctly, from the exact fraction to any stated number of digits. What exactness cannot repair is rounding that happened before the numbers were typed. Data recorded to the nearest whole unit are summarised as whole units, faithfully and to the last digit.

Reporting a number someone else can check

The cure for most of these disagreements costs a few words. Say n. Say whether a variance or deviation divides by n or n minus 1. Name the quantile type, or the program and the function that produced it. Say which skewness and kurtosis formula was used and whether 3 was subtracted. Give the trimming percentage and how a fractional count was handled. Say whether weights count repetitions or express trust, and say how missing or unreadable entries were treated, because a value silently dropped changes every statistic at once.

Even with all that, a reconciler has limits worth stating plainly. It can list the conventions it knows that reproduce a figure at the decimals shown, but it cannot know which program produced the figure, and two rules can agree by coincidence on a particular data set. If no convention matches, the usual suspects are data that differ from yours: entries dropped as missing, a comma read as a thousands separator instead of a list separator, weights applied or ignored. It runs no hypothesis test and gives no interval; it only describes one data set.

The ten values used throughout this essay are a good first test. Load them, ask for the third quartile, and watch nine reasonable rules produce six different answers. Then type the figure your own software printed and see which rule it was using all along.