Hypothesis Testing and Sampling Distributions: What a p-Value Can and Cannot Say

A hypothesis test compares an observed statistic with the distribution that the statistic would have under a specified null model. The reference distribution may be exact, asymptotic, or generated by a justified randomization scheme.

The difficult step is usually establishing that reference distribution. Computing a tail area is straightforward only after the sampling assumptions, statistic, and definition of extremeness have been fixed.

Data distributions and sampling distributions are different objects

A data distribution describes individual observations. A sampling distribution describes a statistic computed from repeated datasets of a specified size under a specified sampling process.

For example, a population might have mean and variance

\[\mu,\qquad \sigma^2.\]

The sample mean is a random variable before data collection:

\[\bar X=\frac1n\sum_{i=1}^nX_i.\]

Under independent observations with the same mean and variance,

\[\mathbb E[\bar X]=\mu,\]

and

\[\begin{aligned} \operatorname{Var}(\bar X) &= \frac1{n^2} \operatorname{Var}\left(\sum_iX_i\right)\\ &= \frac1{n^2}\sum_i\sigma^2\\ &= \frac{\sigma^2}{n}. \end{aligned}\]

Independence removes the covariance terms. If observations are dependent, the general formula is

\[\operatorname{Var}(\bar X) = \frac1{n^2} \left[ \sum_i\operatorname{Var}(X_i) + 2\sum_{i<j}\operatorname{Cov}(X_i,X_j) \right].\]

The denominator involving sample count is therefore not a universal consequence of averaging. It follows from a particular dependence structure.

Once observations have been collected, the numerical sample mean is fixed. Its sampling distribution describes the procedure that produced it, not uncertainty caused by repeatedly recalculating the same dataset.

Defining a p-value through a statistic and a null model

Suppose larger values of a chosen statistic indicate greater disagreement with the null. A simple-null p-value is

\[p_{\mathrm{obs}} = P_{H_0}(T\ge T_{\mathrm{obs}}).\]

The probability is over hypothetical datasets generated under the null model and sampling design.

This is not

\[P(H_0\mid\text{data}).\]

The latter requires probabilities for competing hypotheses and prior weights. A p-value conditions on a null model instead of assigning that model a posterior probability.

The definition also depends on the statistic. A two-sided test might use an absolute deviation. A one-sided test might use the original signed statistic. Choosing the direction after observing the result changes the procedure whose error probability needs to be analyzed.

For a composite null, a reference distribution must remain valid across the allowed null parameters. Sometimes a pivot removes the nuisance parameters. Another construction takes the supremum of the relevant tail probability over the null parameter set. Substituting an arbitrary nuisance estimate without justification does not automatically produce a valid p-value.

An exact coin test by counting sequences

Assume a fixed number of independent tosses with common head probability. Under fairness, each length-fifteen sequence has probability

\[2^{-15}.\]

The number of sequences containing exactly a specified head count is the binomial coefficient, so

\[P_{H_0}(X=k) = \binom{15}{k}2^{-15}.\]

For ten heads,

\[P_{H_0}(X=10) = \frac{3003}{32768} \approx 0.0916443.\]

That is the probability of one head count, not the p-value.

Use absolute deviation from the null expectation as the two-sided statistic:

\[T=\left\lvert X-\frac{15}{2}\right\rvert.\]

Ten heads is as far from the expectation as five heads. Counts at least as extreme are therefore the upper tail starting at ten and the lower tail ending at five.

By symmetry,

\[p = 2\sum_{k=10}^{15}\binom{15}{k}2^{-15}.\]

The upper-tail coefficients are

\[3003,\ 1365,\ 455,\ 105,\ 15,\ 1,\]

which sum to

\[4944.\]

Thus

\[p=\frac{9888}{32768}=0.3017578125.\]

For twelve heads, the upper-tail coefficients are

\[455,\ 105,\ 15,\ 1,\]

which sum to

\[576.\]

The two-sided p-value is

\[p=\frac{1152}{32768}=0.03515625.\]

These results describe tail probabilities under fairness. The first does not establish that there is no evidence of any kind about the head probability; it says this particular test does not reject at conventional small significance levels. The second does not give the posterior probability that the coin is fair.

The definition of “two-sided” deserves attention for asymmetric discrete distributions. Doubling one tail, ordering outcomes by probability, and ordering by distance need not agree. Here symmetry and the chosen absolute-deviation statistic make the calculation explicit.

Why p-values control a repeated-testing error rate

Suppose the null distribution of the statistic is continuous, with cumulative distribution function

\[F_0.\]

For an upper-tail test,

\[p=1-F_0(T).\]

The probability integral transform gives a uniform random variable under the null. To see the relevant consequence directly,

\[P_{H_0}(p\le\alpha) = P_{H_0}\left(T\ge F_0^{-1}(1-\alpha)\right) = \alpha,\]

with the usual continuous-distribution interpretation of the quantile.

Thus rejecting when the p-value is at most the chosen significance level has that null rejection probability.

Discrete statistics usually give a conservative inequality:

\[P_{H_0}(p\le\alpha)\le\alpha.\]

For the fifteen-toss two-sided test at level five hundredths, rejection occurs for

\[X\le 3 \quad\text{or}\quad X\ge 12.\]

Its actual null rejection probability is

\[0.03515625,\]

not exactly five hundredths. The discreteness prevents the non-randomized test from filling the nominal error budget exactly.

This is a property of the testing rule under repeated null sampling. It is not the probability that a particular rejected hypothesis is false.

Power is a different probability:

\[\operatorname{Power}(\theta) = P_\theta(\text{reject }H_0),\]

evaluated under a specified alternative. A large p-value can arise because the null is compatible with the observations or because the test has little power against an effect of interest. Non-rejection alone does not distinguish those possibilities.

Exact normal sampling and the central limit approximation

If observations are independent normal variables with common mean and variance, their average is exactly normal:

\[\bar X\sim \mathcal N\left(\mu,\frac{\sigma^2}{n}\right).\]

For example, this follows by multiplying their moment-generating functions after scaling each observation by the reciprocal sample count. The exponents combine into the normal moment-generating function with the displayed mean and variance.

Hence, with known population variance, the statistic

\[Z=\frac{\sqrt n(\bar X-\mu_0)}{\sigma}\]

is standard normal under the null mean.

For non-normal observations, the corresponding conclusion is generally approximate. The central limit theorem can be understood through characteristic functions. Standardize an individual observation to mean zero and variance one. Near zero, its characteristic function has expansion

\[\varphi(t)=1-\frac{t^2}{2}+o(t^2).\]

For the standardized sum,

\[\varphi_n(t) = \left[\varphi\left(\frac t{\sqrt n}\right)\right]^n.\]

Substituting the expansion gives

\[\varphi_n(t) = \left[ 1-\frac{t^2}{2n}+o(n^{-1}) \right]^n \longrightarrow e^{-t^2/2},\]

the standard normal characteristic function.

This argument explains why averaging can create a normal reference distribution without normal individual observations. It does not supply a universal small sample size at which the approximation becomes accurate. Tail shape, dependence, and moment conditions matter.

Why estimating variance produces Student’s t distribution

Assume independent normal observations with unknown positive variance. Define

\[S^2=\frac1{n-1}\sum_i(X_i-\bar X)^2.\]

The proposed statistic is

\[T=\frac{\bar X-\mu_0}{S/\sqrt n}.\]

To derive its distribution, standardize the observations under the null:

\[Z_i=\frac{X_i-\mu_0}{\sigma}.\]

The vector of these variables has independent standard normal coordinates.

Choose an orthogonal matrix whose first row is

\[\frac1{\sqrt n}(1,\ldots,1).\]

After applying it, the first transformed coordinate is

\[U_1=\frac{\sqrt n(\bar X-\mu_0)}{\sigma}.\]

Orthogonal transformation preserves the standard multivariate normal density because it preserves squared length and has Jacobian magnitude one. The transformed coordinates are therefore still independent standard normals.

The remaining coordinates span the space orthogonal to the all-ones vector. Their squared sum is the squared length of the centred residual vector:

\[V=\sum_{j=2}^nU_j^2 = \frac{\sum_i(X_i-\bar X)^2}{\sigma^2} = \frac{(n-1)S^2}{\sigma^2}.\]

By definition,

\[V\sim\chi^2_{n-1}.\]

It is independent of the first transformed coordinate. Therefore

\[T = \frac{U_1}{\sqrt{V/(n-1)}} \sim t_{n-1}.\]

The unknown scale cancels. The degrees of freedom arise because estimating the mean removes one direction from the residual vector.

Normality is doing more than making a histogram bell-shaped. It supplies the exact independence between the mean direction and the residual directions. Without it, the finite-sample t reference distribution is not generally exact.

Deriving the t density from the normal and chi-squared variables

Let the denominator’s degrees of freedom be

\[\nu.\]

A chi-squared variable has density

\[f_V(v) = \frac{v^{\nu/2-1}e^{-v/2}} {2^{\nu/2}\Gamma(\nu/2)}, \qquad v>0.\]

One route to this density is to write the joint standard normal density in radial coordinates: squared radius is the sum of squared coordinates, and the radial volume factor produces the displayed power of the squared radius.

Conditional on the denominator variable, the t statistic is a scaled standard normal. The change-of-variables factor is

\[\sqrt{\frac v\nu}.\]

Thus its density is

\[f_T(t) = \int_0^\infty \sqrt{\frac v\nu} \frac{e^{-t^2v/(2\nu)}}{\sqrt{2\pi}} f_V(v)\,dv.\]

Collect powers and exponential terms. The remaining integral has the gamma form

\[\int_0^\infty v^{a-1}e^{-bv}\,dv = \frac{\Gamma(a)}{b^a},\]

obtained by the substitution of the product of the rate and integration variable.

The result is

\[f_T(t) = \frac{\Gamma((\nu+1)/2)} {\sqrt{\nu\pi}\,\Gamma(\nu/2)} \left(1+\frac{t^2}{\nu}\right)^{-(\nu+1)/2}.\]

For large absolute arguments, this decays as a power rather than as a Gaussian exponential. Small denominator estimates create the heavy tails.

As the degrees of freedom grow, the normalized chi-squared denominator approaches one, and the ratio approaches a standard normal variable. That is why the t distribution approaches the normal as information about scale increases.

A worked t test with an analytically checkable tail

Take the constructed sample

\[1,\ 2,\ 3,\ 4,\ 5\]

and test a null mean of two under the independent normal model.

The sample mean is three. The centred sum of squares is

\[(-2)^2+(-1)^2+0^2+1^2+2^2=10.\]

Therefore

\[S^2=\frac{10}{4}=\frac52,\]

and

\[\frac S{\sqrt5}=\frac1{\sqrt2}.\]

The observed statistic is

\[T_{\mathrm{obs}} = \frac{3-2}{1/\sqrt2} = \sqrt2.\]

The reference distribution has four degrees of freedom. Its density simplifies to

\[f_4(t)=\frac38\left(1+\frac{t^2}{4}\right)^{-5/2}.\]

The two-sided tail can be integrated explicitly. Substitute

\[u=\frac{t}{\sqrt{4+t^2}}.\]

Then

\[t=\frac{2u}{\sqrt{1-u^2}}, \qquad dt=2(1-u^2)^{-3/2}\,du.\]

The density times the differential becomes

\[f_4(t)\,dt=\frac34(1-u^2)\,du.\]

Hence, for a positive observed statistic,

\[P(\lvert T_4\rvert\ge t) = 1-\frac32u+\frac12u^3.\]

At the observed value,

\[u=\frac1{\sqrt3}.\]

Therefore

\[p = 1-\frac4{3\sqrt3} \approx 0.23019964.\]

This tail probability is derived from the density, not supplied as an unexplained software output.

Using the variance denominator equal to the sample count would instead give

\[S_{\mathrm{wrong}}^2=\frac{n-1}{n}S^2.\]

The resulting statistic would be inflated by

\[\sqrt{\frac n{n-1}}.\]

For this sample it would equal the square root of two and a half rather than the square root of two. Dividing by the sample count is valid for a different estimator, such as the normal variance MLE, but not when inserting that estimate into the stated t pivot unchanged.

Confidence intervals invert the same tests

A two-sided t test accepts a candidate mean when

\[\left\lvert \frac{\bar X-\mu}{S/\sqrt n} \right\rvert \le t_{\nu,1-\alpha/2}.\]

Solve this inequality for the candidate mean:

\[\bar X-t_{\nu,1-\alpha/2}\frac S{\sqrt n} \le \mu \le \bar X+t_{\nu,1-\alpha/2}\frac S{\sqrt n}.\]

This is the corresponding confidence interval.

For four degrees of freedom and a five-percent two-sided tail, the preceding explicit tail formula gives the critical value by solving

\[1-\frac32u+\frac12u^3=0.05\]

and transforming back to the t coordinate. Numerically,

\[t_{4,0.975}\approx 2.776445.\]

The constructed sample therefore gives

\[3\pm\frac{2.776445}{\sqrt2},\]

or approximately

\[[1.036757,\ 4.963243].\]

The null mean of two lies inside, agreeing with non-rejection.

Under the model, this interval-producing procedure covers the fixed population mean with the stated long-run probability. A particular realized interval either contains that mean or does not. The frequentist coverage statement concerns the random interval before observing the data.

Why count deviations lead to a chi-squared statistic

Suppose each of a fixed number of independent trials falls into one of several mutually exclusive categories, with null probabilities

\[p_1,\ldots,p_k>0, \qquad \sum_jp_j=1.\]

Write each count as a sum of indicator variables:

\[O_j=\sum_{r=1}^nI_{rj}.\]

For a single trial,

\[\mathbb E[I_{rj}]=p_j, \qquad \operatorname{Var}(I_{rj})=p_j(1-p_j).\]

Different category indicators for the same trial cannot both equal one. Thus

\[\operatorname{Cov}(I_{ri},I_{rj})=-p_ip_j \qquad\text{for }i\ne j.\]

Summing across independent trials gives

\[\mathbb E[O_j]=np_j,\] \[\operatorname{Var}(O_j)=np_j(1-p_j),\]

and

\[\operatorname{Cov}(O_i,O_j)=-np_ip_j.\]

Define scaled deviations

\[Z_j=\frac{O_j-np_j}{\sqrt{np_j}},\]

and the vector

\[s= \begin{pmatrix} \sqrt{p_1}\\ \vdots\\ \sqrt{p_k} \end{pmatrix}.\]

Their covariance matrix is

\[\operatorname{Cov}(Z)=I-ss^{\mathsf T}.\]

Since the squared length of this vector is one, the covariance matrix is an orthogonal projection with rank one less than the number of categories.

There is also an exact linear restriction:

\[s^{\mathsf T}Z = \frac1{\sqrt n}\sum_j(O_j-np_j)=0.\]

The count total is fixed, so one direction of variation is absent.

By the multivariate central limit theorem, the scaled deviations approach a normal vector on the remaining subspace. Rotating to an orthonormal basis of that subspace produces asymptotically independent standard normal coordinates.

Their squared length is

\[\lVert Z\rVert^2 = \sum_j\frac{(O_j-np_j)^2}{np_j}.\]

Therefore Pearson’s statistic satisfies

\[Q = \sum_j\frac{(O_j-E_j)^2}{E_j} \ \xrightarrow{d}\ \chi^2_{k-1}.\]

The denominator is the expected count because that scaling turns the covariance into a projection. Treating the category deviations as independent normal variables would give the wrong derivation and the wrong degrees of freedom.

A goodness-of-fit calculation and its approximation boundary

Construct three category counts:

\[O=(30,20,10), \qquad n=60.\]

Under equal category probabilities, every expected count is twenty. The statistic is

\[Q = \frac{(30-20)^2+(20-20)^2+(10-20)^2}{20} = 10.\]

The asymptotic reference distribution has two degrees of freedom. Its density is

\[f(v)=\frac12e^{-v/2}, \qquad v>0,\]

so its upper tail is

\[P(\chi^2_2\ge q)=e^{-q/2}.\]

The approximate p-value is therefore

\[p\approx e^{-5}\approx 0.006737947.\]

The integration is exact for the chi-squared distribution. Using that distribution for finite multinomial counts is the approximation.

For the six-category construction

\[22,\ 24,\ 38,\ 30,\ 46,\ 44,\]

the total is two hundred and four, giving expected counts of thirty-four under equal probabilities. The statistic is

\[Q = \frac{144+100+16+16+144+100}{34} = \frac{260}{17}.\]

With five reference degrees of freedom, its approximate tail is

\[p\approx 0.00917684.\]

The statistic’s arithmetic is fully determined by the counts. No simulation discrepancy or benchmark result is needed to justify it.

When expected counts are small, the discrete count distribution may be poorly approximated by the continuous reference distribution. Exact enumeration or a null simulation can instead calibrate the chosen statistic, provided the sampling model is correct.

If category probabilities are fitted from the same observations, estimating parameters typically removes additional directions. Under regular fitting conditions, estimating a specified number of independent parameters changes the reference degrees of freedom to

\[k-1-r.\]

This subtraction is not automatic for arbitrary fitting procedures, boundary parameters, or data-dependent category selection. It follows from the geometry of the fitted model and its asymptotic tangent space.

Dependence and multiple testing alter the reference experiment

Repeated measurements can make a nominal sample count misleading. Suppose there are independent clusters, each containing the same number of observations. Let individual variance be

\[\sigma^2,\]

and let within-cluster pairwise correlation be

\[\rho.\]

For a cluster of size

\[m,\]

the variance of its sum is

\[m\sigma^2+m(m-1)\rho\sigma^2.\]

For a total of

\[n\]

observations across independent clusters, the overall mean therefore has variance

\[\operatorname{Var}(\bar X) = \frac{\sigma^2}{n} \left[1+(m-1)\rho\right].\]

With cluster size four and correlation one half, the inflation factor is

\[1+3\left(\frac12\right)=\frac52.\]

Ignoring that dependence understates the standard error by the square root of this factor. Repeated slices or studies do not become independent merely because they occupy separate rows.

Pairing can also help when used correctly. For two methods evaluated on the same independent experimental units, define a difference per unit and analyze the differences. The relevant variance includes the within-unit covariance between the methods, rather than pretending their measurements came from unrelated units.

Multiple tests create another change to the reference experiment. For independent null tests each having rejection probability five hundredths, the probability of at least one rejection among twenty tests is

\[1-(1-0.05)^{20} \approx 0.641514.\]

Without independence, the union bound still gives

\[P(\text{at least one false rejection}) \le \sum_{j=1}^mP(\text{false rejection in test }j).\]

Assigning each test a level equal to the desired family level divided by the number of tests therefore controls the familywise error probability by this bound.

Likewise, repeatedly checking results and stopping when a threshold is crossed is a different procedure from one fixed test. The sampling distribution or decision rule must account for the stopping design.

Revision checklist

Can I do this without looking?	Check
Distinguish observations from a statistic’s sampling distribution	Specify what is repeatedly sampled
Define a p-value	State the null, statistic, and extremeness ordering
Reproduce the coin tails	Count all included sequences
Explain discrete conservatism	Calculate the actual null rejection probability
Derive the sample-mean variance	Show where covariance terms disappear
Separate exact normal results from asymptotic ones	State the assumptions behind each
Derive the t pivot	Rotate normal observations into mean and residual directions
Explain the degrees of freedom	Count the residual subspace dimensions
Compute the worked t tail	Carry out the substitution in the density integral
Invert a test into a confidence interval	Solve the acceptance inequality for the mean
Derive Pearson’s reference distribution	Obtain the projected covariance matrix
Identify approximation failures	Examine expected counts and fitted parameters
Account for repeated measurements	Include within-cluster covariance
Account for multiple testing	Define the family and repeated-testing procedure

Why it matters for my work

The experimental unit and dependence structure determine the reference distribution. For model comparisons, I should establish those before calculating a p-value, and report effect estimates and uncertainty alongside the test.

What I have not resolved

I need to specify which sources of variation each experiment is intended to represent: patients, acquisition sites, training runs, or combinations of them. A single test cannot answer all of those sampling questions without an explicit design.