Why this file exists
Intro to Sampling Methods, Tuesdays from 5:30 to 8. The requirement was an R file that would run Neyman's allocation, and I didn't know the basics of R to begin with. I consulted a friend working in epidemiology and another in data analysis; they taught me the basics and helped me construct a block of code to do the allocation. Confusion came first, then curiosity to understand it better. Neyman's allocation stayed the most useful concept of the course: widely applicable, near necessary when sampling. I wanted to go back, deconstruct the file I built for that class, then improve on it. As one of my influential psychological stats professors once told me, the best way to learn something for real is to teach it to someone else. So here I am, trying to explain it to someone else.
This post opens a series of prefaces, one for each training file. Each preface tells why its file exists and teaches the idea underneath it; the file holds the working implementation. Public files serve strangers, so the training files are public.
The question
When a population divides into strata (groups that differ meaningfully from one another), a stratified sample usually beats a simple random sample: you sample each stratum separately, then combine the results. A simple random sample draws from the whole population at once and leaves each group's share to chance. The question Neyman's allocation answers is how many of your total observations to spend in each stratum.
The answer
The naive answer, proportional allocation, spends sample in proportion to stratum size alone. Neyman's allocation improves on it by also weighting by each stratum's variability: strata with more internal spread earn more sample, because that is where extra observations buy the most precision. Precision here means how little the survey's answer would move if you ran the whole survey again.
Take a stratum where everyone would give the same answer: one interview tells you everything, and every interview after it buys nothing. Homogeneous strata need fewer observations to pin down. Statisticians measure internal spread with the standard deviation: roughly, how far a typical member of a group sits from the group's average.
Each stratum's weight is its size times its standard deviation. A stratum's share of the sample is its share of the total weight. That ratio is the entire mathematical content of Neyman's allocation. A useful sanity check: when every stratum has the same standard deviation, the spread terms cancel and Neyman's allocation collapses back into proportional allocation.
Jerzy Neyman set out the argument in a 1934 paper in the Journal of the Royal Statistical Society, at a time when stratified surveys mostly sampled each stratum in proportion to its size. Aleksandr Chuprov had derived the same rule in 1923; Neyman overlooked the earlier paper and acknowledged Chuprov's priority in 1952.
The worked example
Suppose a survey of new mothers recruited through three settings (a large hospital, a small clinic, and home births), with a budget of 300 interviews. Pilot data suggest the small-clinic population varies far more on the outcome than the other two strata. Proportional allocation ignores that spread: on size alone it would give the strata 225, 60, and 15 interviews respectively. Neyman's moves nearly fifty interviews from the large hospital to the small clinic, where the pilot standard deviation says each added observation buys the most precision. That reallocation is the entire point of the method.
The standard deviations come from a pilot study, a previous survey wave, or an informed guess. The allocation is only ever as good as these inputs; Neyman's is "optimal" conditional on knowing them.
The file
The training file lives at melissaqwood.com/files/neymans-allocation.R and holds the working implementation, commentary for readers coming from Python, and the refinements and edge cases (left as notes, deliberately). The file also takes the answer from real numbers to whole observations, because you cannot sample 12.4 people.