The 3% margin of error often quoted in polling has the following logic. It uses a sample size of 1000, and calculates a confidence interval as follows: \( \hat \pi \pm 1.96 \times SE \) where \(\hat\pi\) is the sample proportion, and SE, the standard error, is the standard deviation divided by the square-root of the sample size.

(This post relates to a Shiny app that explores the issue interactively.)

The standard deviation of a proportion \(\pi\) is calculated as \(\sqrt{\pi (1 – \pi)}\). This standard deviation is at its maximum value when the proportion is 0.5: \(\sqrt{0.5(1-0.5)}=0.5\). Taking this maximum value, the SE is then \(0.5/\sqrt{1000}\) which, when multiplied by 1.96 (for a 95% confidence interval) gives \(1.96\times\frac{0.5}{\sqrt{1000}} = 0.031\), i.e. \(\pm 3\%\).

In other words, with a sample of 1000, a proportion of around 50% has a 95% confidence interval of ±3%: thus the pollsters’ margin of error.

So far so good. Does this mean that Labour’s projected vote in the latest poll is 7% ± 3%, i.e., 4% to 10%? No, because if the estimated proportion is 0.07, the standard deviation (and thus the standard error) is lower: \(\sqrt{0.07 \times 0.93} = 0.255\) rather than 0.5. Thus the confidence interval is \( 0.07 \pm 1.96 \times \frac{0.255}{\sqrt{1000}} = 0.07 \pm 0.0158 \) Thus, for proportions farther from 0.5, the real confidence interval or margin of error is less than \(\pm 3\%\). Another way of thinking of it is that the pollsters’ \(\pm 3\%\) is the maximum margin of error.

OK. But apparently the \(\sqrt{\pi (1-\pi)}\) formula isn’t final. Agresti and Coull (1998), for instance, point out that it performs poorly as the proportion approaches either 0.0 or 1.0. They suggest an “add two successes and two failures” adjustment to the standard error formula, replacing \(p = \frac{N_{yes}}{N_{total}}\) with \(\tilde{p} = \frac{N_{yes} + 2}{N_{total} + 4}\). This may seem like a very odd thing to do, but as I demonstrate by simulation in Halpin (2011, section 5), it performs distinctly better than the traditional measure as p approches either extreme.

Thus the SE for p=0.07 should be as follows:

• \(\tilde{p} = \frac{70+2}{1000+4} = 0.0717\)
• \(\sigma = \sqrt{\tilde{p}(1-\tilde{p})} = 0.258\)
• \(SE = \frac{\sigma}{\sqrt{1000}} = 0.00816\)
• \(p \pm 1.96\times 0.00816\)
• \(p \pm 0.160\)

But there’s yet another way of looking at it. These confidence intervals are symmetric, like confidence intervals around a mean, but the range of p is closed, in the 0:1 interval. In fact, the intervals can include impossible proportions, such as negative values, or values above 100%. In reality if you are already close to zero it is harder to go down than to go up (and impossible to go outside the 0%–100% range), so symmetric intervals seem inappropriate. Maybe we should work on a log scale? In fact, if we work on a log-odds scale it’s even better. We can get a confidence interval on a log-odds scale by running a logistic regression with no explanatory variables. Using the following Stata code:

set obs 1000
gen x = _n<=70
logit x

we get this result:

. set obs 1000
number of observations (_N) was 0, now 1,000

. gen x = _n<=70

. logit x

Iteration 0:   log likelihood = -253.63895  
Iteration 1:   log likelihood = -253.63895  (backed up)

Logistic regression                 Number of obs     =      1,000
                                    LR chi2(0)        =       0.00
                                    Prob > chi2       =          .
Log likelihood = -253.63895         Pseudo R2         =     0.0000

------------------------------------------------------------------
      x |    Coef.  Std. Err.     z   P>|z|    [95% Conf. Interval]
--------+----------------------------------------------------------
  _cons |-2.586689  .1239394  -20.87  0.000   -2.829606   -2.343773
-------------------------------------------------------------------

That is a symmetric interval around the log-odds, -2.829606 to -2.343773. We can get Stata to give us the asymmetric interval around the odds like so:

. logit, or

Logistic regression                 Number of obs     =      1,000
                                    LR chi2(0)        =       0.00
                                    Prob > chi2       =          .
Log likelihood = -253.63895         Pseudo R2         =     0.0000

------------------------------------------------------------------
      x | Odds Ratio Std. Err.   z  P>|z|     [95% Conf. Interval]
--------+---------------------------------------------------------
  _cons | .0752688  .0093288 -20.87  0.000    .0590361    .0959649
------------------------------------------------------------------

But people only understand odds if they’re into betting! We don’t want to bet on an election! Luckily, it’s easy to transform these back: \( p = \frac{\mathrm{odds}}{1+\mathrm{odds}} \) which gives us an interval of 0.0557 to 0.0876, which is asymmetric around 0.07. Since logistic regression assumes a binomial distribution, we can consider this as a binomal confidence interval.

To explore this issue interactively, see my online Shiny app that compares the four intervals for different proportions and sample sizes.