Twitter is a goldmine of relational info. Who follows whom, who retweets whom, who replies to whom, and so on. And a lot of this data is available for analysis (though some of it is on a drip-feed). I decided to try to understand my twitter identity by looking at who the people who follow me follow.

Continue reading What sort of Tweep am I? Who do people who follow me follow? →

Dendrograms are diagrams that have a tree-like structure, and they’re often used to represent the structure of clustering in a hierarchical (agglomerative) cluster analysis. Agglomerative clustering starts from the bottom up, joining the nearest pairs of objects into clusters, and then clusters with objects and finally clusters with clusters, until eventually everything is a single cluster. The single cluster is the root, the objects are the leaves, and in between is a binary tree, where objects and clusters are combined depending on their distance from each other.

This process depends on being able to define a distance between an object and a cluster, and between pairs of clusters, and there are various ways to do this. However, some algorithms may cause the distance between an object/cluster and another cluster to change after the amalgamation of other clusters. This permits “reversals”, which are difficult or impossible to represent in a dendrogram-like structure. But clustering algorithms or “linkages” such as Ward’s are not subject to this problem.

OK, so far so good. What’s the problem? Stata’s dendrogram code is slightly buggy, and can give an error:
currently can't handle dendrogram reversals
even when you are using a linkage that is not subject to reversals. The explanation is that it is comparing distances between pairs of clusters where one must be greater than or equal to the other for the dendrogram to be drawable (otherwise it’s a reversal), and due to numeric precision is finding pairs where one is fractionally less. The correct code should test the difference in the distances is not less than a very small number (e.g., 10^-7) to take account of precision.

I have had a number of people report this error to me, in connection with my SADI Stata ado package, and have been able to reproduce it. In cooperation with some of these respondents, we have been able to get a work-around from Stata Technical Support (this bug persists into Stata 14).

The following command displays the information that Stata holds about the current clustering (apologies for the linewrapping):

. char list _dta[_clus_1_info]
_dta[_clus_1_info]: `"t hierarchical"' `"m wards"' `"d user matrix omd"' `"v id _clus_1_id"' `"v order _clus_1_ord"' `"v height _clus_1_hgt"'


A small change to this will allow the dendrogram to be drawn:

. char _dta[_clus_1_info] `"`"t hierarchical"' `"m wards"' `"d user matrix omd"' `"v id _clus_1_id"' `"v order _clus_1_ord"' `"v height _clus_1_hgt"'"'

Estimating the effect of the gender quota: The initial question

           |        gender
      dail |         f          m |     Total
-----------+----------------------+----------
      2011 |        21        145 |       166 
           |     12.65      87.35 |    100.00 
-----------+----------------------+----------
      2016 |        32        116 |       148 
           |     21.62      78.38 |    100.00 
-----------+----------------------+----------
     Total |        53        261 |       314 
           |     16.88      83.12 |    100.00

The 3% margin of error often quoted in polling has the following logic. It uses a sample size of 1000, and calculates a confidence interval as follows: \( \hat \pi \pm 1.96 \times SE \) where \(\hat\pi\) is the sample proportion, and SE, the standard error, is the standard deviation divided by the square-root of the sample size.

Continue reading Pollsters’ 3% Margin of Error →

People like the R² stat from linear regression so much that they re-invent it in places it doesn’t naturally arise, such as logistic regression. The true R² has nice clean interpretations, as the proportion of variation explained or the square of the correlation between observed and predicted values. The fake or pseudo-R² statistics are often based on relating the loglikelihood of the current model against that of the null model (intercept only) in some way. There is a good overview at UCLA.

One of the most popular pseudo-R² is McFadden’s. This is defined as 1 – LLm/LL0 where LLm is the log-likelihood of the current model, and LL0 that of the null model. This appears to have the range 0-1 though 1 will never be reached in practice.

It is well known that if we fit linear regressions by maximum-likelihood, we get exactly the same parameter estimates as if we fit by ordinary least squares. We can demonstrate this in Stata:

. sysuse auto
. reg price headroom mpg
. glm price headroom mpg

Since the ML estimation of the linear regression gives us loglikelihoods, we can calculate pseudo-R2 and true R2 for the same model. This code does it for a range of simple models with Stata’s demonstration “auto” data set:


sysuse auto, clear
glm price
local basell = e(ll)
local vars "mpg rep78 headroom trunk weight length turn displacement gear_ratio foreign"
local rhs = ""

label var mcf "McFadden Pseudo-R2"
label var r2 "R-squared"
scatter mcf r2


This generates the following graph, in which we see that there is a monotonic but non-linear relationship between the two measures. We can also see very clearly that pseudo-R2 is always substantially lower than R2. Thus it should be clear that while it emulates R2 in spirit, it doesn’t actually approximate it. So when people talk about proportion of variation explained in a logistic regression, shoot them down.

I had an interesting exchange by e-mail with Maggie Smith, the analyst at UCAS responsible for the note discussed in my previous blog entry. She tells me that the conclusion I derived from the released data (the small but significant ethnicity effect) is very much attenuated when the data is broken down by provider, and the published UCAS analysis is based on such disaggregated data. Continue reading Update on UCAS post →