{"id":580,"date":"2019-10-06T16:45:21","date_gmt":"2019-10-06T16:45:21","guid":{"rendered":"http:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/?p=580"},"modified":"2019-10-08T20:08:55","modified_gmt":"2019-10-08T20:08:55","slug":"webscraping-wikipedia-with-emacs","status":"publish","type":"post","link":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/?p=580","title":{"rendered":"Webscraping Wikipedia with Emacs"},"content":{"rendered":"
\n

Idle hands<\/h2>\n
\n

\nFor the want of something better to do (okay, because procrastination), a pass at webscraping Wikipedia. For fun. I’m going to use it’s “Random Page” to sample pages, and then extract the edit history (looking at how often edited, and when). Let’s say we’re interested in getting an idea of the distribution of interest in editing pages.\n<\/p>\n

See update: tidier code<\/a>.<\/p>\n

\nI’m going to use Emacs lisp for the web scraping.\n<\/p>\n

\nOK, Wikipedia links to a random page from the Random Page<\/a> link in the lefthand menu. This is a URL:\n<\/p>\n

\r\nhttps:\/\/en.wikipedia.org\/wiki\/Special:Random\r\n<\/pre>\n
\nHow random is this page? See https:\/\/en.wikipedia.org\/wiki\/Wikipedia:FAQ\/Technical#random<\/a>\n<\/p>\n<\/div>\n<\/div>\n
<\/p>\n
\n
Scraping code<\/h2>\n
\n
\nThe little function below uses url-retrieve-synchronously<\/code> to retrieve the page, into an emacs buffer, and then uses regular-expression search to extract the title of the page.\n<\/p>\n
\n
(defun<\/span> getrandompage<\/span> ()\r\n  (set-buffer (url-retrieve-synchronously \"https:\/\/en.wikipedia.org\/wiki\/Special:Random\"<\/span>))\r\n  (re-search-forward \"<title><\/span>\\\\<\/span>(<\/span>.+<\/span>\\\\<\/span>)<\/span> - Wikipedia<\/title>\"<\/span> nil t)\r\n  (match-string 1))\r\n<\/pre>\n<\/div>\n
\nA page’s edit-history is available at this URL\n<\/p>\n
\r\nhttps:\/\/en.wikipedia.org\/w\/index.php?title=PAGENAME&offset=&limit=5000&action=history\r\n<\/pre>\n
\nNormally, there are links for varying numbers edits shown per page, up to a max of 500, but I’ve put 5000 in as the limit, and it seems to work.\n<\/p>\n
\nThe edits are in a HTML list, and can be picked up with a regexp of the following form:\n<\/p>\n
\r\n\"<li data-mw-revid=.+<\/li>\"\r\n\r\n<\/pre>\n
\nThe following function takes a page name, and returns a list of the edit entries, with the name of the page at the top of the list.\n<\/p>\n
\n
(defvar<\/span> history-url<\/span> \r\n  \"https:\/\/en.wikipedia.org\/w\/index.php?title=%s&offset=&limit=5000&action=history\"<\/span>)\r\n(defun<\/span> gethistory<\/span> (page)\r\n  (let<\/span> (results)\r\n    (save-mark-and-excursion<\/span>\r\n      (set-buffer (url-retrieve-synchronously\r\n                   (format history-url page)))\r\n      (while<\/span> (re-search-forward \"<li data-mw-revid=.+<\/li>\"<\/span> nil t)\r\n        (push<\/span> (match-string 0) results))\r\n      (cons page results))))\r\n\r\n<\/pre>\n<\/div>\n
\nThere is lots of information in the edit entry (date, user, size of resulting file, change in file size, comment, on top of URLs to compare versions). I just want dates, which I capture with this regexp:\n<\/p>\n
\r\n\"mw-changeslist-date[^>]+?>\\\\([^<]+\\\\)\"\r\n\r\n<\/pre>\n
\nThe following block puts this all together, writing the results to a tab-delimited file for further processing. Putting everything in a dotimes<\/code> loop is crude and fragile (any error and the whole batch of data is lost; a more professional approach would catch and deal with errors) but it’s simple. It also ties up the Emacs process, so it is probably best done in a separate process (or --batch<\/code> style).\n<\/p>\n
\n
\r\n(defun<\/span> parse-wiki-edit<\/span> (slug)\r\n  (string-match \"mw-changeslist-date[<\/span>^<\/span>>]+?><\/span>\\\\<\/span>(<\/span>[<\/span>^<\/span><]+<\/span>\\\\<\/span>)<\/span>\"<\/span> slug)\r\n  (match-string 1 slug))\r\n\r\n;; <\/span>Do it 1000 times, store in a tab-delimited CSV<\/span>\r\n(with-temp-buffer<\/span>\r\n  (insert \"pagetitle\\tedittime\\n\"<\/span>)\r\n  (dotimes<\/span> (i 1000)\r\n    (setq<\/span> page (save-excursion<\/span> (gethistory (getrandompage))))\r\n    (setq<\/span> title (car page))\r\n    (mapcar (lambda<\/span> (x) (insert\r\n                         (format\r\n                          \"%s\\t%s\\n\"<\/span>\r\n                          title\r\n                          (format-time-string \"%Y-%m-%d-%T\"<\/span>\r\n                                              (encode-time (parse-time-string\r\n                                                            (parse-wiki-edit x)))))))\r\n            (cdr page)))\r\n  (write-file \"\/tmp\/wiki.csv\"<\/span>))\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Some results<\/h2>\n
\n
\nI switch to Stata to analyse the resulting file (you can use anything you like, as long as it’s not Excel<\/a>. We see an average of about 100 edits per page, but the range is from 1 to almost 2750:\n<\/p>\n
\r\n Variable |     Obs      Mean  Std. Dev.    Min     Max\r\n----------+--------------------------------------------\r\n        N |     996  103.1697  265.6243       1    2748\r\n<\/pre>\n
\n
\"histogN.png\"<\/p>\n<\/div>\n
\nAn extract from the page frequency table gives a flavour of the eclecticism of Wikipedia:\n<\/p>\n
\r\n. tab pagetitle, sort\r\n\r\n                              pagetitle |   Freq. Percent    Cum.\r\n----------------------------------------+------------------------\r\n Mitt Romney 2012 presidential campaign |   2,748    2.67    2.67\r\n               Resident Evil: Afterlife |   2,466    2.40    5.07\r\n          Cartoon Network (Philippines) |   2,255    2.19    7.27\r\n                     Adamson University |   2,226    2.17    9.43\r\nWorld War II casualties of the Soviet.. |   2,176    2.12   11.55\r\n                                 Cumans |   2,056    2.00   13.55\r\n                         IndyCar Series |   1,950    1.90   15.45\r\n               Syrian Democratic Forces |   1,881    1.83   17.28\r\n                        Australian Army |   1,835    1.79   19.07\r\n                 Padmanabhaswamy Temple |   1,829    1.78   20.85\r\n              List of Cluedo characters |   1,538    1.50   22.34\r\n[ . . . ]\r\n             Podosinovets, Kirov Oblast |       1    0.00  100.00\r\n                     Stuart Fitzsimmons |       1    0.00  100.00\r\n                              Ust-Morzh |       1    0.00  100.00\r\n----------------------------------------+------------------------\r\n                                  Total | 102,757  100.00\r\n<\/pre>\n
\nHere is a picture of how long ago the earliest edit was:\n<\/p>\n
\n
\"histogD.png\"\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Idle hands For the want of something better to do (okay, because procrastination), a pass at webscraping Wikipedia. For fun. I’m going to use it’s “Random Page” to sample pages, and then extract the edit history (looking at how often edited, and when). Let’s say we’re interested in getting an idea of the distribution of … Continue reading Webscraping Wikipedia with Emacs<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/580"}],"collection":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=580"}],"version-history":[{"count":6,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/580\/revisions"}],"predecessor-version":[{"id":600,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/580\/revisions\/600"}],"wp:attachment":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=580"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=580"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=580"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}