{"id":560,"date":"2018-12-07T23:21:46","date_gmt":"2018-12-07T23:21:46","guid":{"rendered":"http:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/?p=560"},"modified":"2019-02-16T00:27:08","modified_gmt":"2019-02-16T00:27:08","slug":"simple-parallelism-in-stata","status":"publish","type":"post","link":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/?p=560","title":{"rendered":"Simple parallelism in Stata"},"content":{"rendered":"
\n

Speeding up embarrassingly parallel work in Stata<\/h2>\n
\n

Stata-MP enables parallel processing, but this really kicks in only in estimation commands that are programmed to exploit it. A lot of the time we may have very simply repetitive tasks for Stata to do, that could be packaged for parallel operation very easily. This sort of work is often called embarrassingly parallel, and isn’t automatically catered for by Stata-MP.<\/p>\n

The simplest situation is where the same work needs to be done multiple times, with some different details each time but no dependence whatsoever on the other parts. For instance we may have a loop where the work in the body takes a long time but doesn’t depend on the results of the other runs.<\/p>\n

I came across this situation the other day: I have data created by running a simulation with different values of two key parameters, and I wanted to create graphs for each combination of the parameters (PNGs rather than Stata graphs, but that’s just a detail). So I wrote code like the following:
\n<\/p>\n

 <\/p>\n

\n
forvalues i = 0\/20 {\r\n  forvalues j = 1\/8 {\r\n[... complicated commands making the graph<\/span>, depending in `i<\/span>' and `j<\/span>' ...]\r\n  }\r\n}\r\n<\/pre>\n<\/div>\n
OK, so that specified 21 times 8 = 168 graphs. But each one took about 25 seconds to generate, so it takes over an hour. On the other hand, it uses only one core, and my computer has eight. Eight? What if I ran this code eight times, varying the 1 in the second line:<\/p>\n
\n
use<\/span> simdata\r\nlocal<\/span> j<\/span> 1\r\nforvalues i = 0\/20 {\r\n[... complicated commands making the graph<\/span>, depending in `i<\/span>' and `j<\/span>' ...]\r\n}\r\n<\/pre>\n<\/div>\n
In fact, if the code looks like this and resides in smallfile.do<\/code>:\u2026<\/p>\n
\n
use<\/span> simdata\r\nlocal<\/span> j<\/span> `1'\r\nforvalues i = 0\/20 {\r\n[...]\r\n}\r\n<\/pre>\n<\/div>\n
\u2026I can run it from within Stata as follows (varying the 1 from 1 to 8):<\/p>\n
\n
!stata -b do<\/span> smallfile.do 1 &\r\n<\/pre>\n<\/div>\n
Under Unix, the “&” at the end makes this work in the background, as long as the “!stata -b do \u2026” command is issued in an interactive Stata session.<\/p>\n
There is a slight problem: I can’t run files with the same name simultaneously in the same folder, without Stata fighting with itself about writing to the log files. One strategy I’ve used before is to make each command work in its own subfolder: that’s good but requires a bit more prep. What I did yesterday was to copy the smallfile.do<\/code> to temporary files:<\/p>\n
\n
forvalues x = 1\/8 {\r\n  tempname<\/span> aux`<\/span>x<\/span>'<\/span>\r\n  !cp -p smallfile.do `aux`x<\/span>''.do\r\n  !stata -b do<\/span> `aux`x<\/span>''.do `x<\/span>' &\r\n}\r\n<\/pre>\n<\/div>\n
This creates files named like __000001.do<\/code>, and Stata logs to files like __000001.log<\/code>. Since the outputs (PNG-files) are named depending on the `x'<\/code> value, they don’t clash.<\/p>\n
Finally, I realised that the “!” or shell<\/code> command in Stata will run in the background (asychronously, in parallel) only if it is invoked in an interactive Stata session. If the main file is run in batch mode, the sub-tasks are executed sequentially. So I tried the winexec<\/code> command, which according to the documentation is intended explicitly for calling graphical programs like a web browser, while in interactive sessions. But it works:<\/p>\n
\n
forvalues x = 1\/8 {\r\n  tempname<\/span> aux`<\/span>x<\/span>'<\/span>\r\n  !cp -p smallfile.do `aux`x<\/span>''.do\r\n  winexec stata -b do<\/span> `aux`x<\/span>''.do `x<\/span>'\r\n}\r\n<\/pre>\n<\/div>\n
winexec<\/code>, without an &<\/code>, will run the subtasks in parallel, even from batch mode!<\/p>\n<\/div>\n<\/div>\n
<\/p>\n
<\/p>\n
\n
Summary<\/h2>\n
\n
This strategy depends on the subtasks being independent: while they all use the same .dta<\/code> file, they don’t depend on each other, and they produce output that is named independently and doesn’t clash. The smallfile.do<\/code> takes a command line parameter, so it operates differently for each run. It uses this parameter to name the outputs (e.g., gr_`x'_`1'.png<\/code>). Copying the smallfile.do<\/code> to a temporary file (or functionally equivalently, to a run-specific folder) means that the parallel Stata tasks don’t clash with each other. The temporary files should be handled more cleanly (check they don’t exist first, delete them after use, etc), but are easy to remove afterwards.<\/p>\n
This example creates lots of graphs. We could equally generate new data files, and use a simple append to pull them together afterwards:<\/p>\n
\n
clear<\/span>\r\nforvalues x = 1\/8 {\r\n  append<\/span> using<\/span> smalldata`x<\/span>'\r\n}\r\n<\/pre>\n<\/div>\n
It’s a good speedup: I get my PNG files generated at 16\/min rather than 2.5\/min, so it gets done in about 10 minutes rather than about an hour. Easy to code and saves time. And it makes my CPU monitor look like this:<\/p>\n
\n
\"Over-active<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Speeding up embarrassingly parallel work in Stata Stata-MP enables parallel processing, but this really kicks in only in estimation commands that are programmed to exploit it. A lot of the time we may have very simply repetitive tasks for Stata to do, that could be packaged for parallel operation very easily. This sort of work … Continue reading Simple parallelism in Stata<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/560"}],"collection":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=560"}],"version-history":[{"count":7,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/560\/revisions"}],"predecessor-version":[{"id":569,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/560\/revisions\/569"}],"wp:attachment":[{"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=560"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=560"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/teaching.sociology.ul.ie\/bhalpin\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=560"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}