The unreasonable effectiveness of getting your code to run on someone else’s machine

January 12th, 2017 | Categories: programming, RSE, Scientific Software | Tags:

If you are a researcher and are currently writing scripts or developing code then I have a suggestion for you. If you haven’t done it already, get yourself a willing volunteer and send them your code/analysis/simulation/voodoo and ask them to run it on their machine to see what happens. Bonus points are awarded for choosing someone who uses a different operating system from you!

This simple act is one of the things I recommend in my talk Is Your Research Software Correct and it can often help improve both code and workflow.

It quickly exposes patterns that are not good practice. For example, scattered references to ‘/home/walkingrandomly/mydata.dat’ suddenly don’t seem like a great idea when your code buddy is running windows. The ‘minimal tweaking’ required to move your analysis from your machine to theirs starts to feel a lot less minimal as you get to the bottom of the second page of instructions.

Crashy McCrashFace

When I start working with someone new, the first thing I ask them to do is to provide access to their code and simple script called runme or similar that will build and run their code and spit out an answer that we agree is OK. Many projects stumble at this hurdle! Perhaps my compiler is different to theirs and objects to their abuse (or otherwise) of the standards or maybe they’ve forgotten to include vital dependencies or input data.

Email ping-pong ensues as we attempt to get the latest version…zip files with names like PhD_code_ver1b_ForMike_withdata_fixed.zip get thrown about while everyone wonders where Bob is because he totally got it working on Windows back in 2009.

git clone

‘Hey Mike, just clone the git repo and run the test suite. It should be fine because the latest continuous integration run didn’t throw up any issues. The benchmark code and data we’d like you to optimise is in the benchmarks folder along with the timings and results from our most recent tests. Ignore the papers folder, that just reproduces all of the results from our recent papers and links to Zenodo DOIs’

‘…………’

‘Are you OK Mike?’

‘I’m…..fine. Just have something in my eye’

 

  1. Ian
    January 12th, 2017 at 16:54
    Reply | Quote | #1

    So the software in our latest paper *almost* fits under that last git clone bit. You can git clone our software. 7 of its 8 dependencies install painlessly with the setup script, tested on minimal install of two Linux flavours and OSX (the one that doesn’t – cython, isn’t hard to install manually). All commits to the repository trigger a run of the test suite on a CI server.

    The thing that really doesn’t is the “papers” folder bit. Because the results from the paper have to get the data from the public data repository of raw data, to a processed state ready to be used. This takes 100s, perhaps even 1000s of hours of CPU time and many tools (probably 50 separate binaries). In addition to the tools is the infrastructure that glues it all together and produces the plots at the end. This can be installed with package managers, but comes with over 100 dependencies of its own, not all of which are well behaved. And these all have there own dependencies. Because of the CPU time the whole thing needs to run on a cluster. In my experience, no two clusters are the same, and even where things are compatible, their is quite a lot of intricate configuration to do.

    Titus Brown has some good thoughts on this: http://ivory.idyll.org/blog/2017-pof-software-archivability.html

    We worked very hard at making the software itself reliable, installable and easy to use. But for the analysis pipeline? We just throw up our hands and said – look its there if you want it (it does have a zenodo), but there is no way we can provide support on it.

  2. January 12th, 2017 at 17:45
    Reply | Quote | #2

    I think method on the bottom needs to be more actively enforced. The current system indirectly punishes you for producing reproducible software with CI tests by not caring about this in the review process. People who disregard sustainable software development can more easily force papers through the system and pass through the publish or perish world. I find myself facing major backlash for any time spent on making sure research software is correct/reproducible (since it lowers my paper productivity), when in reality it’s the true way to do science. I think that the new generation has to simply regard any result from any script/software that does not have testing and ease of reproducibility as unscientific and not fit for publication/grants. Otherwise the system will continue to reward unscientific behavior and results.

  3. ‘Dragon’ David McKee
    January 14th, 2017 at 03:53
    Reply | Quote | #3

    I’ve enjoyed using Docker as a build-management tool – it tends to enforce (code-as-)documentation of how you went about configuring your environment (whether this’d stand up to being looked at a decade later is another matter), requires all the files to be present and correct, etc.