Walking Randomly

I recently had the opportunity to appear on Jousef Murad’s podcast Deep Dive. My appearance was sponsored by my employer, MathWorks, but I was given free reign to talk about pretty much whatever I wanted. So I chose to talk about Research Software Engineering and the techniques from software engineering that I think every scientist and engineer should employ (e.g. Version Control!). MATLAB and MathWorks is mentioned a bit but the aim of the podcast was that it would be of interest to anyone who has to do technical computing in any programming language.

Check it out and let me know what you think.

WalkingRandomly is coming up to its 15th Anniversary!  When I started blogging in 2007, I never would have believed that this little corner of the web would still be going in 2022!

As you might have noticed, posting frequency has waned over recent years which is partly to do with Real Life taking over somewhat but also because my professional life has become rather more hectic over the years. When I started posting here, I was a junior research computing specialist and when I left academia in  2019 I was a head of Research Computing for a major UK University!  It was a wild ride.

In 2019 I joined the commercial world with The Numerical Algorithms Group where I spent a lovely couple of years immersing myself in commercial cloud, HPC and numerical computing. 

A new beginning

At the beginning of 2021, in the deepest depths of the pandemic, I was offered the opportunity to join MathWorks, The Makers of MATLAB, to work with academia as a ‘Customer Success Engineer’.  Broadly speaking, I help researchers and educators solve their problems with MATLAB.

Things have been going pretty well and earlier this month they gave me my own blog on MathWorks Central.  Little old me right next to the original developer of MATLAB, Cleve Moler, in the gallery of MathWorks bloggers. 

The new blog is called simply The MATLAB blog but as you might expect from someone who has spent 15 years writing about many aspects of scientific computing, there’s going to be a lot more than just MATLAB on there.  Expect to see posts on subjects such as HPC, Research Software Engineering, C++, Python, GPUs and maybe even a little Fortran!

Here’s a few sample posts from the last few weeks

• How to make a GPU version of this MATLAB program by changing two lines
• Exploring the MATLAB beta for Native Apple Silicon
• Finding if all elements of a matrix are finite, fast!

www.walkingrandomly.com won’t be going away and I’ll still post here from time to time but, for the forseeable future at least, most of my output will be on The MATLAB blog.  

It only took me 15 years but I finally figured out how to get paid for blogging :)

See you around kid. 

Audiences can be brutal

I still have nightmares about the first talk I ever gave as a PhD student. I was not a strong presenter, my grasp of the subject matter was still very tenuous and I was as nervous as hell. Six months or so into my studentship, I was to give a survey of the field I was studying to a bunch of very experienced researchers.  I spent weeks preparing…practicing…honing my slides…hoping that it would all be good enough.

The audience was not kind to me! Even though it was only a small group of around 12 people, they were brutal! I felt like they leaped upon every mistake I made, relished in pointing out every misunderstanding I had and all-round gave me a very hard time.  I had nothing like the robustness I have now and very nearly quit my PhD the very next day. I can only thank my office mates and enough beer to kill a pony for collectively talking me out of quitting.

I remember stopping three quarters of the way through saying ‘That’s all I want to say on the subject’ only for one of the senior members of the audience to point out that ‘You have not talked about all the topics you promised’.  He made me go back to the slide that said something like ‘Things I will talk about’ or ‘Agenda’ or whatever else I called the stupid thing and say ‘Look….you’ve not mentioned points X,Y and Z’ [1].

Everyone agreed and so my torture continued for another 15 minutes or so.

Practice makes you tougher

Since that horrible day, I have given hundreds of talks to audiences that range in size from 5 up to 300+ and this amount of practice has very much changed how I view these events.  I always enjoy them…always!  Even when they go badly!

In the worst case scenario, the most that can happen is that I get given a mildly bad time for an hour or so of my life but I know I’ll get over it. I’ve gotten over it before. No big deal! Academic presentations on topics such as research computing rarely lead to life threatening outcomes.

But what if it was recorded?!

Anyone who has worked with me for an appreciable amount of time will know of my pathological fear of having one of my talks recorded. Point a camera at me and the confident, experienced speaker vanishes and is replaced by someone much closer to the terrified PhD student of my youth.

I struggle to put into words what I’m so afraid of but I wonder if it ultimately comes down to the fact that if that PhD talk had been recorded and put online, I would never have been able to get away from it. My humiliation would be there for all to see…forever.

JuliaCon 2018 and Rise of the Research Software Engineer

When the organizers of JuliaCon 2018 invited me to be a keynote speaker on the topic of Research Software Engineering, my answer was an enthusiastic ‘Yes’. As soon as I learned that they would be live streaming and recording all talks, however, my enthusiasm was greatly dampened.

‘Would you mind if my talk wasn’t live streamed and recorded’ I asked them.  ‘Sure, no problem’ was the answer….

Problem averted. No need to face my fears this week!

A fellow delegate of the conference pointed out to me that my talk would be the only one that wouldn’t be on the live stream. That would look weird and not in a good way.

‘Can I just be live streamed but not recorded’ I asked the organisers.  ‘Sure, no problem’ [2] was the reply….

Later on the technician told me that I could have it recorded but it would be instantly hidden from the world until I had watched it and agreed it wasn’t too terrible.  Maybe this would be a nice first step in my record-a-talk-a-phobia therapy he suggested.

So…on I went and it turned out not to be as terrible as I had imagined it might be.  So we published it. I learned that I say ‘err’ and ‘um’ a lot [3] which I find a little embarrassing but perhaps now that I know I have that problem, it’s something I can work on.

Rise of the Research Software Engineer

Anyway, here’s the video of the talk. It’s about some of the history of The Research Software Engineering movement and how I worked with some awesome people at The University of Sheffield to create a RSE group. If you are the computer-person in your research group who likes software more than papers, you may be one of us. Come join the tribe!

Slide deck at mikecroucher.github.io/juliacon2018/

Feel free to talk to me on twitter about it: @walkingrandomly

Thanks to the infinitely patient and wonderful organisers of JuliaCon 2018 for the opportunity to beat one of my long standing fears.

Footnotes

[1] Pro-Tip: Never do one of these ‘Agenda’ slides…give yourself leeway to alter the course of your presentation midway through depending on how well it is going.

[2] So patient! Such a lovely team!

[3] Like A LOT! My mum watched the video and said ‘No idea what you were talking about but OMG can you cut out the ummms and ahhs’

Taps microphone: ‘Is this still on?’

I’ve been blogging on here for over 10 years and this article marks the end of the largest gap in posting that I’ve ever done — almost 6 months!  A couple of people have asked me if I’ve given up on WalkingRandomly and the answer is an emphatic ‘No’….I’ve just been extremely busy elsewhere.

Sheffield Research Software Engineering

The primary use of my time has been working with fellow RSE Fellow, Paul Richmond, to set up and run The University of Sheffield’s Research Software Engineering group.  There’s now 8 of us in total with the promise of more on the horizon.

The group has a blog over at http://rse.shef.ac.uk/blog/ and a twitter feed at https://twitter.com/RSE_Sheffield

WalkingRandomly

I’ve not given up on blogging here and there will be more in the future.

I’ve worked with computers for a long time. Decades in fact, and yet I still routinely make the same rookie mistake when discussing how long a computer-related job is going to take. Programming, sysadmin, installing a game….whatever….things almost always take much longer than I expect them to. This is true even when I take the previous statement into account.

This morning, for example, I am supposed to be on annual leave but I needed to set up a license server for a new product that a colleague of mine has just  bought. The plan was ‘Get up really early, take the dog out for morning walk and get this work done before my wife is even out of bed.’ I’d then make breakfast in bed and be the model geek-husband.

I figured it would take about 5 minutes….I’ve administered dozens of network license servers for thousands of users. Nothing phases me in this area…..I am supremely confident.  Since I know that things take longer than expected, I gave myself an hour to do this 5 minute job and set the alarm clock. This morning, I woke up before the alarm and and ended up with a whole 2 hours to do this 5 minute job.

How prepared am I?!

4 hours later, I still can’t get the chuffing thing to work and I’m wondering when on Earth I’ll ever learn…..

I sometimes give a talk on basic research software engineering called ‘Is your research correct?’ (slides here). Near the beginning of this talk I refer to what I’ve modestly named ‘Croucher’s Law’

CROUCHER’S LAW
I CAN BE AN IDIOT AND WILL MAKE MISTAKES.

Croucher’s law has a corollary:

YOU ARE NO DIFFERENT!

The idea is that once you accept this aspect of yourself, you can start to adopt working practices to mitigate against it. In the context of programming, it includes things such as automation, version control, adopting testing and so on.

For me, this isn’t just a law for programming — it’s a law that can be applied to every aspect of life. Unlike my parents, for example, I automate the payment of my bills by using direct debit because I know I’ll eventually forget to pay something otherwise.

The genesis of Croucher’s law demonstrate’s its truth. While sat in a talk given by Jos Martin of The Mathworks, he suddenly stopped and said ‘Mike. We need to talk about Croucher’s law!’ before moving to his next slide which had the title ‘Martin’s Law’. It was very similar to ‘mine’ and it turns out that I had seen his talk years before and had subconsciously ripped him off!

The fact that I had forgotten this demonstrates to me that Croucher’s law is the stronger result :)

Other relevant posts from WalkingRandomly

• On failure – I fail all the time…and that’s OK.
• Is your research software correct? 

While waiting for the rain to stop before heading home, I started messing around with the heart equation described in an old WalkingRandomly post. Playing code golf with myself, I worked to get the code tweetable. In Python:

from pylab import *
x=r_[-2:2:0.001]
show(plot((sqrt(cos(x))*cos(200*x)+sqrt(abs(x))-0.7)*(4-x*x)**0.01)) pic.twitter.com/gbOTbYSaIG

— Mike Croucher (@walkingrandomly) February 8, 2016

In R:

x=seq(-2,2,0.001)
y=Re((sqrt(cos(x))*cos(200*x)+sqrt(abs(x))-0.7)*(4-x*x)^0.01)
plot(x,y)#rstats pic.twitter.com/trpgEnNna4

— Mike Croucher (@walkingrandomly) February 8, 2016

I liked the look of the default plot in R so animated it by turning 200 into a parameter that ranged from 1 to 200. The result was this animation:

Finding this animation based on previous tweets oddly mesmerising #rstats pic.twitter.com/e3q6lZqWcP

— Mike Croucher (@walkingrandomly) February 8, 2016

The code for the above isn’t quite tweetable:

options(warn=-1)
for(num in seq(1,200,1))
{
    filename = paste("rplot" ,sprintf("%03d", num),'.jpg',sep='')
    jpeg(filename)
    x=seq(-2,2,0.001)
    y=Re((sqrt(cos(x))*cos(num*x)+sqrt(abs(x))-0.7)*(4-x*x)^0.01)
    plot(x,y,axes=FALSE,ann=FALSE)
    dev.off()
}

This produces a lot of .jpg files which I turned into the animated gif with ImageMagick:

convert -delay 12 -layers OptimizeTransparency -colors 8 -loop 0 *.jpg animated.gif 

“It’s OK for you, you never really fail at anything.” my friend accused me as I was trying to convince her to apply for a hotly-contested promotion.

This left me a little stunned because I fail all the time. Not just occasionally, not just with small-time stuff but ALL. THE. TIME! Sometimes I fail so hard that the sense of loss, of failure that I feel is almost physical.

I wanted to set the record straight…give an idea of how I fail..in the large and the small. So I told her of some of the huge bucket of fail that is me. Bear in mind that this is by no means a complete list…this is just some of the stuff that I feel comfortable talking about!

Career

Before I started my PhD, I attempted to complete a PGCE (Qualification required to become a school teacher in the UK). I crashed and burned halfway through the course and dropped out. I don’t remember a time when I was more miserable in my professional life! I still find it difficult to think about that year and even more difficult to talk about it.

After my PhD, I applied for dozens of jobs, both academic and commercial, before I landed the job that changed my life at The University of Manchester. During my time at Manchester I failed to be promoted several times before I finally achieved it.

Now I’m at The University of Sheffield and things are better than they’ve ever been! I have a truly wonderful job! I tried to get here several years ago, however, and..you guessed it…I failed (Worst. Interview. Ever!)

Computing

To program, to sysadmin is to fail…often. I try something, it fails. I try something else, it fails. On and on it goes until success is mine. Sometimes I never succeed. Researchers often come to me asking if I can speed up their code and sometimes, after several days of intense effort, I report back with a resigned ‘No. Sorry! – Here’s a list of things that don’t work’.

When I write code, I consider failure to be so likely that I assume that I definitely will fail (See Croucher’s law in this talk). I engineer my working practices to get past the inevitable failures as quickly as possible.

Physical failures

At the beginning of this article I said the feeling of failure can feel almost physical. Sometimes it IS physical such as the time I fought in the TAGB Semi-Contact Taekwondo World Championships (long time ago!) and had my ass neatly handed to me by the Canadian national champion! I subsequently failed to eat properly for 5 days straight thanks to the hardest left hook I’ve ever experienced in my life (semi-contact kinda goes out of the window at that level).

I gave everything I had in that fight and lost! When I tell people the story, however, I don’t dwell on the fact that I lost. I concentrate on the fact that I had the chops to be there. I remember the bit where I kicked him off his feet, I remember being knocked to the ground and feeling afraid of getting back up, of going back in but doing it anyway. I remember that one of the 4 judges awarded the fight to me so I couldn’t have done THAT badly. I remember my opponent hugging me after the fight and telling me that it had ‘been awesome’. I’m proud of that fight!

My Taekwondo days are far behind me now and, these days, I get my physical kicks by lifting weights. I concentrate on the major lifts such as Squat, Deadlift, Press and Bench Press. The aim is to always lift more and I still consider myself a novice. Most of the time, when I increase the weight, I fail. Failure is much more likely than success since I am working at the limit of my strength. The successes feel amazing though…to be able to say I’m stronger than I’ve ever been and to have numerical evidence…good times!

Future failures

I currently have several projects on the go – some personal, some professional — some large and some small. I fully expect to succeed with some of them and fail miserably in others. Some of the failures are going to hurt! A lot!

I’m waiting on the outcome of a major endeavour for me (EPSRC Research Software Engineering Fellowship) — something for which failure is highly likely since the competition is so fierce. I’ll not deny that I’m afraid but I’m also excited.

What has failure taught me?

When I reflect on my many failures, only a tiny subset of which are given above, I note that there are some common threads.

Success (and failure) is often half-chance I always suspected that this was the case and now that Neil Lawrence has done his NIPS experiment, we have data to back it up.

“whatever you do, don’t congratulate yourself too much or berate yourself either – your choices are half chance, so are everybody else’s.” Don’t forget the sunscreen

I fail a lot! Over time this has taught me that, although it hurts, the pain is rarely permanent. Over time, you get desensitised to it. Not so much that you become careless — you still prefer to avoid it — but enough to stop being afraid all the time. I’ve failed before, it’s not so bad!

My sense of identity is dominated more by my successes than my failures. I don’t dwell on my failures too much. I might allow myself to wallow in self-pity for a short-time following something big but at some point I’ll channel my inner-Pratchett “If failure had no penalty success would not be a prize”, see what I can learn from the failure and move on. My successes, on the other hand, stay with me for a long time! I don’t regret my failures but I relish my successes. This is probably why some people, such as my friend, think I don’t fail all that much. I rarely tell the stories unless I can spin them to make my failures look like a success.

People respect you more when you own up to failure. In my career, I’ve worked with a huge number of risk-averse people who are more interested in ensuring that no one thinks a failure is their fault than they are in fixing the failure. I try to put my hand up and say ‘My bad! Really sorry! I’m working on fixing it. Feel free to deliver me a kicking if you think I deserve it.’

I find that, rather than delivering said-kicking, most people choose to appear alongside me, shovel in hand and we dig ourselves out of the hole together.

I succeed a lot! I put myself ‘out there’ a lot. On this blog, in my job, with my friends. I try things that other people might shy away from, I take risks. I fail.

I also succeed! The net result has been a wonderful wife, a great group of friends, fantastic job, good fitness…great life.

Resources

A CV of failure – The Nature article that inspired me to write this blog post

I’m a geek and love hanging out with other geeks. There are many different flavours of geek: gym geeks, food geeks, beer geeks, math geeks, science geeks, greyhound geeks…you get the idea! There’s a quote attributed to Simon Pegg that sums up geekdom perfectly for me:

“Being a geek is all about being honest about what you enjoy and not being afraid to demonstrate that affection. It means never having to play it cool about how much you like something. It’s basically a license to proudly emote on a somewhat childish level rather than behave like a supposed adult. Being a geek is extremely liberating.” Simon Pegg

Pure, unbridled enthusiasm is extremely infectious. Since I work at a University, I am extremely fortunate in that I often find myself on the receiving end of a combination of outrageous optimism and advanced-geek enthusiasm – a powerful combination that results in people changing the world.

Long ago, I discovered that you could learn a lot about the world simply by sharing a coffee or beer with a geek or two and finding out what they think is cool. Keep a note of any google-able phrases they come up with and follow up later. Much more fun, and probably more efficient, than sitting through dozens of conference talks.

Here are a few computer-based things I learned from such conversations this week:

• Microsoft Azure have built a platform for teaching Machine Learning and they have some open datasets too.
• Microsoft also have a holographic operating system!
• zsh is cooler than bash
• ESC and DOT brings up the argument of the previous command in Bash
• Ansible exists and I’ll probably find it useful
• Jugs is a Task Based Parallelisation framework for Python

With apologies to Ridley Scott and Rutger Hauer:

I’ve…seen things you people wouldn’t believe.

Compute kernels on fire while looking over the shoulder of Brian. I watched C-code glitter in the dark as it flowed from the automatic generator with no test-rig in sight. There was no version control, no back-up so all those…results…will be lost in time, like tears in rain.

Time to debug.