Archive for the ‘Linux’ Category

April 23rd, 2019

My preferred workflow for writing technical documents these days is to write in Markdown (Or Jupyter Notebooks) and then use Pandoc to convert to PDF, Microsoft Word or whatever format is required by the end client.

While working with a markdown file recently, the pandoc conversion to PDF failed with the following error message

! Undefined control sequence.
l.117 \[ \coloneqq

This happens because Pandoc first converts the Markdown file to LaTeX which then gets compiled to PDF and the command \coloneqq isn’t included in any of the LaTeX packages that Pandoc uses by default.

The coloneqq command is in the mathtools package which, on Ubuntu, can be installed using

apt-get install texlive-latex-recommended

Once we have the package installed, we need to tell Pandoc to make use of it. The way I did this was to create a Pandoc metadata file that contained the following

---
header-includes: |
            \usepackage{mathtools}
---

I called this file pandoc_latex.yml and passed it to Pandoc as follows

pandoc --metadata-file=./pandoc_latex.yml ./input.md -o output.pdf

On one system I tried this on, I received the following error message

pandoc: unrecognized option `--metadata-file=./pandoc_latex.yml'

which suggests that the –metadata-file option is a relatively recent addition to Pandoc. I have no idea when this option was added but if this happens to you, you could try installing the latest version from https://github.com/jgm/pandoc/

I used 2.7.1.1 and it was fine so I guess anything later than this should also be OK.

January 23rd, 2019

I have been an advocate of the Windows Subsytem for Linux ever since it was released (See Bash on Windows: The scripting game just changed) since it allows me to use the best of Linux from my windows laptop.  I no longer dual boot on my personal machines and rarely need to use Linux VMs on them either thanks to this technology.  I still use full-blown Linux a lot of course but these days it tends to be only on servers and HPC systems.

I recently needed to compile and play with some code that was based on the GNU Scientific Library. Using the Ubuntu 18.04 version of the WSL this is very easy. Install the GSL with

sudo apt-get install libgsl-dev

A simple code that evaluates Dawson’s integral over a range of x values is shown below. Call this dawson.cpp

#include<iostream>
#include<vector>
#include<gsl/gsl_sf.h>

int main(){

double range = 6; // max/min values
int N = 100000; // Number of evaluations
double step = 2 * range / N;
std::vector<double> x(N);
std::vector<double> result(N);

for (int i=0;i<=N;i++){
     x[i] = -range + i*step;
     result[i] = gsl_sf_dawson(x[i]);
}


for (int i=0;i<=N;i++){
	std::cout << x[i] << "," << result[i] << std::endl;
}

return 0;
}

Compile with

g++ -std=c++11 dawson.cpp -o ./dawson -lgsl -lgslcblas -lm

Run it and generate some results

./dawson > results.txt

If we plot results.txt we get

dawson

This code is also available on GitHub: https://github.com/mikecroucher/GSL_example

August 5th, 2016

Like many people, I was excited to learn about the new Linux subsystem in Windows announced by Microsoft earlier this year (See Bash on Windows: The scripting game just changed).

Along with others, I’ve been playing with it on the Windows Insider builds but now that the Windows Anniversary Update has been released, everyone can get in on the action.

Activating the Linux Subsystem in Windows

Once you’ve updated to the Anniversary Update of Windows, here’s what you need to do.

Open settings

windows_settings

In settings, click on Update and Security

windows_update

In Update and Security, click on For developers in the left hand pane. Then click on Developer mode.

windows_for_developers

Take note of the Use developer features warning and click Yes if you are happy. Developer mode gives you greater power, and with great power comes great responsibility.

windows_develper_warning

Reboot the machine (may not be necessary here but it’s what I did).

Search for Features and click on Turn Windows features on or off

windows_features

Tick Windows Subsystem for Linux (Beta) and click OK

Screen Shot 2016-08-05 at 15.30.08

When it’s finished churning, reboot the machine.

Launch cmd.exe

Screen Shot 2016-08-05 at 15.36.14

Type bash, press enter and follow the instructions

Screen Shot 2016-08-05 at 15.37.58

The linux subsystem will be downloaded from the windows store and you’ll be asked to create a Unix username and password.

Try something linux-y

The short version of what’s available is ‘Every userland tool that’s available for Ubuntu’ with the caveat that anything requiring a GUI won’t work.

This isn’t emulation, it isn’t cygwin, it’s something else entirely. It’s very cool!

The gcc compiler isn’t installed by default so let’s fix that:

sudo apt-get install gcc

Using your favourite terminal based editor (I used vi), enter the following ‘Hello World’ code in C and call it hello.c.

/* Hello World program */

#include

int main()
{
    printf("Hello World from C\n");
    
    return(0);
}

Compile using gcc

gcc hello.c -o hello

Run the executable

./hello
Hello World from C

Now, transfer the executable to a modern Ubuntu machine (I just emailed it to myself) and run it there.

That’s right – you just wrote and compiled a C-program on a Windows machine and ran it on a Linux machine.

Now install cowsay — because you can:

sudo apt-get install cowsay
cowsay 'Hello from Windows'
 ____________________
< Hello from Windows >
 --------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Update 1:

I was challenged by @linuxlizard to do a follow up tutorial that showed how to install the scientific Python stack — Numpy, SciPy etc.

It’s all there :)

sudo apt-get install python-scipy

Screen Shot 2016-08-05 at 16.42.30

Update 2

TensorFlow on LinuxOnWindows is also easy: http://www.hanselman.com/blog/PlayingWithTensorFlowOnWindows.aspx

March 31st, 2016

Yesterday saw the biggest highlight in the technical calendar so far for me… Microsoft has brought the Linux command line to Windows.

Excited by the command line

OK, so I’ll admit it….improvements to command line tools seriously excite me! Windows 10 brought in a couple of minor improvements to the command line last year and I acted like it was a second birthday. Imagine then, the excitement I felt while watching Microsoft’s announcement of linux integration into Windows (Spin to 2:24 to see it). I could barely form sentences! When my wife came into the kitchen to see what the commotion was, all I could manage was “Bash! On Windows!….GIT! OMG! Not a VM! vi..sed..awk…gcc! OMG!’

So why so much excitement?

Shell scripts are more than mere automation, they are repositories of knowledge  – they do some task for you and also explain how that task is done. They are wonderful things that can educate, take the drudgery out of a researcher’s life and lead to more reproducible research. The problem is that Linux and Mac users speak a different scripting language (Bash) to Windows users (Windows batch or more, recently, PowerShell).

In short, a script written on a Linux or Mac machine wouldn’t run on a Windows machine unless you jumped through some hoops.

Bash just became a cross-platform scripting solution

Various solutions for running Bash scripts on Windows have existed for a while. Cygwin, for example, compiles linux tools to run on Windows which works very effectively for many situations. Additionally, the Windows version of git comes with an emulated Bash mode that’s good enough to teach the scripting lesson from Software Carpentry. Neither of these solutions are perfect, however, and to me they’ve always felt like slightly awkward patches. The resulting binaries in projects such as Cygwin are necessarily different from those used in Linux Land.

This new collaboration between Canonical and Microsoft changes the game! Now, Linux tools appear like first-class citizens in the Windows world. When you run Bash on Windows, it will be the exact same Bash that’s run on Linux. An automated research analysis developed on a Linux machine will work exactly the same way on Windows.

The same skills apply, from tablet to supercomputer

Furthermore, when we teach introductory shell scripting to researchers we will be teaching them tools that allows them to work on all operating systems and all hardware types.

The same skills will apply to Mac, Linux and Windows from tablets to High Performance Computing clusters and that’s a wonderful thing.

 

February 26th, 2015

Environment modules are widely used in the High Performance Computing (HPC) world where sysadmins need to install dozens, or maybe hundreds of potentially conflicting applications, libraries and compilers on multi-user machines. The University of Manchester’s Computational Shared Facility (CSF), for example, makes extensive use of environment modules and would be extremely difficult to run without them.

Once the sysadmin has correctly installed an application (MATLAB 2014a say) and set up the corresponding module file, making it available to your shell is as easy as doing

module load apps/binapps/matlab/R2014a

Unloading the module is just as easy

module unload apps/binapps/matlab/R2014a

On a heavily used, multi-user system environment modules are invaluable! Every user can have whatever compilers, libraries and applications they like — they just load and unload whatever they need from the huge selection supported by their ever-friendly sysadmins.

Environment modules on Ubuntu

I needed to install environment modules on a VM running Ubuntu 14.04 for my own use. I found a very nice setup guide at http://www.setuptips.com/unix/setup-environment-modules-on-ubuntu/ but it didn’t work. On attempting to compile, I got the error message

cmdModule.c:644:15: error: 'Tcl_Interp' has no member named 'errorLine'

This is a known bug in version 3.2.9c of environment modules and has a work-around.

I also found a set up guide at http://nickgeoghegan.net/linux/installing-environment-modules which had some useful advice on configuration..

Combining information from these sources, I managed to get a working install. Here are the steps I did in full for a clean Ubuntu 14.04 image

#Install the tcl development package
sudo apt-get install tcl-dev

#Make the directories where my modules and packages are going to live
sudo mkdir /opt/modules
sudo mkdir /opt/packages

#Get the source code. This was the most up to date version as of 25th Feb 2015
wget http://downloads.sourceforge.net/project/modules/Modules/modules-3.2.9/modules-3.2.9c.tar.gz

#unpack and enter source directory
tar xvzf modules-3.2.9c.tar.gz
cd modules-3.2.9

#Configure using the workaround and selecting my module folder 
CPPFLAGS="-DUSE_INTERP_ERRORLINE" ./configure --with-module-path=/opt/modules/

#make and install
make
sudo make install

#Edit the modulefiles path. Comment out all lines starting /usr so that only /opt/modules is used
sudo sed -i 's~^/usr~#/usr~' /usr/local/Modules/3.2.9/init/.modulespath

#Configure the shell to use modules
sudo tee /etc/profile.d/modules.sh > /dev/null << 'EOF'
#----------------------------------------------------------------------#
# system-wide profile.modules #
# Initialize modules for all sh-derivative shells #
#----------------------------------------------------------------------#
trap "" 1 2 3

MODULES=/usr/local/Modules/3.2.9

case "$0" in
-bash|bash|*/bash) . $MODULES/init/bash ;;
-ksh|ksh|*/ksh) . $MODULES/init/ksh ;;
-sh|sh|*/sh) . $MODULES/init/sh ;;
*) . $MODULES/init/sh ;; # default for scripts
esac

trap - 1 2 3
EOF

#Add modules to your .bashrc file
echo '#For modules' >> ~/.bashrc
echo '. /etc/profile.d/modules.sh' >> ~/.bashrc

That takes care of the basic setup but modules is pretty useless at this stage. To make it useful, you need to install some extra software and the corresponding module file.

Installing a module file for Anaconda Python 2.1
This is a really simple example of how to set up a basic module file

I downloaded and installed Anaconda Python 2.1 to /opt/packages and created a file called anaconda2.1 in /opt/modules containing the following

#%Module1.0
proc ModulesHelp { } {
global dotversion
 
puts stderr "\tAnaconda Python 2.1 providing Python 2.7.8"
}
 
module-whatis "Anaconda Python 2.1"
prepend-path PATH /opt/packages/anaconda/bin

Now, when I do the command

module avail

I get

-------------------------- /opt/modules/ ---------------------------
anaconda2.1

I can load my anaconda2.1 module with the command

module load anaconda2.1

Now, when I type python at the command prompt, I’ll be using Anaconda’s python rather than the system python. Once I’m done, I can unload with

module unload anaconda2.1

This example is so trivial it’s almost not worth it — modules really come into their own when you need to support loads of compilers and corresponding libraries. There’s an example using gcc at http://nickgeoghegan.net/linux/installing-environment-modules.

August 10th, 2012

I recently installed Ubuntu 12.04 on my laptop.  I gave Unity a chance for a few days just in case it had improved since I last used it but still found it unusable.  The following tweaks made Ubuntu usable again (for me at least).

That was pretty much it and I’m very happy with the result.  Do you use Ubuntu?  If so, are there any tweaks that you simply must make to the default setup before you feel that it’s usable for you?

July 10th, 2011

I recently maxed out my credit card in order to treat myself to a shiny new Dell XPS L720X laptop that comes complete with Intel i7 sandy bridge processor and Nvidia GeForce GT 555M. The NVidia graphics card was one of the biggest selling points for me because I wanted to do some GPU work at home and on the train using both CUDA and OpenCL.  I get asked about these technologies a lot by researchers at The University of Manchester and I wanted to beef up my experience levels.

I wanted this machine to be dual boot Windows 7 and Linux so, before I shelled out my hard-earned cash, I thought I would check that Nvidia’s Linux driver supported the GT 555M. A quick look at their official driver page confirmed that it did so I handed over the credit card. After all, if Nvidia themselves say that it is supported then you’d expect it to be supported right?

It's supported, honest!

Wrong! Here’s my story.

I installed Ubuntu 11.04 from DVD without a hitch and updated all packages to the very latest versions. I then hopped over to NVidia’s website, got the driver (version 275.09.07) and installed it. I’ve gone through this process dozens of times on Desktop machines at work and wasn’t expecting any problems but boy did I get problems. After installing NVidia’s driver, the Dell simply would not boot into Linux. Not only that but it never seemed to fail in exactly the same place twice…the boot process would start just fine and then it would crash…seemingly at random.  So, off to the forums I went where I quickly discovered that my system was not as supported as I originally thought.

You see, my laptop has two graphics systems on it: A relatively low-powered Intel one and the NVidia one. It also comes with some cool sounding technology called Optimus that helps save battery power on systems like mine. Rather than explain the details of Optimus, I’m just going to refer you to both Nvidia’s web page about it along with the Wikipedia page.

Here’s the kicker…Nvidia’s Linux driver does not support optimus, even though Optimus is Nvidia’s own technology. They even say that they don’t support it in the Additional Information tab. Furthermore, they have no plans to support it.  Sadly, I didn’t even realise that my new laptop was an Optimus laptop until I tried to get the Nvidia drivers on it.

If only I had thought to myself ‘Well, Nvidia may say that they support the 555M but do they really mean it?’ If I mis-trusted the information given on the supported products page then perhaps then I would have read the further information tab and trawled the forums. I chose to trust Nvidia and assume that when a product was listed under ‘supported products’ then I didn’t need to worry. Well you live and learn I guess.

Project Bumblebee

One of the fantastic things about the Linux community is that even if you are let down by your hardware vendor then someone, somewhere may well come to your rescue. For Nvidia Optimus, that someone is Martin Juhl. Martin’s project, Bumblebee, brings Optimus support to Linux which is useful since it seems that Nvidia can’t be bothered!

Installation for Ubuntu users is easy.  All you need to do is open up a terminal, type the following and follow the instructions to download and install both the Nvidia drivers along with bumblebee.

sudo apt-add-repository ppa:mj-casalogic/bumblebee
sudo apt-get update && sudo apt-get install bumblebee

To run an application, glxgears for example, you just type the following at the command line

optirun glxgears

Sadly for me this didn’t work.  All I got was the following

 * Starting Bumblebee X server bumblebee
_PS0 Enabling nVidia Card Succeded. [ OK ]

 * Stopping Bumblebee X server bumblebee
_DSM Disabling nVidia Card Succeded.
_PS3 Disabling nVidia Card Succeded.

Nothing else happened. I’d report it as a bug-report but it seems that someone with a very similar configuration to me has already done so and work is being done on it as we speak.  Plenty of other people have reported success with bumblebee though and I am confident that I will be up and running soon.  As soon as I am up and running I’ll owe the developer of bumblebee a beer!

Update 11th July: The bumblebee bug mentioned above has been fixed.  I can now run apps via optirun.  Not done much more than run glxgears though so far.

June 15th, 2011

I needed to install Labview 2010 onto a Ubuntu Linux machine but when I inserted the DVD nothing happened.  So, I tried to manually mount it from the command line in the usual way but it didn’t work. It turns out that the DVD isn’t formatted as iso9660 but as hfsplus. The following incantations worked for me

sudo mount -t hfsplus /dev/sr0 /media/cdrom0 -o loop
sudo /media/cdrom0/Linux/labview/INSTALL

The installer soon became upset and gave the following error message

/media/cdrom0/Linux/labview/bin/rpmq: error while loading shared libraries: libbz2.so.1: 
cannot open shared object file: No such file or directory

This was fixed with (original source here)

 cd /usr/lib32
 sudo ln -s libbz2.so.1.0 libbz2.so.1
 sudo ldconfig
June 13th, 2011

When installing MATLAB 2011a on Linux you may encounter a huge error message than begins with

Preparing installation files ...
Installing ...
Exception in thread "main" com.google.inject.ProvisionException: Guice provision
 errors:

1) Error in custom provider, java.lang.RuntimeException: java.lang.reflect.Invoc
ationTargetException
  at com.mathworks.wizard.WizardModule.provideDisplayProperties(WizardModule.jav
a:61)
  while locating com.mathworks.instutil.DisplayProperties
  at com.mathworks.wizard.ui.components.ComponentsModule.providePaintStrategy(Co
mponentsModule.java:72)
  while locating com.mathworks.wizard.ui.components.PaintStrategy
    for parameter 4 at com.mathworks.wizard.ui.components.SwingComponentFactoryI
mpl.(SwingComponentFactoryImpl.java:109)
  while locating com.mathworks.wizard.ui.components.SwingComponentFactoryImpl
  while locating com.mathworks.wizard.ui.components.SwingComponentFactory
    for parameter 1 at com.mathworks.wizard.ui.WizardUIImpl.(WizardUIImpl.
java:64)
  while locating com.mathworks.wizard.ui.WizardUIImpl
  while locating com.mathworks.wizard.ui.WizardUI annotated with @com.google.inj
ect.name.Named(value=BaseWizardUI)

This is because you haven’t mounted the installation disk with the correct permissions. The fix is to run the following command as root.

mount -o remount,exec /media/MATHWORKS_R2011A/

Assuming, of course, that /media/MATHWORKS_R2011A/ is your mount point. Hope this helps someone out there.

Update: 7th April 2014
A Debian 7.4 user had this exact problem but the above command didn’t work. We got the following

mount -o remount,exec /media/cdrom0

mount: cannot remount block device /dev/sr0 read-write, is write-protected

The fix was to modify the command slightly:

mount -o remount,exec,ro /media/cdrom0
June 8th, 2011

I’ve been a user of Ubuntu Linux for years but the recent emphasis on their new Unity interface has put me off somewhat.  I tried to like it but failed.  So, I figured that it was time for a switch to a different distribution.

I asked around on Twitter and got suggestions such as Slackware, Debian and Linux Mint.  I’ve used both Slackware and Debian in the past but, while they might be fine for servers or workstations, I prefer something more shiny for my personal laptop.

I could also have stuck with Ubuntu and simply installed GNOME using synaptic but I like to use the desktop that is officially supported by the distribution.

So, I went with Linux Mint.  It isn’t going well so far!

I had no DVDs in the house so I downloaded the CD version, burned it to a blank CD and rebooted only to be rewarded with

Can not mount /dev/loop0 (/cdrom/casper/filesystem.squashfs) on //filesystem.squashfs

I checked the md5sum of the .iso file and it was fine. I burned to a different CD and tried again. Same error.

I was in no mood for a trawl of the forums so I simply figured that maybe something was wrong with the CD version of the distribution – at least as far as my machine was concerned. So, I started downloading the DVD version and treated my greyhound to a walk to the local computer shop to buy a stack of DVDs.

When I got back I checked the .md5 sum of the DVD image, burned it to disk and…got the same error. A trawl of the forums suggests that many people have seen this error but no reliable solution has been found.

Not good for me or Linux Mint but at least Desmond (below) got an extra walk!

Desmond the greyhound

Update 1 I created a bootable USB memory stick from the DVD .iso to elimiate any problems with my burning software/hardware. Still get the same error message. MD5 checksum of the .iso file is what it should be:

md5sum ./linuxmint-11-gnome-dvd-64bit.iso
773b6cdfe44b91bc44448fa7b34bffa8  ./linuxmint-11-gnome-dvd-64bit.iso

My machine is a Dell XPS M1330 which has been running Ubuntu for almost 3 years.

Update 2: Seems that this bug is not confined to Mint. Ubuntu users are reporting it too. No fix yet though
https://bugs.launchpad.net/ubuntu/+bug/636711

Update 3: There is DEFINITELY nothing wrong with the installation media.  Both USB memory stick and DVD versions boot on my wife’s (much newer)HP laptop with no problem.  So, the issue seems to be related to my particular hardware.  This is like the good old days of Linux where installation was actually difficult.  Good times!

Update 4: After much mucking around I finally gave up on a direct install of Mint 11.  The installer is simply broken for certain hardware configurations as far as I can tell.  Installed Mint 10 from the same pen drive that failed for Mint 11 without a hitch.

Update 5: As soon as the Mint 10 install completed, I did an apt-get dist-upgrade to try to get to Mint 11 that way. The Mint developers recommend against doing dist-upgrades but I don’t seem to have a choice since the Mint 11 installer won’t work on my machine. After a few minutes I get this error

dpkg: error processing python2.7-minimal (--configure):
 subprocess installed post-installation script returned error exit status 3
Errors were encountered while processing:
 python2.7-minimal

This is mentioned in this bug report.  I get over that (by following the instructions in #9 of the bug report) and later get this error

p: cannot stat `/usr/lib/pango/1.6.0/module-files.d/libpango1.0-0.modules': No such file or directory
cp: cannot stat `/usr/lib/pango/1.6.0/modules/pango-basic-fc.so': No such file or directory
E: /usr/share/initramfs-tools/hooks/plymouth failed with return 1.
update-initramfs: failed for /boot/initrd.img-2.6.35-22-generic
dpkg: error processing initramfs-tools (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 initramfs-tools

I fixed this with

sudo ln -s x86_64-linux-gnu/pango /usr/lib/pango

Trying the apt-get dist-upgrade again leads to

The following packages have unmet dependencies:
 python-couchdb : Breaks: desktopcouch (< 1.0) but 0.6.9b-0ubuntu1 is to be installed
 python-desktopcouch-records : Conflicts: desktopcouch (< 1.0.7-0ubuntu2) but 0.6.9b-0ubuntu1 is to be installed

Which, thanks to this forum post, I get rid of by doing

sudo dpkg --configure -a
sudo apt-get remove python-desktopcouch-records desktopcouch evolution-couchdb python-desktopcouch

A few more packages get installed before it stops again with the error message

Unpacking replacement xserver-xorg-video-tseng ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Errors were encountered while processing:
 /var/cache/apt/archives/xserver-xorg-core_2%3a1.10.1-1ubuntu1.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

I get past this by doing

sudo apt-get -f install

Then I try apt-get upgrade and apt-get dist-update again…possibly twice and I’m pretty much done it seems.

Update 6: On the train to work this morning I thought I’d boot into my shiny new Mint system. However I was faced with nothing but a blank screen.  I rebooted and removed quiet and splash from the grub options to allow me to see what was going on. The boot sequence was getting stuck on something like checking battery state. Up until now I had only been using Mint while connected to the Mains. Well, this was the final straw for me.  As soon as I got into work I shoved in a Ubuntu 11.04 live disk which installed in the time it took me to drink a cup of coffee. I’ve got GNOME running and am now happy. 

My Linux Mint adventure is over.