Walking Randomly

I recently installed Ubuntu 12.04 on my laptop.  I gave Unity a chance for a few days just in case it had improved since I last used it but still found it unusable.  The following tweaks made Ubuntu usable again (for me at least).

• Install Bumblebee – I have an NVIDIA Graphics card that makes use of Optimus technology.  I learned that NVIDIA do not support Optimus laptops on Linux the hard way but, thankfully, this open source project makes everything work.
• Install Cinnamon – Unity is an awful desktop interface in my opinion.  Cinnamon, on the other hand, is very nice indeed
• Fix the scroll bars – Even after switching to Cinnamon, many applications in Ubuntu still use some truly awful window scrollbars.  This link shows you how to make them go away.

That was pretty much it and I’m very happy with the result.  Do you use Ubuntu?  If so, are there any tweaks that you simply must make to the default setup before you feel that it’s usable for you?

I recently maxed out my credit card in order to treat myself to a shiny new Dell XPS L720X laptop that comes complete with Intel i7 sandy bridge processor and Nvidia GeForce GT 555M. The NVidia graphics card was one of the biggest selling points for me because I wanted to do some GPU work at home and on the train using both CUDA and OpenCL.  I get asked about these technologies a lot by researchers at The University of Manchester and I wanted to beef up my experience levels.

I wanted this machine to be dual boot Windows 7 and Linux so, before I shelled out my hard-earned cash, I thought I would check that Nvidia’s Linux driver supported the GT 555M. A quick look at their official driver page confirmed that it did so I handed over the credit card. After all, if Nvidia themselves say that it is supported then you’d expect it to be supported right?

Wrong! Here’s my story.

I installed Ubuntu 11.04 from DVD without a hitch and updated all packages to the very latest versions. I then hopped over to NVidia’s website, got the driver (version 275.09.07) and installed it. I’ve gone through this process dozens of times on Desktop machines at work and wasn’t expecting any problems but boy did I get problems. After installing NVidia’s driver, the Dell simply would not boot into Linux. Not only that but it never seemed to fail in exactly the same place twice…the boot process would start just fine and then it would crash…seemingly at random.  So, off to the forums I went where I quickly discovered that my system was not as supported as I originally thought.

You see, my laptop has two graphics systems on it: A relatively low-powered Intel one and the NVidia one. It also comes with some cool sounding technology called Optimus that helps save battery power on systems like mine. Rather than explain the details of Optimus, I’m just going to refer you to both Nvidia’s web page about it along with the Wikipedia page.

Here’s the kicker…Nvidia’s Linux driver does not support optimus, even though Optimus is Nvidia’s own technology. They even say that they don’t support it in the Additional Information tab. Furthermore, they have no plans to support it.  Sadly, I didn’t even realise that my new laptop was an Optimus laptop until I tried to get the Nvidia drivers on it.

If only I had thought to myself ‘Well, Nvidia may say that they support the 555M but do they really mean it?’ If I mis-trusted the information given on the supported products page then perhaps then I would have read the further information tab and trawled the forums. I chose to trust Nvidia and assume that when a product was listed under ‘supported products’ then I didn’t need to worry. Well you live and learn I guess.

Project Bumblebee

One of the fantastic things about the Linux community is that even if you are let down by your hardware vendor then someone, somewhere may well come to your rescue. For Nvidia Optimus, that someone is Martin Juhl. Martin’s project, Bumblebee, brings Optimus support to Linux which is useful since it seems that Nvidia can’t be bothered!

Installation for Ubuntu users is easy.  All you need to do is open up a terminal, type the following and follow the instructions to download and install both the Nvidia drivers along with bumblebee.

sudo apt-add-repository ppa:mj-casalogic/bumblebee
sudo apt-get update && sudo apt-get install bumblebee

To run an application, glxgears for example, you just type the following at the command line

optirun glxgears

Sadly for me this didn’t work.  All I got was the following

 * Starting Bumblebee X server bumblebee
_PS0 Enabling nVidia Card Succeded. [ OK ]

 * Stopping Bumblebee X server bumblebee
_DSM Disabling nVidia Card Succeded.
_PS3 Disabling nVidia Card Succeded.

Nothing else happened. I’d report it as a bug-report but it seems that someone with a very similar configuration to me has already done so and work is being done on it as we speak.  Plenty of other people have reported success with bumblebee though and I am confident that I will be up and running soon.  As soon as I am up and running I’ll owe the developer of bumblebee a beer!

Update 11th July: The bumblebee bug mentioned above has been fixed.  I can now run apps via optirun.  Not done much more than run glxgears though so far.

I needed to install Labview 2010 onto a Ubuntu Linux machine but when I inserted the DVD nothing happened.  So, I tried to manually mount it from the command line in the usual way but it didn’t work. It turns out that the DVD isn’t formatted as iso9660 but as hfsplus. The following incantations worked for me

sudo mount -t hfsplus /dev/sr0 /media/cdrom0 -o loop
sudo /media/cdrom0/Linux/labview/INSTALL

The installer soon became upset and gave the following error message

/media/cdrom0/Linux/labview/bin/rpmq: error while loading shared libraries: libbz2.so.1: 
cannot open shared object file: No such file or directory

This was fixed with (original source here)

 cd /usr/lib32
 sudo ln -s libbz2.so.1.0 libbz2.so.1
 sudo ldconfig

When installing MATLAB 2011a on Linux you may encounter a huge error message than begins with

Preparing installation files ...
Installing ...
Exception in thread "main" com.google.inject.ProvisionException: Guice provision
 errors:

1) Error in custom provider, java.lang.RuntimeException: java.lang.reflect.Invoc
ationTargetException
  at com.mathworks.wizard.WizardModule.provideDisplayProperties(WizardModule.jav
a:61)
  while locating com.mathworks.instutil.DisplayProperties
  at com.mathworks.wizard.ui.components.ComponentsModule.providePaintStrategy(Co
mponentsModule.java:72)
  while locating com.mathworks.wizard.ui.components.PaintStrategy
    for parameter 4 at com.mathworks.wizard.ui.components.SwingComponentFactoryI
mpl.(SwingComponentFactoryImpl.java:109)
  while locating com.mathworks.wizard.ui.components.SwingComponentFactoryImpl
  while locating com.mathworks.wizard.ui.components.SwingComponentFactory
    for parameter 1 at com.mathworks.wizard.ui.WizardUIImpl.(WizardUIImpl.
java:64)
  while locating com.mathworks.wizard.ui.WizardUIImpl
  while locating com.mathworks.wizard.ui.WizardUI annotated with @com.google.inj
ect.name.Named(value=BaseWizardUI)

This is because you haven’t mounted the installation disk with the correct permissions. The fix is to run the following command as root.

mount -o remount,exec /media/MATHWORKS_R2011A/

Assuming, of course, that /media/MATHWORKS_R2011A/ is your mount point. Hope this helps someone out there.

I’ve been a user of Ubuntu Linux for years but the recent emphasis on their new Unity interface has put me off somewhat.  I tried to like it but failed.  So, I figured that it was time for a switch to a different distribution.

I asked around on Twitter and got suggestions such as Slackware, Debian and Linux Mint.  I’ve used both Slackware and Debian in the past but, while they might be fine for servers or workstations, I prefer something more shiny for my personal laptop.

I could also have stuck with Ubuntu and simply installed GNOME using synaptic but I like to use the desktop that is officially supported by the distribution.

So, I went with Linux Mint.  It isn’t going well so far!

I had no DVDs in the house so I downloaded the CD version, burned it to a blank CD and rebooted only to be rewarded with

Can not mount /dev/loop0 (/cdrom/casper/filesystem.squashfs) on //filesystem.squashfs

I checked the md5sum of the .iso file and it was fine. I burned to a different CD and tried again. Same error.

I was in no mood for a trawl of the forums so I simply figured that maybe something was wrong with the CD version of the distribution – at least as far as my machine was concerned. So, I started downloading the DVD version and treated my greyhound to a walk to the local computer shop to buy a stack of DVDs.

When I got back I checked the .md5 sum of the DVD image, burned it to disk and…got the same error. A trawl of the forums suggests that many people have seen this error but no reliable solution has been found.

Not good for me or Linux Mint but at least Desmond (below) got an extra walk!

Update 1 I created a bootable USB memory stick from the DVD .iso to elimiate any problems with my burning software/hardware. Still get the same error message. MD5 checksum of the .iso file is what it should be:

md5sum ./linuxmint-11-gnome-dvd-64bit.iso
773b6cdfe44b91bc44448fa7b34bffa8  ./linuxmint-11-gnome-dvd-64bit.iso

My machine is a Dell XPS M1330 which has been running Ubuntu for almost 3 years.

Update 2: Seems that this bug is not confined to Mint. Ubuntu users are reporting it too. No fix yet though
https://bugs.launchpad.net/ubuntu/+bug/636711

Update 3: There is DEFINITELY nothing wrong with the installation media.  Both USB memory stick and DVD versions boot on my wife’s (much newer)HP laptop with no problem.  So, the issue seems to be related to my particular hardware.  This is like the good old days of Linux where installation was actually difficult.  Good times!

Update 4: After much mucking around I finally gave up on a direct install of Mint 11.  The installer is simply broken for certain hardware configurations as far as I can tell.  Installed Mint 10 from the same pen drive that failed for Mint 11 without a hitch.

Update 5: As soon as the Mint 10 install completed, I did an apt-get dist-upgrade to try to get to Mint 11 that way. The Mint developers recommend against doing dist-upgrades but I don’t seem to have a choice since the Mint 11 installer won’t work on my machine. After a few minutes I get this error

dpkg: error processing python2.7-minimal (--configure):
 subprocess installed post-installation script returned error exit status 3
Errors were encountered while processing:
 python2.7-minimal

This is mentioned in this bug report.  I get over that (by following the instructions in #9 of the bug report) and later get this error

p: cannot stat `/usr/lib/pango/1.6.0/module-files.d/libpango1.0-0.modules': No such file or directory
cp: cannot stat `/usr/lib/pango/1.6.0/modules/pango-basic-fc.so': No such file or directory
E: /usr/share/initramfs-tools/hooks/plymouth failed with return 1.
update-initramfs: failed for /boot/initrd.img-2.6.35-22-generic
dpkg: error processing initramfs-tools (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 initramfs-tools

I fixed this with

sudo ln -s x86_64-linux-gnu/pango /usr/lib/pango

Trying the apt-get dist-upgrade again leads to

The following packages have unmet dependencies:
 python-couchdb : Breaks: desktopcouch (< 1.0) but 0.6.9b-0ubuntu1 is to be installed
 python-desktopcouch-records : Conflicts: desktopcouch (< 1.0.7-0ubuntu2) but 0.6.9b-0ubuntu1 is to be installed

Which, thanks to this forum post, I get rid of by doing

sudo dpkg --configure -a
sudo apt-get remove python-desktopcouch-records desktopcouch evolution-couchdb python-desktopcouch

A few more packages get installed before it stops again with the error message

Unpacking replacement xserver-xorg-video-tseng ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Errors were encountered while processing:
 /var/cache/apt/archives/xserver-xorg-core_2%3a1.10.1-1ubuntu1.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

I get past this by doing

sudo apt-get -f install

Then I try apt-get upgrade and apt-get dist-update again…possibly twice and I’m pretty much done it seems.

Update 6: On the train to work this morning I thought I’d boot into my shiny new Mint system. However I was faced with nothing but a blank screen.  I rebooted and removed quiet and splash from the grub options to allow me to see what was going on. The boot sequence was getting stuck on something like checking battery state. Up until now I had only been using Mint while connected to the Mains. Well, this was the final straw for me.  As soon as I got into work I shoved in a Ubuntu 11.04 live disk which installed in the time it took me to drink a cup of coffee. I’ve got GNOME running and am now happy. 

My Linux Mint adventure is over.

In my previous blog post I mentioned that I am a member of a team that supports High Throughput Computing (HTC) at The University of Manchester via a 1600+ core ‘condor pool’.  In order to make it as easy as possible for our researchers to make use of this resource one of my colleagues, Ian Cottam, created a system called DropAndCompute.  In this guest blog post, Ian describes DropAndCompute and how it evolved into the system we use at Manchester today.

The Evolution of “DropAndCompute” by Ian Cottam

DropAndCompute, as used at The University of Manchester’s Faculty of Engineering and Physical Sciences, is an approach to using network (or grid or cloud based) computational resources without having to know the operating system of the resource’s gateway or any command line tools of either the resource itself –Condor in our case — or in general. Most such gateways run a flavour of Unix, often Linux. Many of our users are either unfamiliar with Linux or just prefer a drag-and-drop interface, as I do myself despite using various flavours of Unix since Version 6 in the late 70s.

Why did I invent it? On its original web site description page wiki.myexperiment.org/index.php/DropAndCompute the following reasons are given:

• A simple and uniform drag-and-drop graphical user interface, potentially, to many resource pools.
• No use of terminal windows or command lines.
• No need to login to remote hosts or install complicated grid-enabling software locally.
• No need for the user to have an account on the remote resources (instead they are accounted by having a shared folder allocated).  Of course, nothing stops the users from having accounts should that be preferred.
• No need for complicated Virtual Private Networks, IP Tunnelling, connection brokers, or similar, in order to access grid resources on private subnets (provided at least one node is on the public Internet, which is the norm).
• Pop-ups notify users of important events (basically, log and output files being created when a job has been accepted, and when the generated result files arrive).
• Somewhat increased security as the user only has (indirect) access to a small subset of the computational resource’s commands.

Version One
The first version was used on a Condor Pool within our interdisciplinary biocentre (MIB).  A video of it in use is shown below

Please do take the time to look at this video as it shows clearly how, for example, Condor can be used via this type of interface.

This version was notable for using the commercial service: Dropbox and, in fact, my being a Dropbox user inspired the approach and its name. Dropbox is trivial to install on any of the main platforms, on any number of computers owned by a user, and has a free version giving 2GB of synchronised and shared storage. In theory, only the computational resource supplier need pay for a 100GB account with Dropbox, have a local Condor submitting account, and share folders out with users of the free Dropbox-based service.

David De Roure, then at the University of Southampton and now Oxford, reviewed this approach here at blog.openwetware.org/deroure/?p=97, and offers his view as to why it is important in helping scientists start on the ‘ramp’ to using what can be daunting, if powerful, computational facilities.

Version Two

Quickly the approach migrated to our full, faculty-wide Condor Pool and the first modification was made. Now we used separate accounts for each user of the service on our submitting nodes; Dropbox still made this sharing scheme trivial to set up and manage, whilst giving us much better usage accounting information. The first minor problem came when some users needed more –much more in fact– than 2GB of space.  This was solved by them purchasing their own 50GB or 100GB accounts from Dropbox.

Problems and objections

However, two more serious problems impacted our Dropbox based approach. First, the large volume of network traffic across the world to Dropbox’s USA based servers and then back down to local machines here in Manchester resulted in severe bottlenecks once our Condor Pool had reached the dizzy heights of over a thousand processor cores. We could have ameliorated this by extra resources, such as multiple submit nodes, but the second problem proved to be more of a showstopper.

Since the introduction of DropAndCompute several people –at Manchester and beyond– have been concerned about research data passing through commercial, USA-based servers. In fact, the UK’s National Grid Service (NGS) who have implemented their own flavour of DropAndCompute did not use Dropbox for this very reason. The US Patriot Act means that US companies must surrender any data they hold if officially requested to do so by Federal Government agencies. Now one approach to this is to do user-level encryption of the data before it enters the user’s dropbox. I have demonstrated this approach, but it complicates the model and it is not so straightforward to use exactly the same method on all of the popular platforms (Windows, Mac, Linux).

Version Three

To tackle the above issues we implemented a ‘local version’ of DropAndCompute that is not Dropbox based. It is similar to the NGS approach, but, in my opinion, much simpler to setup. The user merely has to mount a folder on the submit node on their local computer(s), and then use the same drag-and-drop approach to get the job initiated, debugged and run (or even killed, when necessary). This solves the above issues, but could be regarded as inferior to the Dropbox based approach in five ways:

1. The convenience and transparency of ‘offline’ use. That is, Dropbox jobs can be prepared on, say, a laptop with or without net access, and when the laptop next connects the job submissions just happens. Ditto for the results coming back.

2. When online and submitting or waiting for results with the local version, the folder windows do not update to give the user an indication of progress.

3. Users must remember to use an email notification that a job has finished, or poll to check its status.

4. The initial setup is a little harder for the local version compared with using Dropbox.

5. The computation’s result files are not copied back automatically.

So far, only item 5 has been remarked on by some of our users, and it, and the others, could be improved with some programming effort.

A movie of this version is shown below; it doesn’t have any commentary, but essentially follows the same steps as the Dropbox based video. You will see the network folder’s window having to be refreshed manually –this is necessary on a Mac (but could be scripted); other platforms may be better– and results having to be dragged back from the mounted folder.

I welcome comments on any aspect of this –still evolving– approach to easing the entry ‘cost’ to using distributed computing resources.

Acknowledgments
Our Condor Pool is supported by three colleagues besides myself: Mark Whidby, Mike Croucher and Chris Paul. Mark, inter alia, maintains the current version of DropAndCompute that can operate locally or via Dropbox. Thanks also to Mike for letting me be a guest on Walking Randomly.

I thought I’d try out the new image capturing functionality in Mathematica 8 on my Linux based laptop with built in webcam but…

In[1]:= ImageCapture[]
During evaluation of In[1]:= ImageCapture::notsupported: Image acquisition is not supported on Unix. >>
Out[1]= ImageCapture[]

That makes me a sad panda!

Christmas isn’t all that far away so I thought that it was high time that I wrote my Christmas list for mathematical software developers and vendors.  All I want for christmas is….

Mathematica

• A built in ternary plot function would be nice
• Ship workbench with the main product  please
• An iPad version of Mathematica Player

MATLAB

• Merge the parallel computing toolbox with core MATLAB.  Everyone uses multicore these days but only a few can feel the full benefit in MATLAB.   The rest are essentially second class MATLAB citizens muddling by with a single core (most of the time)
• Make the mex interface thread safe so I can more easily write parallel mex files

Maple

• More CUDA accelerated functions please.  I was initially excited by your CUDA package but then discovered that it only accelerated one function (Matrix Multiply).  CUDA accelerated Random Number Generators would be nice along with fast Fourier transforms and a bit more linear algebra.

MathCAD

• Release Mathcad Prime.
• Mac and Linux versions of Mathcad.  Maple,Mathematica and MATLAB have versions for all 3 platforms so why don’t you?

NAG Library

• Produce vector versions of functions like g01bk (poisson distribution function).  They might not be needed in Fortran or C code but your MATLAB toolbox desperately needs them
• A Mac version of the MATLAB toolbox.  I’ve got users practically begging for it :)
• A NAG version of the MATLAB gamfit command

Octave

• A just in time compiler.  Yeah, I know, I don’t ask for much huh ;)
• A faster pdist function (statistics toolbox from Octave Forge).  I discovered that the current one is rather slow recently

SAGE Math

• A Locator control for the interact function.  I still have a bounty outstanding for the person who implements this.
• A fully featured, native windows version.  I know about the VM solution and it isn’t suitable for what I want to do (which is to deploy it on around 5000 University windows machines to introduce students to one of the best open source maths packages)

SMath Studio

• An Android version please.  Don’t make it free – you deserve some money for this awesome Mathcad alternative.

SpaceTime Mathematics

• The fact that you give the Windows version away for free is awesome but registration is a pain when you are dealing with mass deployment.  I’d love to deploy this to my University’s Windows desktop image but the per-machine registration requirement makes it difficult.  Most large developers who require registration usually come up with an alternative mechanism for enterprise-wide deployment.  You ask schools with more than 5 machines to link back to you.  I want tot put it on a few thousand machines and I would happily link back to you from several locations if you’ll help me with some sort of volume license.  I’ll also give internal (and external if anyone is interested) seminars at Manchester on why I think Spacetime is useful for teaching mathematics.  Finally, I’d encourage other UK University applications specialists to evaluate the software too.
• An Android version please.

How about you?  What would you ask for Christmas from your favourite mathematical software developers?

I recently had a set of files that were named as follows

frame1.png
frame2.png
frame3.png
frame4.png
frame5.png
frame6.png
frame7.png
frame8.png
frame9.png
frame10.png

and so on, right up to frame750.png. The plan was to turn these .png files into an uncompressed movie using mencoder via the following command (original source)

mencoder mf://*.png -mf w=720:h=720:fps=25:type=png -ovc raw -oac copy -o output.avi

but I ended up with a movie that jumped all over the place since the frames were in an odd order. In the following order in fact

frame0.csv
frame100.csv
frame101.csv
frame102.csv
frame103.csv
frame104.csv
frame105.csv
frame106.csv
frame107.csv
frame108.csv
frame109.csv
frame10.csv
frame110.csv

This is because globbing expansion (the *.png bit) is alphabetical in bash rather than numerical.

One way to get the frames in the order that I want is to zero-pad them. In other words I replace file1.png with file001.png and file20.png with file020.png and so on. Here’s how to do that in bash

#!/bin/bash
num=`expr match "$1" '[^0-9]*\([0-9]\+\).*'`
paddednum=`printf "%03d" $num`
echo ${1/$num/$paddednum}

Save the above to a file called zeropad.sh and then do the following command to make it executable

chmod +x ./zeropad.sh

You can then use the zeropad.sh script as follows

./zeropad.sh frame1.png

which will return the result

frame001.png

All that remains is to use this script to rename all of the .png files in the current directory such that they are zeropadded.

for i in *.png;do mv $i `./zeropad.sh $i`; done

You may want to change the number of digits used in each filename from 3 to 5 (say). To do this just change %03d in zeropad.sh to %05d

Let me know if you find this useful or have an alternative solution you’d like to share (in another language maybe?)

I was recently asked to install 32bit Gaussian 03 binaries on an Ubuntu 9.10 machine and when I tried to run a test job I got the following error message

Erroneous write during file extend. write -1 instead of 4096
Probably out of disk space.
Erroneous write during file extend. write -1 instead of 4096
Probably out of disk space.
Write error in NtrExt1
Write error in NtrExt1: Bad address
Segmentation fault

A bit of googling suggested that the following might work

sudo echo 0 > /proc/sys/kernel/randomize_va_space

but this will result in permission denied (explanation here). The command you really want to use is

sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"

Once this was done, Gaussian worked as advertised. Maybe this post will help a googler sometime in the future.