November 17th, 2014 | Categories: general math, Linear Algebra, NAG Library, Numerics, programming, python | Tags:

Given a symmetric matrix such as

    \light \[ \left( \begin{array}{ccc} 1 & 1 & 0 \\ 1 & 1 & 1 \\ 0 & 1 & 1 \end{array} \right)\]

What’s the nearest correlation matrix? A 2002 paper by Manchester University’s Nick Higham which answered this question has turned out to be rather popular! At the time of writing, Google tells me that it’s been cited 394 times.

Last year, Nick wrote a blog post about the algorithm he used and included some MATLAB code. He also included links to applications of this algorithm and implementations of various NCM algorithms in languages such as MATLAB, R and SAS as well as details of the superb commercial implementation by The Numerical algorithms group.

I noticed that there was no Python implementation of Nick’s code so I ported it myself.

Here’s an example IPython session using the module

In [1]: from nearest_correlation import nearcorr

In [2]: import numpy as np

In [3]: A = np.array([[2, -1, 0, 0], 
   ...:               [-1, 2, -1, 0],
   ...:               [0, -1, 2, -1], 
   ...:               [0, 0, -1, 2]])

In [4]: X = nearcorr(A)

In [5]: X
array([[ 1.        , -0.8084125 ,  0.1915875 ,  0.10677505],
       [-0.8084125 ,  1.        , -0.65623269,  0.1915875 ],
       [ 0.1915875 , -0.65623269,  1.        , -0.8084125 ],
       [ 0.10677505,  0.1915875 , -0.8084125 ,  1.        ]])

This module is in the early stages and there is a lot of work to be done. For example, I’d like to include a lot more examples in the test suite, add support for the commercial routines from NAG and implement other algorithms such as the one by Qi and Sun among other things.

Hopefully, however, it is just good enough to be useful to someone. Help yourself and let me know if there are any problems. Thanks to Vedran Sego for many useful comments and suggestions.

November 12th, 2014 | Categories: programming, software deployment, The internet | Tags:

I follow a lot of software developers on twitter so I get to see a lot of opinions and comments about git.  Here are a few of my recent favourites

Opinions on git
I find that git is a technology that polarizes people. They either love it or they hate it…often both at the same time.


Git Tutorials
I find that I learn something new every time I read a different tutorial

Git tips and tricks
No matter how long you’ve used git, you can always learn a new trick or two.

November 11th, 2014 | Categories: math software, mathematica, programming, R | Tags:

RLink is Mathematica’s interface to the R language – a feature that has been extremely popular since its debut in Mathematica version 9. It’s a great package but has one or two issues. For example, RLink makes use of a built in version of R which is currently stuck at the rather old version 2.14. Official support for the use of external versions of R and adding third-party libraries varies by operating system and version of Mathematica. Windows support is great — OS X support, not so much.

Expert Mathematica user Szabolcs Horvát has written the definitive guide on how to get RLink up and running with the latest version of R on all three major operating systems, building on earlier work by Leonid Shifrin and members of the Mathematica Stack Exchange community. Thanks to this work, we can now enjoy any version of R we like with Mathematica!

October 22nd, 2014 | Categories: Books, Free software, math software, matlab | Tags:

Many engineering textbooks such as Ogata’s Modern Control Engineering include small code examples written in languages such as MATLAB. If you don’t have access to MATLAB and if the examples don’t run in GNU Octave for some reason, the value of these textbooks is reduced.

Professor Kannan M. Moudgalya et al of the Indian Institute of Technology Bombay have developed an ambitious project that has ported the code examples of over 400 textbooks to the open-source computational system, Scilab.

The Textbook Companion Project has free Scilab code for textbooks from a range of subject areas including Fluid Mechanics,  Control Systems, Chemical Engineering and Digital Electronics.


September 22nd, 2014 | Categories: matlab, python, Scientific Software | Tags:

The Mathematics department at The University of Manchester runs a third year undergraduate module called ‘Problem solving by computer’ which invites students to solve complex mathematical problems by doing a little programming. Along with some interesting mathematics, the course exposes students to a wide variety of languages and numerical libraries including MATLAB, Octave, NAG, Mathematica and, most recently, Python.

Earlier this year, Python was introduced as an option for students who wanted to use it for a project in this course and, despite only being given two lectures in the language, quite a few people chose to use it. Much of this success must be attributed to the Python for MATLABers notes written by Manchester PhD student, Sophia Coban which is why I’m providing links to them here.

August 21st, 2014 | Categories: control theory, Free software, math software, simulink | Tags:

I currently work at The University of Manchester in the UK as a ‘Scientific Applications Support Specialist’. In recent years, I have noticed a steady increase in the use of open source software for both teaching and research – something that I regard as a Good Thing.

Even though Manchester has, what I believe is, a world-class site licensed software portfolio, researchers, lecturers and students often prefer open source solutions for all sorts of reasons. For example, researchers at Manchester can use MATLAB while they are associated with the University but their right to do so ceases as soon as they leave. If all of your research code is in the form of MATLAB and Simulink models, you had better hope that your next employer or school has the requisite licenses.

This summer, a few people in the Control Systems Centre of Manchester’s Electrical and Electronic Engineering department asked the question ‘Is it possible to implement all of the simple MATLAB/Simulink examples we use in a second year undergraduate introduction to Control Theory using free software?’ In particular, they chose the programs Scilab and Xcos.

Since the aim of this course is to teach control theory principles rather than any particular software solution, it would ideally be software agnostic. Students aren’t asked to develop models, they are just asked to play with pre-packaged models in order to improve their understanding of the material.

Student intern Danail Stoychev was tasked with attempting to port all of the examples from the course and in fairly short order he determined that the answer to their question was a resounding ‘Yes’.

For example, the model below is an example of feedback with a first order transfer function and a delay.  First in Simulink:

Simulink version 1st order transfer function with delay

and now in xcos

xcos version 1st order transfer function with delay

Part of the exercise set for the students is to define all of the relevant parameters in the workspace: b,a,k and so on. If you attempt to download and run the above, you’ll have to do that in order to make them work. You’ll also need extract and plot the results from the workspace.

It can be seen that the two models look very similar and, for these examples at least, it really doesn’t matter which piece of software the students use.

The full set of MATLAB/Simulink examples along with Danail’s Scilab/Xcos conversions can be found at

August 15th, 2014 | Categories: Carnival of Math | Tags:

Welcome to the rather delayed Carnival of Mathematics for July. This is the 113th edition of the mathematical blogging tradition that’s organised by Katie and company over at The Aperiodical.

Number Trivia and a challenge

Long held tradition dictates that I find something interesting about this month’s edition number – 113 – and I turn to Number Gossip for help. It comes up with 3 very nice little prime tidbits:

  • 113 is the smallest three-digit permutable prime
  • 113 is the smallest three-digit Unholey prime: such primes do not have holes in their digits
  • 113 is the smallest three-digit prime whose product and sum of digits is prime

Walking Randomly Challenge: Prove the above using the programming language of your choice and post in the comments section. Here’s a demonstration of the first statement using Mathematica.

(*Returns true if num is a permutable prime*)
permutePrimeQ[num_] := 
 AllTrue[PrimeQ[Map[FromDigits, Permutations[IntegerDigits[num]]]], 

(*prints all permutable primes from 1 to 1000*)
 If[permutePrimeQ[x], Print[x]]
 , {x, 1, 1000}


On with the show

Mathematical software blogs

A math carnival wouldn’t be a WalkingRandomly math carnival without some focus on mathematical software blogs. Loren of The Art of MATLAB brings us Analyzing Fitness Data from Wearable Devices in MATLAB which is a guest post by Toshi Takeuchi. The new computational programming language on the block is Julia and the Julia blog contains videos and code on mathematical Optimization in Julia. The Wolfram Blog has a video on how to Create Escher-Inspired art with Mathematica (Don’t forget that Mathematica can be had for free for the Raspberry Pi).

Sage is a free, open-source alternative to Mathematica, Maple and MATLAB and the SageMath blog recently published SageMathCloud — history and status. The Numerical Algorithms Blog, on the other hand, brings us Testing Matrix Functions Using Identities. Efficient linear algebra routines form one of the cornerstones of modern scientific computing and July saw the publication of a tutorial on how to write your own, super-fast Matrix-Matrix Multiply routine.

Stamps, Making Change and Dealing Cards

When was the last time you used a postage stamp? Even if it was a long time ago, you may have held in your hands a strip of stamps. Maybe you have even tried to fold it into a stack, one stamp wide, so that the strip was easier to store. Have you ever wondered how many ways there are to do so? This post reviews a recent research survey about the topic.

How many ways can you make change for a dollar? This post gives a Lisp program that solves the problem and an analytical solution based on generating functions.

Over at The Aperiodical, home of the math carnival, Dave Wilding has been Discovering Integer Sequences by dealing cards.

Ninjas, Lord Voldemort and Hairy Hay Balls

Colin Beveridge has written a delightful follow-up to Pat Ballew’s post which featured in CoM112 Trigonometric Trick Secrets of the Mathematica Ninja.

Ben of Math with Bad Drawings fame has written a highly readable rant about a bit of syllabus design which will resonate with anyone teaching (or learning) mathematics in The Voldemort of Calculus Classes.

New math blogger, Grace, recently had an experience in which her math education intersected with her everyday life in Hay Ball Meets the Hairy Ball Theorem.

Tweeting bots, 3D Printed Geometry and Hair ties

(/begin shameless plug) I spend a lot of time on twitter posting as @walkingrandomly (/end shameless plug) and have discovered quite a few mathematical twitter bots in my time.  Evelyn Lamb has discovered many many more and has posted reviews of them all over at Roots of Unity.

James Tanton posts fun problems on twitter all the time. One particular problem caught the attention of Mike Lawler because of how 3D printing could help younger children see the geometry.

If you’ve ever wondered what a 120-cell would look like if it were made out of hair ties, wonder no longer because Andrea Hawksley has made one – Hair Tie 120-Cell (By the way, hold your mouse of the title of her blog – it’s cool!)

More Ninjas, Dots and Mandelbrots.

A round-up of fun stuff from Colin Beveridge’s “Flying Colours Maths” – a kind of @icecolbeveridge carnival for Mathematics Carnival.

Have you ever wondered what mathematics is behind those pretty Mandelbrot posters that are all over the place? Find out over at Grey Matters: Blog – Mandelbrot Set: What Exactly Are We Looking At, Anyway?

Did you know that dots have power? To see how much power, check out Keith Devlin’s article The Power of Dots.

Football, drugs and underwear

Since I’m not a big fan of football at the best of times, the football world cup is a distant memory for me (England’s dismal performance didn’t help much either).  Fortunately, there’s more than one way to enjoy a world cup and Maxwell’s Demon and The Guardian helped me enjoy it in a data science way: How shocking was Brazil’s 7-1 defeat, mathematically speaking? and Data Visualization, From The World Cup To Drugs In Arkansas.

At this time of year, many of us turn our thoughts to vacations. If you are a math geek, you owe it to yourself to optimise your underwear and pack like a nerd.

Puzzles, Games and playing like a mathmo

When I’m on vacation, I often take a notebook with me so I can do a little mathematics during downtime. My wife and friends find this extremely odd behaviour because mathematics looks like hard work to short, they don’t know how to play like a mathematician.

My job at The University of Manchester is wonderful because it often feels like I am being paid to solve puzzles. While on vacation, however, I have to find some unpaid puzzles to solve and this concentric circles puzzle is an example of one thats fun to solve.

I think that one of the best ways to learn is through play and games and, in a new post over at Math Frolic, a Li’l Game From Martin Gardner introduces mathematical “isomorphism.”

Cartoons and Limits

Mathematics is everywhere, it’s even in the voice bubbles used in web comics!  As an added bonus, this blog post contains a little Python programming too.

Next up, we have Part 2 of Bressoud’s masterful investigation (Part I featured in Carnival 112) of how students understand limits.

Resources, Exams and Books

Colleen Young brings us a great set of Standard Form Resources.

Patrick Honner continues his long-running evaluation of New York State math Regents exams, the high school required exams there. In this post, he looks at a multiple choice question that asks the student to identify the graph of an “absolute value equation”.

The Maths Book Club gives details of their most anticipated maths books for the rest of 2014.

Bad Mnemonics and a Dislike of Mathematics 

Andrea Hawklsey muses on why some people dislike mathematics when they have interests that suggest they should.  Perhaps it has something to do with poorly executed mnemonics when students are taught mathematics, or perhaps its just because they had dull math teachers? Most of my math teachers were awesome and I’ve always felt that this was a major factor in me enjoying the subject.

Skirts, Snow globes and Mathematical Mind Hacking 

This edition of the carnival is in danger of becoming the Carnival of Andrea Hawksley since so many of her great posts were submitted! In one of her July posts, she manages to combine fashion and hyperbolic geometry – which is quite a feat!

The author of cavmaths has been musing over the dynamics governing snow globes. Can anyone help out?

Hacking your mind sounds like it might be dangerous but it turns out that it’s really quite safe. Head over to Moebius Noodles to see an example of a mathematical mind hack.

That’s all folks

Thanks to Katie for inviting me to host this month’s carnival, thanks to everyone for submitting so many great articles and thanks to you for reading. Carnival #114 will be hosted over at SquareCirclez.

August 4th, 2014 | Categories: HPC, Scientific Software, walking randomly | Tags:

I work at The University of Manchester as part of IT Service’s Research Support department. This month, I was given the task of editing the newsletter that we send out to the academics we serve.

For me, this month’s highlights include the fact that our Condor Pool (which I help build and run) has now delivered over 20 million CPU hours to academics at Manchester by making use of desktop PCs around campus. Another HPC system we have at Manchester is our Computational Shared Facility which is jointly funded by the University and all of the research groups who use it. This system has just seen the installation of its 6000-th CPU core.

Other news includes the Greater Manchester Data Dive, an AGM for Research Software Engineers, Image based modelling and more.

If you are interested in seeing the sort of thing we get up to, check out this months newsletter at

July 21st, 2014 | Categories: Carnival of Math | Tags:

It’s been quite some time since I hosted a Carnival of Mathematics with the last one being #90 back in September 2012. I am very pleased to announce that I’ll be hosting #113 next month and am currently accepting articles.  If you’ve written a mathematical blog post recently and would like to give it more exposure, send in a submission.

July 17th, 2014 | Categories: Powershell, Scientific Software, software deployment | Tags:

Update: September 2014 – The notes in this blog post have been uploaded to github: The blog post will be kept as-is for posterity reasons. For the most up to date version of the notes, see the github version.

Some time in 2013, I helped out at a Software Carpentry event at The University of Bath.  As with most software carpentry boot camps, one of the topics covered was shell scripting and the scripting language of choice was bash.  As I wandered around the room, I asked the delegates which operating system they use for the majority of their research and the most popular answer, by far, was Windows.

This led me to wonder if we should teach using a native Windows solution rather than relying on bash?

A few years ago, this would be an insane proposition since the Windows command shell is very weak compared to bash.  PowerShell, on the other hand, is modern, powerful and installed on all modern Windows operating systems by default.

My problem was that I didn’t know PowerShell very well.  So, I took the notes for the 2013 Bath shell scripting session – – and gave myself the exercise of converting them to PowerShell.

I got close to completing this exercise last summer but various things took higher priority and so the project languished.  Rather than sit on the notes any longer, I’ve decided to just publish what I have so far in case they are useful to anyone.

You are free to use them with the following caveats

  • This is not necessarily the right way to teach PowerShell. It is an experiment in converting some classroom-tested Linux based notes to PowerShell.
  • If you use them, attribution would be nice. I am Mike Croucher, my site is Details on how to contact me at
  • I have not yet tested these notes in a classroom situation
  • These notes aren’t finished yet
  • These notes have been developed and tested on Windows 7.  Behaviour may be different using different versions of Windows.
  • These notes are given as they were left sometime in mid 2013. Some things may be out of date.
  • I was learning PowerShell as I developed these notes. As such, I fully expect them to be full of mistakes.  Corrections and improvements would be welcomed.

If anyone is interested in developing these notes into something that’s classroom-ready, contact me.

The old Windows Command Shell

The traditional Windows command shell is a program called cmd.exe which can trace its roots all the way back to the old, pre-Windows DOS prompt.

You can launch this command shell as follows

  • Hold down both the Windows button and the letter R to open the Run prompt
  • Type cmd and press Enter or click OK

Launch Cmd

  • You should see a window similar to the one below

Windows Command Prompt

The Windows command shell hasn’t changed significantly for over twenty years and is relatively feature poor compared to more modern shells. For this reason, it is recommended that you use Windows PowerShell instead. Mention of cmd.exe is only included here since, despite its deficiencies, it is still widely in use


To launch PowerShell:

  • Hold down both the Windows button and the letter R to open the Run prompt
  • Type powershell and press Enter or click OK

Launch PowerShell

  • You should see a window similar to the one below

PowerShell prompt

Note that although the header of the above window mentions v1.0, it could be a screenshot from either version 1.0 or version 2.0. This is a well-known bug. If you are using Windows 7 you will have version 2 at the minimum.

PowerShell versions

At the time of writing, PowerShell is at version 3. Ideally, you should at least have version 2.0 installed. To check version:


Major  Minor  Build  Revision
-----  -----  -----  --------
3      0      -1     -1

If this variable does not exist, you are probably using version 1.0 and should upgrade.

Version 3.0 is available at


# This is a comment in Powershell. It is not executed


Users of Bash will feel right at home at first since PowerShell appears to have the same set of commands

pwd                         #Path to current folder
ls                          #List directory
ls *.txt                    #Wild Card
ls *_hai*
ls -R                       #Recursive folder listing
ls .                        #List current folder
ls ..                       #List Parent folder
cd ..                       #Change current folder to parent. (Move up a folder)
cd ~                        #Change current folder to your user directory.
mkdir myfolder              #Create a folder
mkdir ~/myfolder    
mv myfolder new_myfolder    #rename myfolder to new_myfolder
rm -r new_myfolder          #Delete new_myfolder if its empty


cat file                    # View file
more file                   # Page through file
cat file | select -first 3  # first N lines
cat file | select -last 2   # Last N lines
cp file1 file2              # Copy
cp *.txt directory
rm file.txt                 # Delete - no recycle bin.
rm -r directory             # Recurse

Different command types in PowerShell: Aliases, Functions and Cmdlets

Many of the PowerShell ‘commands’ we’ve used so far are actually aliases to Powershell Cmdlets which have a Verb-Noun naming convention. We can discover what each command is an alias of using the get-alias cmdlet.

PS > get-alias ls

CommandType     Name                                                Definition
-----------     ----                                                ----------
Alias           ls                                                  Get-ChildItem

This shows that ls is an alias for the Cmdlet Get-ChildItem

A list of aliases for common Bash commands:

  • cat (Get-Content)
  • cd (Set-Location)
  • ls (Get-ChildItem)
  • pwd (Get-Location)

One reason why aliases were created is to make PowerShell a more familiar environment for users of other shells such as the old Windows cmd.exe or Linux’s Bash environment and also to save on typing.

You can get a list of all aliases using get-alias on its own.

PS > get-alias

Finally, here’s how you get all of the aliases for the Get-ChildItem cmdlet.

get-alias | where-object {$_.Definition -match "Get-Childitem"}

For more details on Powershell aliases, see Microsoft’s documentation at

What type of command is mkdir?

The mkdir command looks like it might be an alias as well since it doesn’t have the verb-noun naming convention of Cmdlets. Let’s try to see which Cmdlet it might be an alias of:

PS > get-alias mkdir

Get-Alias : This command cannot find a matching alias because alias with name 'mkdir' do not exist. 
At line:1 char:6
+ alias <<<<  mkdir
    + CategoryInfo          : ObjectNotFound: (mkdir:String) [Get-Alias], ItemNotFoundException
    + FullyQualifiedErrorId : ItemNotFoundException,Microsoft.PowerShell.Commands.GetAliasCommand

It turns out that mkdir isn’t an alias at all but is actually yet another PowerShell command type, a function. We can see this by using the get-command Cmdlet

PS > get-command mkdir
CommandType     Name                                                Definition
-----------     ----                                                ----------
Function        mkdir                                               ...
Application     mkdir.exe                                           C:\Program Files (x86)\Git\bin\mkdir.exe

Now we can clearly see that mkdir is a PowerShell function. The mkdir.exe is an Application which you’ll only see if you installed git for windows as I have.


A Cmdlet (pronounced ‘command-let’) is a .NET class but you don’t need to worry abut what this means until you get into advanced PowerShell usage. Just think of Cmdlets as the base type of PowerShell command. They are always named according to the convention verb-noun; for example Set-Location and Get-ChildItem.

Listing all Cmdlets

The following lists all Cmdlets


You can pipe this list to a pager

Get-Command | more

Getting help

You can get help on any PowerShell command using the -? switch. For example

ls -?

When you do this, you’ll get help for the Get-ChildItem Cmdlet which would be confusing if you didn’t know that ls is actually an alias for Get-ChildItem


Up arrow browses previous commands.

By default, PowerShell version 2 remembers the last 64 commands whereas PowerShell version 3 remembers 4096. This number is controlled by the $MaximumHistoryCount variable

PS > $MaximumHistoryCount           #Display the current value
PS > $MaximumHistoryCount=150       #Change it to 150
PS > history                        #Display recent history using the alias version of the command
PS > get-history                    #Display recent history using the Cmdlet direct

Although it remembers more, PowerShell only shows the last 32 commands by default. To see a different number, use the count switch

PS > get-history -count 50

To run the Nth command in the history use Invoke-History

PS > invoke-history 7

Word count (and more) using Measure-Object

Linux has a command called wc that counts the number of lines and words in a file. Powershell has no such command but we can do something similar with the Measure-Object Cmdlet.

Say we want to count the number of lines, words and characters in the file foo.txt. The first step is to get the content of the file

get-content foo.txt                 # gets the content of foo.txt

Next, we pipe the result of the get-content Cmdlet to Measure-Object, requesting lines, words and characters

get-content foo.txt | measure-object -line -character -word

The measure-object Cmdlet can also count files

ls *.txt | measure-object           #Counts number of .txt files in the current folder

When you execute the above command, a table of results will be returned:

Count    : 3
Average  :
Sum      :
Maximum  :
Minimum  :
Property :

This is because the measure-object Cmdlet, like all PowerShell Cmdlets, actually returns an object and the above table is the textual representation of that object.

The fields in this table hint that measure-object can do a lot more than simply count things. For example, here we find some statistics concerning the file lengths found by the ls *.txt command

ls *.txt | measure-object -property length -minimum -maximum -sum -average

You may wonder exactly what type of object has been returned from measure-object and we can discover this by running the gettype() method of the returned object

(ls *.txt | measure-object).gettype()

Request just the name as follows

(ls *.txt | measure-object).gettype().Name


To find out what properties an object has, pass it to the get-member Cmdlet

#Return all member types
ls *.txt | get-member

#Return only Properties
ls *.txt | get-member -membertype property

Sometimes, you’ll want to simply return the numerical value of an object’s property and you do this using the select-object Cmdlet. Here we ask for just the Count property of the GenericMeasureInfo object returned by measure-object.

#Counts the number of *.txt files and returns just the numerical result
ls *.txt | measure-object | select-object -expand Count

Searching within files

The Unix world has grep, PowerShell has Select String.  Try running the following on haiku.txt

Select-String the haiku.txt                             #Case insensitive by default, unlike grep
Select-String the haiku.txt -CaseSensitive              #Behaves more like grep
Select-String day haiku.txt -CaseSensitive
Select-String is haiku.txt -CaseSensitive
Select-String 'it is' haiku.txt -Casesensitive

There is no direct equivalent to grep’s -w switch.

grep -w is haiku.txt            #exact match

However, you can get the same behaviour using the word boundary anchors, \b

Select-String \bis\b haiku.txt -casesensitive

Grep has a -v switch that shows all lines that do not match a pattern. Select-String makes use of the -notmatch switch.

BASH: grep -v "is" haiku.txxt
PS: select-string -notmatch "is" haiku.txt -CaseSensitive

Grep has an -r switch which stands for ‘recursive’. The following will search through all files and subfolders of your current directory, looking for files that contain is

grep -r is *

Select-String has no direct equivalent to this. However, you can do the same thing by using get-childitem to get the list of files, piping the output to select-string

get-childitem * -recurse | select-string is

One difference between grep and Select-String is that the latter includes the filename and line number of each match.

grep the haiku.txt

Is not the true Tao, until
and the presence of absence:

Select-String the haiku.txt -CaseSensitive

haiku.txt:2:Is not the true Tao, until
haiku.txt:6:and the presence of absence:  

To get the grep-like output, use the following

Select-String the haiku.txt -CaseSensitive | ForEach-Object {$_.Line}

Is not the true Tao, until
and the presence of absence:

To understand how this works, you first have to know that Select-String returns an array of MatchInfo objects when there is more than one match. To demonstrate this:

$mymatches = Select-String the haiku.txt -CaseSensitive  #Put all matches in the variable 'mymatches'
$mymatches -is [Array]          #query if 'match' is an array


So, mymatches is an array. We can see how many elements it has using the array’s Count property



The type of elements in PowerShell arrays don’t necessarily have to be the same. In this case, however, they are.


both of these give the output

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    MatchInfo                                System.Object

If all you wanted was the name of the first object type, you’d do



Alternatively, we could have asked for each element’s type using the For-Each-Object Cmdlet to loop over every object in the array.

$mymatches | Foreach-Object {$_.gettype().Name}

Where $_ is a special variable that effectively means ‘current object’ or ‘The object currently being considered by Foreach-Object’ if you want to be more verbose.

So, we know that we have an array of 2 MatchInfo objects in our variable mymatches. What does this mean? What properties do MatchInfo objects have? We can find out by piping one of them to the Get-Member Cmdlet.

$mymatches[0] | Get-Member

   TypeName: Microsoft.PowerShell.Commands.MatchInfo

Name         MemberType Definition
----         ---------- ----------  
Equals       Method     bool Equals(System.Object obj)
GetHashCode  Method     int GetHashCode()
GetType      Method     type GetType()
RelativePath Method     string RelativePath(string directory)
ToString     Method     string ToString(), string ToString(string directory)
Context      Property   Microsoft.PowerShell.Commands.MatchInfoContext Context {get;se
Filename     Property   System.String Filename {get;}
IgnoreCase   Property   System.Boolean IgnoreCase {get;set;}
Line         Property   System.String Line {get;set;}   
LineNumber   Property   System.Int32 LineNumber {get;set;}
Matches      Property   System.Text.RegularExpressions.Match[] Matches {get;set;}
Path         Property   System.String Path {get;set;}
Pattern      Property   System.String Pattern {get;set;}    

Now we can see that each MatchInfo object has a Line property and it’s reasonable to guess that this contains a Line containing a match. Taking a look:


Is not the true Tao, until

Bringing together everything we’ve seen so far, we pull out the Line property of each element in the array as follows

$mymatches | Foreach-Object {$_.Line}

Alternatively, we can ditch the $mymatches variable and pipe in the output of Select-String directly

Select-String the haiku.txt -CaseSensitive | ForEach-Object {$_.Line}

Is not the true Tao, until
and the presence of absence:

Regular expressions

select-string 's*is' haiku.txt        # * Zero or more of preceding token
select-string 's+is' haiku.txt        # + On or more of preceding token
select-string '.nd'  haiku.txt        # . Any token followed by 'nd'
select-string 'es'   haiku.txt        # matches 'es'
select-string 'es[ht]' haiku.txt      # Exactly one of the characters listed
select-string 'es[^ht]' haiku.txt     # Matches everything except h and t
select-string 'ex[
select-string '\bis\b' haiku.txt      # \b word boundaries

Input and output redirection

> redirects output (AKA standard output). This works in both Bash and Powershell scripts. For example, in Bash we might do

grep -r not * > found_nots.txt

Drawing on what we’ve learned so far, you might write the PowerShell version of this command as

get-childitem *.txt -recurse | select-string not > found_nots.txt

However, if you do this, you will find that the script will run forever with the hard-disk chugging like crazy. If you’ve run the above command, CTRL and C will stop it. This is because Powershell is including the output file, found_nots.txt, in its input which leads to an infinite loop. To prevent this, we must explicitly exclude the output file from the get-childitem search

get-childitem *.txt -Exclude 'found_nots.txt' -recurse | select-string not > found_nots.txt

cat found_nots.txt
ls *.txt > txt_files.txt
cat txt_files.txt

In Linux, < redirects input (AKA standard input). This does not work in PowerShell:

cat < haiku.txt
At line:1 char:5
+ cat < haiku.txt
+     ~
The '<' operator is reserved for future use.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : RedirectionNotSupported

The above is a forced use of < since one could simply do

cat haiku.txt

Recall that cat is an alias for get-content. The use of get-content is an idiom that gets around the lack an < operator. For example, instead of

foo < input.txt

One does

get-content input.txt | foo

Error messages are output on standard error

ls idontexist.txt > output.txt  
cat output.txt                  #output.txt is empty
ls idontexist.txt 2> output.txt               # 2 is standard error
ls haiku.txt 1> output.txt                    # 1 is standard output
ls haiku.txt,test_file.txt 2>&1 > output.txt  # Combine the two streams.

Searching for files

# Find all     
UNIX: find .
PS: get-childitem .  -Recurse 
PS: get-childitem .  -Recurse | foreach-object {$_.FullName}    #To give identical output as `find`   

To save on typing, you can use the alias gci instead of get-childitem

# Directories only
UNIX: find . -type d        
PS2: gci . -recurse | where { $_.PSIsContainer }
PS3: gci -recurse -Directory 

If you have PowerShell 2, you can only use the long winded version. It’s simpler in PowerShell 3. Similarly for searching for files only.

# Files only
UNIX: find . -type f          
PS2: get-childitem -recurse | where { ! $_.PSIsContainer }
PS3: gci -recurse -File

With the Unix find command, you can specify the maximum and minimum search depths. There is no direct equivalent in PowerShell although you could write a function that will do this. Such a function can be found at although I have not tested this!

# Maximum depth of tree
UNIX: find . -maxdepth 2
PS : No direct equivalent
# Minimum depth of tree
UNIX: find . -mindepth 3 
PS : No direct equivalent

You can also filter by name. Confusingly, PowerShell offers two ways of doing this. More details on the differences between these can be found at

One key difference between find and get-childitem is that the latter is case-insenstive whereas find is case sensitive.

# Find by name
UNIX: find . -name '*.txt' 
PS: gci -recurse -include *.txt
PS: gci -recurse -filter *.txt

#Find empty files
UNIX: find . -empty    
PS: gci -recurse | where ($_.Length -eq 0) | Select FullName 

#Create empty file
UNIX: touch emptyfile.txt
PS: new-item emptyfile.txt -type file

Command Substituion

In bash, you can execute a command using backticks and the result is substituted in place. i.e.

foo `bar`

The backticks are used as escape characters in PowerShell so you do the following instead

foo $(bar)

In both cases, the command bar is executed and result is substituted into the call to foo.

Power of the pipe

| is a pipe. Use the pipe to connect the output of one command to the input of another:

Count text files

ls -filter *.txt | measure

ls outputs a list of files, measure inputs a list of files.

echo "Number of .txt files:"; ls -filter *.txt | measure | select -expand count

; equivalent to running two commands on separate lines.

Question: what does this do?

ls -name | select-string s | measure

Answer: counts the number of files with s in their name.

history | select-string 'echo'

Power of well-defined modular components with well-defined interfaces,

  • Bolt together to create powerful computational and data processing workflows.
  • Good design principle applicable to programming – Python modules, C libraries, Java classes – modularity and reuse.
  • “little pieces loosely joined” – history + select-string = function to search for a command.


get-variable                            # See all variables
$MYFILE="data.txt"                      # Need quotes around strings
echo $MYFILE
echo "My file name is $MYFILE"
$num = 1                                #Numbers don't need quotes
$num = $num+1                           #Simple Arithmetic
$TEXT_FILES=get-childitem               Save output of get-childitem

Variables only persist for the duration of the current PowerShell Session

Environment variables

Windows environment variables don’t show up when you execute get-variable; to list them all you do

get-childitem env:                  #Show all Windows Environment variables 
echo $env:PATH                      #Show the contents of PATH
$env:Path = $env:Path + ";C:\YourApp\bin\"  #temporarily add a folder to PATH

This modification to PATH will only last as long as the current session. It is possible to permanently modify the system PATH but this should only be done with extreme care and is not covered here.

PowerShell Profile

The PowerShell profile is a script that is executed whenever you launch a new session. Every user has their own profile. The location of your PowerShell profile is defined by the variable $profile


Open it with

notepad $profile

Add something to it such as

echo "Welcome to PowerShell.  This is a message from your profile"

Restart PowerShell and you should see the message. You can use this profile to customise your PowerShell sessions. For example, if you have installed NotePad++, you might find adding the following function to your PowerShell Profile to be useful.

# Launch notepad++ using the npp command
function npp($file)
if ($file -eq $null)
        & "C:\Program Files (x86)\Notepad++\notepad++.exe";
        & "C:\Program Files (x86)\Notepad++\notepad++.exe" $file;

With this function in your profile, you can open Notepad++ with the command npp or npp(filename.txt)


$num = 1
if($num -eq 1)
  write-host 'num equals 1'

if($word -eq "hello")
  write-host 'The same'

By default, string tests are case insensitive

if($word -eq "HELLO")
  write-host 'The same'

To force them to be case sensitive, add a c to the operator:

if($word -ceq "HELLO")
  write-host 'The Same. This won't be printed'

You can similarly be explicitly case insensitive by adding an i. Although this is the the same behaviour as the undecorated operators and so might seem unnecessary, it shows your intent to the reader.

    if($word -ieq "HELLO")
      write-host 'The same'
Comparison Operators
-eq Equal to
-lt Less than
-gt Greater than
-ge Greater than or equal to
-le Less than or equal to
-ne Not equal to
Logical operators
-not    Not
!       Not
-or     Or
-and    And


PowerShell has several looping constructs. Here, I only consider two.


Allows you to run a block of code a set number of times.

for ($i=1; $i -le 5; $i=$i+1)
   Write-Host $i


Do something with every element of a collection

foreach($item in $(ls *.txt)) {echo $item.Name}

foreach vs foreach-object


Shell scripts

  • Save retyping.
  • PowerShell scripts have the extension .ps1
  • PowerShell scripts must be created with plain text editors such as Notepad or Notepad++. NEVER use Microsoft Word!

Here is a very simple script

notepad protein_filter.ps1                  #Open the file

#A simple protein filter
$DATE = get-date
echo "Processing date: $DATE"

foreach($item in get-childitem *.pdb)
   echo $item.Name

echo "Processing complete"

To run this just type the filename:


If you get an error message, it may be because your execution policy is set not to run scripts locally. Change this with the command

Set-ExecutionPolicy RemoteSigned            #Allow local scripts to run.  Needs to be run with admin privileges
protein_filter.ps1                          #Run the script

Download files via command-line

Primary Care Trust Prescribing Data – April 2011 onwards

$path="$pwd\$file"   #Path needs to be fully qualified. This puts it in the current folder
$url = "$file"
$client = new-object System.Net.WebClient
$client.DownloadFile( $url, $path )


Windows file permissions are rather more complicated than those of Linux but most users won’t need to worry about them in day to day use.

The call operator

It is sometimes convenient to construct strings that contain the full path to a PowerShell script we want to execute. For example:

$fullpath = "$pwd\myscript.ps1"

To actually run the script pointed to by this variable, you need to use the call operator &

& $fullpath                 #Runs myscript.ps1

You also need to do this if you try to call any script where the path contains spaces

"C:\Program Files\myscript.ps1"             #Will just display the string literal
& "C:\Program Files\myscript.ps1"           #Runs the script

Background Jobs

Consider the script counter.ps1

#counter.ps1: A simple, long running job to demonstrate background jobs


while ( $i -lt 200000 )
    echo $i

This counts up to 200000 in user-defined steps.

./counter.ps1   > 1step.txt                 #Counts in steps of 1
./counter.ps1 -step 2 > 2step.txt           #Counts in steps of 2

The script takes quite a while to complete and you cannot do anything else in your PowerShell session while it is working. Using the start-job Cmdlet, we can run counter.ps1 in the background

start-job -scriptblock { C:\Users\walkingrandomly\Dropbox\SSI_Windows\dir_full_of_files\some_directory\counter.ps1 >  C:\Users\walkingrandomly\Dropbox\SSI_Windows\dir_full_of_files\some_directory\outcount1.txt }

Note that you have to use the full path to both the counter.ps1 script and the output file. You can see the status of the job with get-job


Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1              Job1             Running    True            localhost             C:\Users\walkingrando...

Eventually, your job will complete


Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1               Job1            Completed  False           localhost             C:\Users\walkingrando...

ls outcount*                    #Ensure that output file has been created
remove-job 1                    #remove remnants of job 1 from the queue
get-job                         #Check that queue is empty

You can run as many simultaneous jobs as you like but it is best not to run too many or your computer will grind to a halt.

Here’s an example that runs 5 counter.ps1 jobs concurrently

#Runs 5 instances of counter.ps1 in parallel

$scriptname     = "counter.ps1"
$outputfileBase = "outfile"
$outputfileExt  = ".txt"

$scriptPath = "$pwd\$scriptname"

for ($i=1; $i -le 5; $i++)
  $outputfilePath = "$pwd\$outputfileBase" + $i + $outputfileExt
  $command = "$scriptPath -step $i `> $outputfilePath"
  $myScriptBlock = [scriptblock]::Create($command)
  start-job -scriptblock $myScriptBlock

Run this as a demonstration

get-job                             #Keep running until all have completed
ls outfile*
more outfile5.txt
more outfile2.txt
$myjob=get-job 2                    #Get info on job Id 2 and store in variable $myjob
$myjob.Command                      #Look at the command that comprised job 2
remove-job *                        #Remove all job remnants from the queue
get-job                             #Should be empty

TODO: Dealing with output, recieve-job

Secure Shell

There is no equivalent to the Linux commands ssh and sftp in PowerShell. The following free programs are recommended


There are no direct PowerShell equivalents to zip, unzip, tar etc. There are write-zip, write-tar and write-gzip cmdlets in the third party, free PowerShell Community Extensions but I have not investigated them yet.


Start-transcript Initializes a transcript file which records all subsequent input/Output. Use the following syntax:

Start-Transcript [[-path] FilePath] [-force] [-noClobber] [-append]

Stop-transcript Stops recording and finalizes the transcript.

start-transcript -path ./diary.txt
echo "Hello dear diary"
cat diary.txt
  • Record commands typed, commands with lots of outputs, trial-and-error when building software.
  • Send exact copy of command and error message to support.
  • Turn into blog or tutorial.

Shell power

(Bentley, Knuth, McIlroy 1986) Programming pearls: a literate program Communications of the ACM, 29(6), pp471-483, June 1986. DOI: [10.1145/5948.315654].

Dr. Drang, More shell, less egg, December 4th, 2011.

Common words problem: read a text file, identify the N most frequently-occurring words, print out a sorted list of the words with their frequencies.

10 plus pages of Pascal … or … 1 line of shell

#BASH version
$ nano
tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q
$ chmod +x
$ nano <
$ nano < 10

The PowerShell version is more complicated but still very short compared to the 10 pages of Pascal

$text=get-content $filename
$split = foreach($line in $text) {$line.tolower() -split "\W"}
$split | where-object{-not [string]::IsNullorEmpty($_)} |  group -noelement | sort Count -Descending | select -first $num 10

“A wise engineering solution would produce, or better, exploit-reusable parts.” – Doug McIlroy