Should MATLAB have a validation suite?
Earlier this month I had a query from a colleague who had installed MATLAB on some very large and powerful workstations which are used by many researchers at our University. This colleague is a sysadmin of the old school and he doesn’t like installing any program that doesn’t have some sort of validation suite since he wants to be sure that the program is working OK on his hardware for his people to the best of his ability.
He can do this for the NAG library and he can do it for SAGE math (among others) but not, as far as he knew,for MATLAB. So he asked me the question “Does MATLAB have a validation suite?” and since I didn’t have a clue I passed it onto Mathworks Tech support who quickly informed us that the answer was “No.”
The essential gist of the reply was that they assume that MATLAB produces the same results on different machines except, of course, when it doesn’t!
Of course, people such as my colleague feel that ASSUMING that MATLAB produces the same result on different machines is not good enough which is why he wanted a validation suite in the first place. Life is short though and we didn’t pursue it further – partly because we didn’t have an example which proved Mathwork’s assumption wrong.
We do now though!
On a 64bit Windows 7 install of MATLAB 2009b we get the following (incorrect) behaviour:
p = [1,11-i;2,2;3,3;4,4;5,5;6,6;7,7;8,8;9,9;10,10;11,11]; sort(p) ans = 1.0000 2.0000 - 1.0000i 2.0000 3.0000 3.0000 4.0000 4.0000 5.0000 5.0000 6.0000 6.0000 7.0000 7.0000 8.0000 8.0000 9.0000 9.0000 10.0000 10.0000 11.0000 11.0000 11.0000
On the same machine but booted into 64bit Ubuntu 9.10 we get the correct behaviour:
p = [1,11-i;2,2;3,3;4,4;5,5;6,6;7,7;8,8;9,9;10,10;11,11]; sort(p) ans = 1.0000 2.0000 2.0000 3.0000 3.0000 4.0000 4.0000 5.0000 5.0000 6.0000 6.0000 7.0000 7.0000 8.0000 8.0000 9.0000 9.0000 10.0000 10.0000 11.0000 11.0000 11.0000 - 1.0000i
Is this exactly the type of bug that might have been caught in a validation suite? Would a user-runnable validation suite be useful?
Thanks to Michal Kvasnicka who alerted me to this bug which was first described in a discussion thread on MATLAB central. Mathworks are aware of the bug and have logged it as issue number 594221.
Should MATLAB have a validation suite? Well, is MATLAB just something fun to use or do lives depend on its reliability?
Therefore, the underlying issue is one of risk. And, if I may go (slightly) off topic, the possibility of the orthogonality of benefits versus consequences. Those getting the most benefits may not be those exposed to the greatest consequences. This can lead to a divergence of common interests.
For example, if you type into the MathWorks’ website search field the word “validation,” your top hit is a discussion of an automated way MathWorks can “validate” that you have a license to use their product. This makes sense in the MathWorks universe. The only benefit of MATLAB to MathWorks is the revenue stream generated by licensing. And so this has real consequences on how (and if) the software functions.
However, what did the readers of this post think when they read the title? That you were taking about some licensing validation issue?
I am not picking on MathWorks. AFAIK, they are a great company. I am just noting an inherent feature of proprietary software in general. At best, with proprietary software you get only what you think you pay for.
A final note. There is the consensus software engineering principle that the quality of software is dependent on the processes used to develop and maintain it. Therefore, the quality of MATLAB cannot be entirely determined simply by testing. Does MathWorks publish or audit its software development processes? What is their development organization’s “culture” toward software reliability?
I’m in two minds about your posts about bugs in large maths related programs like MATLAB. One the hand, especially as a software developer, I think it’s a little bit rude. On the other hand, it is amusing.
And there are larger issues of concern. MATLAB’s tracker isn’t public (you have to be registered to even browse it), which sucks. I think the fact that MATLAB don’t make their issue tracker public does make it less rude to talk about their bugs in public blogs.
And bugs like this (where the real part is sorted, but the imaginary part is just “left in place”), are kind of “omg how did that get through even the most cursory test?”. It’s hard to imagine that their test suite has even one test where they sort an array of complex numbers given that this bug evidently slipped through.
In the end people using MATLAB, or any piece of scientific software, should not be trusting its results absolutely. They should have some other way of at least checking the output is plausible, at least in a sort of “back of the envelope” fashion. If they don’t, they really are stuck.
Hi David
I’ve never thought of it as rude but it’s an interesting thought. What do other people think?
Best Wishes,
Mike
The other question is: Is anything better than MATLAB out there? The answer unfortunately is no.
In such complex projects like MATLAB there can be no single validation suite to test everything. How can you validate a very complex simulation in SIMULINK? The only reliable way of validation is the constant feedback from its users about errors and bugs.
A final comment, a lot of people are using MATLAB to generate code from simulation models that will be used in actual products, lets say something dangerous like an UAV. This code should be validated by its creators who are the people responsible and not Mathworks.
I don’t think it is rude at all, I also did my fair share of programming. But I think it is a good thing, especially if it is a fundamental error. Which can have a lot of effect on other functions. If Matlab doesn’t have a public tracker, then the end-user has to make their own; i’d say: good job!
@GG – Is anything better than MATLAB? Well that depends on your problem domain. For some problems I completely agree with you and yet for others I completely disagree. You are right in that we should be validating our own code but there is an expectation that the fundamental building blocks will be working OK. The sort function isn’t exactly rocket science!
@David – If there were bugs in our C or Fortran compilers then would you want to know about them? Why should MATLAB or Mathematica be any different? If you suspect a bug then it is highly likely that you will google for it. If no one talks about them then what do you expect to find?
I have a lot of respect for companies such as The Mathworks, Wolfram Research and Maplesoft and think that they all do some extremely cool stuff. Like all of us, however, they make mistakes and those mistakes can affect people’s work.
Walking Randomly is a blog about maths and math software so is it not reasonable to expect to see both the good and the bad?
Yes, the full validation is very probably unrealistic request, because of MATLAB extremely complexity.
But, the MATLAB math-core must be always highly reliable and credible, because users are not able to check out again and again that all low-end math function working still properly as at previous versions.
And finally, if there are any serious bugs (just like few weeks ago; R2009b … http://www.walkingrandomly.com/?p=1964), MathWorks should respond by immediately releasing of the fix-package and not by postponing the fix to the next version!?
I am really expecting quick respond and conceptual solution of these serious bugs, because this kind of action is most important evidence that MathWorks take care of his customers and that MATLAB is credible and trustworthy computing environment. Otherwise, Matlab will no longer remain the industry standard for numerical calculations, but only toy for students.
For some things, Sage (which includes scipy, numpy, and much much more) can be a viable replacement for Matlab. As an open-source project, it also improves faster as more people use it (and some fraction become developers). Give it a try: http://www.sagemath.org.
Sorry for commenting on a 6-month old article, but I am slowly reading all the archives from the first post, and I really wanted to comment on two points.
I am an Octave user, and I have tried to get some of my colleagues to try it out (they use Matlab). The main reason many of them won’t even consider it is because they think Octave must be full of bugs, whereas Matlab, being developed by professionals, and so expensive, surely is worth of their trust. Since Matlab’s bug tracker is not public, openly talking about bugs may help convince people not to be so trusting. By the way, Octave has a testsuite — incomplete but growing.
The second thing I wanted to talk about is the story of John Cooley. He got into a lot of (legal) trouble for openly talking about bugs in very expensive software, and for encouraging others to do the same. John held his ground and won the battle, and the result is better software and better software companies. The situation with Matlab is obviously not as dire, but I think the benefits of open discussion trump any other consideration (such as rudeness). Story here: http://www.deepchip.com/esnug_origins.html
Thanks!