Most of the polemic against Matlab you will find on the web has a distinctly partisan feeling to it: '... you should code in my favorite language instead!' It is not clear that the authors have any actual experience with the language. I have used Matlab almost exclusively in my professional life for the last ten years, and feel qualified (and motivated) to write the gripes of my people, knowing full well I will wake up in the morning and code in it again. This is not a recitation of the gotchas and idiosyncracies of the language that burn novice users; I no longer notice them. These are the lamentations of an old hand. You, too, can experience these frustrations if you stick with Matlab.

A Walled Garden

Sure, Matlab is expensive (although the Home edition is 'only' 150 bucks as of this writing). Conditional, however, on you (or, more likely, your employer) having shelled out for a license, what are the implications of the pricetag?

For one, it might be hard to find help in an open source forum like Cross Validated, or Stack Overflow. It is not the case that an arbitrary user on one of these sites, one who might posess domain expertise in the problem you are trying to solve, has access to Matlab. Perhaps they do, but maybe you are using an older version of Matlab, or a toolbox which they have not bought, etc. There is a decidedly pro-FOSS feeling on these sites, and Matlab users tend to stick to the Matlab Answers stack, where the questions are generally practical, aimed at the novice user finding their way around the language, rather than theoretical, aimed at the expert user.

Because the Mathworks also sells toolboxes, the bundles of add-on code for specific domains, like statistics, signal processing, optimization, etc., they do not provide much support for (free) third party packages. Somewhat ominously, the Mathworks provides a 'File Exchange', where you can download such code. However, there is no tool for figuring out the prerequisite packages, installing those, automatically downloading the code, building it and testing the build. If you wanted to install package A, which depended on B, which dependend on C, you would have to read this in the documentation for each package, download them yourself, and manually build them.
Of course, this is actually an absurd counterfactual: I suspect there are very few packages that depend on other packages because the install process is entirely manual and error prone. Packages are essentially flat. This means if there is some common functionality used by many packages, each of the authors have reimplemented it themselves. All packages are islands.

Shoddy Code

I have been generally unimpressed with the quality of Matlab code written by the Mathworks, and shipped with Matlab or toolboxes. My favorite example of this is the stock code for computing the median. This routine actually sorts the data, then returns the middle value. This has been the wrong algorithm since the mid 70's. When I was 7, typing games into a Pet from Byte magazine, some hack in a windowless room was proably coding K-select in Fortran. Unfortunately, said hack coded in vain: the Mathworks took no notice, chosing instead the dumbest obviously correct algorithm.

This example has colored my perceptions, to be sure. But I have found numerous other examples where the Mathworks traded away efficiency or the vectorized idiom to gain obvious (often lazy) correctness. For example, the RSI code in the Financial toolbox is written to accept a vector of returns. A more idiomatic realization would accept an array of arbitrary dimensions, and would compute the RSI along a given dimension, much like e.g. the sort function, or the mean function, etc.

Part of the problem here is that I have high standards for code, while the Mathworks evidently has other priorities.

No multiline anonymous functions

I am a believer in literate programming. When I post research on my employer's internal blog, I like to include the code to repeat the analysis. This helps greatly when, years down the road, a colleague wants to go back and perform the analysis again. Much of this kind of work involves nonce functions--those which have no real lasting purpose outside that specific task. To make the analysis replicable, I either have to put the code into its own file (an '.m' file), and check that file into the repo, or I can make the function an anonymous function. The same dilemma applies if you want to ask for help on e.g. Stack Overflow, with the added constraint that you are unlikely to check your code into a permanent public repo. While asking for free advice on such a site, you have to ask a user to either copy your code to a file with a specific name so they can run it, or you must write your code as anonymous functions.

In Matlab, however, anonymous functions are highly restricted: they may consist of only a single statement, without assignment. They are equivalent to lambdas in Python. They source the namespace in which they were constructed, and cannot modify it.

My response to this limitation, over the years, has been to adapt to an awful programming idiom of chaining together several anonymous functions. I essentially write the multiline program backwards, and stick it in my script. As a silly example, suppose you wanted to regress some matrix against a vector, then normalize the regression coefficient by dividing by its norm. In Matlab you might write it like so:

betas = X \ y;
nbetas = betas ./ norm(betas,2);

Because anonymous functions cannot handle assignment, to put this into an single line anonymous function you could either compute the regression twice, or compose anonymous functions together, like so:

normalize = @(w)(w ./ norm(w,2))
norm_reg  = @(X,y)(normalize(X\y));

Note how the code is now written backwards. For the norm_reg function to find the normalize function, normalize has to be defined first, even though it is executed last. In reality, these often balloon to five or more anonymous functions chained together.

As a side note of interest to probably noone who reads this, you can overload the > operator for function handles by placing the following code into the file @function_handle/gt.m:

function C = gt(A,B);
% gt: overload '>' for function_handles
% C = A > B
% creates the composition of two functions; one would like to write this as 
% C = B o A
% but it is more natural to consider the output of A as being 'piped' to B.
c = @(varargin)(B(A(varargin{:})))

Then you could write the above mess as

normalize = @(w)(w ./ norm(w,2))
reg  = @(X,y)(X\y);
norm_reg  = reg > normalize
% or, more compactly:
norm_reg  = @(X,y)(X\y) > @(w)(w ./ norm(w,2))

This is inspired by, but nowhere near as elegant and flexible as, R pipes, via e.g. the pipeR package.

Welcome to my library

This is really a quibble in comparison, but the Matlab executable loads its own version of some dynamic libraries, which can stymie execution of C code compiled via the 'mex' system to use with Matlab. I have run into this when writing C code to read and write HDF5 files. Because the Matlab '.mat' file format is layered on top of the HD5 standard, Matlab itself links to a library. Depending on how old your Matlab install (or OS) is, this may not match what you want to use in your C code. The tie goes to the runner, in this case Matlab.

OO snap!

Matlab has two Object Oriented Programming systems. The pre-2009 system is supported in current releases for the sake of backwards compatibility. Some of us have never migrated. This system is an utter mess. For starters, you declare a class by putting a directory called @TheClassName in your matlab path, with a literal @ symbol. This was the Mathworks cleverly maintaining backwards compatibility, since no sane person would start a directory with an @. They repeated this trick in the 2009 release by changing the cursed symbol to a plus: +TheClassName. You are just not living until you have tried to get Subversion to play nicely with a directory called @TheClassName.

Beyond this oddity, the design of the OOP system means that one has to write around 5 to 8 functions, almost all entirely boilerplate, to get a fully working class. The most authoritative book on the topic provides a 'wizard' (on a CD!) to 'simplify' this process for you. Any programming language that requires a wizard to maintain boilerplate code has a problem. Because of how OO was bolted on to the base Matlab language as an afterthought, it is hard to achieve encapsulation of data, a sane public API to objects, and actual inheritence of methods and attributes, much less advanced OO concepts like virtual classes, friend classes, multiple inheritence, etc. Again, these issues have probably been resolved in the post 2009 OOP system, but I would not be surprised if they have not.

I would go so far as to say that the weak OO system, and the lack of tools for packaging code might contribute to the apparent 'lone nut in his basement' feel of many a Matlab codebase. This pattern consists of several dozen files dumped into a single directory.

Simply put, Matlab code is not made for sharing. The hurdles for sharing are many: the cost, lack of packaging tools and independent repository, and barriers to creating good APIs. This is not to say it is not a fine system when you know how to use it, have exacting standards for performance, and do not mind writing everything yourself. In that case, if your employer buys the license, have at it!