Automation of Matlab on Linux can be a bit tricky, because Matlab does not behave like a regular Java Application.
For the start you will need to configure the JRE used by Matlab for Squish to use as you would with any other Java Application.
But after that additional steps are needed. Matlab must be started with the -desktop argument to allow it to start its UI from a process that has no terminal, otherwise Squish will just sit there and hook the application based on the splash screen, but the real UI will never be shown.
The shell script provided below can be used for that. In this case the script must be used as the AUT.
USAGE : run_matlab.sh MATLAB_BIN_DIRECTORY startApplication("run_matlab.sh MATLAB_BIN_DIRECTORY")
with this the setup required is done. Matlab will still require time to show the UI after the initial hookup, so you should insert code to synchronize for the appearance of the main window.