A filter set in a file dialog is working fine on Windows if the application is run standalone but does not work if run as an AUT in Squish.
Squish disables native file dialogs (see Automating native or Qt (file) dialogs in Qt applications).
Therefore you must use the filter syntax that is supported by
QFileDialog instead of relying on a platform-dependent syntax:
Example of a platform-dependent filter (Win32 separates filters using a semicolon):
"Text Documents (*.txt;*.doc)"
Example of a filter suitable for QFileDialog (Qt uses space as a separator):
"Text Documents (*.txt *.doc)"