Image comparison modes in screenshot verification points

Skip to end of metadata
Go to start of metadata

Table of Contents

Overview

Obvious reasons for screenshot differences

Taking screenshots of the same object or window can result in larger or minor pixel differences between the screenshots taken at different times. Here is a list of possible reasons:

  • The application or the respective object may have changed appearance in general.
  • The application may be displaying constantly changing things, like the date or time.

Unknown reasons for screenshot differences

In addition to these above reasons for screenshot differences, there seem to be other reasons for such differences.

These differences tend to be so small that when comparing screenshots the human eye cannot see them or modern monitors are not able to display the differences. Typically these differences are just a few pixels, and the individual pixel values typically only differ by a very small amount.

Unfortunately we do not know the cause of these differences. The following things are suspected to contribute to the problem:

  • Anti-aliasing
  • Graphics card driver in general
  • Color management of the graphics card driver
  • Color management of the monitor
  • Lighting of the monitor surroundings in case of monitors with automatic brightness adjustment

Possible solutions

Squish supports different comparison modes for screenshot verification points ("Strict", "Pixel", "Histogram").

The comparison mode of a screenshot verification point can only be modified after it has been created. (The default comparison mode is "Strict".)

Editing the comparison mode

To edit the comparison mode open the screenshot verification point in the Squish IDE:

Then click on the button "Edit Verification Point" to see the configuration tool and click on the tab "Comparison Mode". Here you can choose the desired mode, configure it and save the changes to the verification point.

Comparison modes

Strict mode

The "Strict" mode is a pixel by pixel comparison which offers no configuration options:

Note that pixel differences can occur which are not noticeable by the human eye due to various factors (graphics card drivers and hardware, operating system, anti-aliasing, etc.) which makes the Strict mode unsuitable at times.

Pixel (fuzzy) mode

This mode compares the color of corresponding pixels of two images with the same dimensions.

It presents the images differences in percentage. For example, a difference of 10% means that 10% of the pixels are different.

The algorithm has a threshold parameter that defines the criterion by which images are considered to be equal. Therefore, for the example of 10% different pixels, a threshold of 11% would consider both images to be equal:

The Max. Color Difference parameter is explained here.

Normalized color histogram mode

This mode is based on comparing image color histograms. It is useful for cases when images are resized/scaled or rotated, so that their colors remain the same or are not changed much:

How it works:

  • Divide the color range (0-255) of each color component (RGB) of every pixel by the number of bins (or baskets) and calculate the number of pixels that fall into each bin based on their color characteristics.
  • Divide the total number of pixels in the image by the number of pixels in each bin to get "normalized values" (which allows comparison even if they have different sizes). These respective values are put back into the corresponding bins.
  • The values of all corresponding bins are subtracted from another and the resulting values summed up. This value represents the difference of the images.

This mode allows to configure the number of bins and the threshold, which highest difference between the values of both images to consider them to be "equal".

Correlation Mode

See Correlation Comparison Mode in the Squish manual.

The Max. Color Difference Parameter

The Max. Color Difference parameter is useful when you compare images that have different shades (not detectable by eye). This can happen to screenshots that are taken on machines that have different video adapters. The screenshots look very similar and you want to ignore tiny color differences they have.

Technically speaking, Max. Color Difference is the distance between two colors in 3D (RGB) color space (also see Euclidean distance).

For example you have two pixels with the following colors: (0, 0, 0) and (1, 1, 1) - one is absolutely black and another is almost black (you cannot distinguish their colors by eye). The difference between these two colors can be calculated with the following formula:

Here r, g and b are red, green and blue color component values. So for our example the calculation would be:

Also, the color difference between black and white colors is ~441.67. So, if you want to compare solid black image with white one and set the "Max Color Difference" to be 441.67, result will be positive, i.e. images match.

In the Test Results view you may find this additional information in failed screenshots:

...(difference: 0.7976%, max. color difference: 80.6040)

The "difference: 0.7976%" part denotes the number of pixels that are different in percent. In this example it is 0.7976% and not 79.76%, i.e. the relative number is very small, less than one percent. For example, if you compare images of 10x10px, and only one pixel is different, you will get 1% difference.

The "max. color difference: 80.6040" part denotes the largest pixel color difference - in this case 80.6040. So if you set "Max Color Difference" to be 81, your test will pass, because all the pixels that failed previously will be considered equal then.

Related Information

Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.