Running Squish tests and uploading results to Test Center

Introduction¶

It’s very important to make automated GUI tests a part of a Continuous Integration (CI) process. Although we don't offer an addon for GitLab, it can be still integrated with Squish because we support a command-line interface for all of our tools. Moreover, in this solution, the results of tests are pushed to Test Center to analyze there.

GitLab and Test Center¶

This article demonstrates how to run Squish tests in GitLab. Squish will execute multiple test suites and upload results to Test Center. No artifacts are produced, the pipeline log will contain the link to view results directly in Test Center. The pipeline id is used as a batch name for results upload.

Pipeline configuration¶

The next step is to define a pipeline in the file .gitlab-ci.yml. Pipelines are defined by specifying jobs that run in stages. In our example, we define a job (run in a test stage) for each executed test suite. The --exitCodeOnFail switch is used for squishrunner in order to give the job fail status if any test has failed. Because each test suite is executed in a separate job, all suites are executed even if one of them fails.

In order to send results to Test Center the testcenter report generator is used for squishrunner. The parameter for this report generator contains information about Test Center installation, project, upload token, labels and batch. You can read more here about grouping results in Test Center.

The presented example is for GitLab runner of shell type (powershell) for Windows. Other shells or platforms require modifications. On top of that, SQUISH_RUNNER_BIN, TESTCENTER_HOST, TESTCENTER_PROJECT, and TESTCENTER_TOKEN need to be adopted.

stages:          # List of stages for jobs, and their order of execution
  - test

variables:
  SQUISH_RUNNER_BIN: "D:/Squish/Qt/Squish7.0.1-Qt6.2/bin/squishrunner.exe"
  TESTCENTER_HOST: "https://eimsbuettel.froglogic.com"
  TESTCENTER_PROJECT: "pawtom-demo"
  TESTCENTER_BATCH: $CI_PIPELINE_ID
  TESTCENTER_TOKEN: "foorp4unfxlXpImb123c0VjriqjXD99O8up1bar-foo"
  REPORTGEN: "testcenter,${TESTCENTER_HOST}/project/${TESTCENTER_PROJECT}?token=${TESTCENTER_TOKEN}&label=os%3Dwin&batch=${TESTCENTER_BATCH}"

default:
  before_script:
    - $env:Path = 'D:/Squish/Qt/Squish7.0.1-Qt6.2/bin/;' + $env:Path 
    - echo "TESTCENTER_PROJECT = $TESTCENTER_PROJECT" 
    - echo "TESTCENTER_HOST = $TESTCENTER_HOST"
    - echo "TESTCENTER_BATCH = $TESTCENTER_BATCH"

  after_script:  
    - echo "${TESTCENTER_HOST}/Testcenter/explore?&project0=${TESTCENTER_PROJECT}&batch0=${TESTCENTER_BATCH}"

squish-test-job-suite-py:  
  stage: test    
  
  script:
    - echo "Running suite_py ... "
    - squishrunner.exe --testsuite D:\\suites\\examples\\qt\\addressbook\\suite_py  --exitCodeOnFail 13 --reportgen "$REPORTGEN"

squish-test-job-suite-bdd-py:  
  stage: test    
  
  script:
    - echo "Running suite_bdd_py ... "
    - squishrunner.exe --testsuite D:\\suites\\examples\\qt\\addressbook\\suite_bdd_py  --exitCodeOnFail 13 --reportgen "$REPORTGEN"

.gitlab-ci.yml

Executing the Example¶

We introduced a fail verification for suite_py for demo purpose.

[OK]Running with gitlab-runner 15.2.1 (32fc1585)
[OK]  on LENOVO-TOMASZ NHueiFPj
section_start:1660820566:resolve_secrets
[OK][OK]Resolving secrets
section_end:1660820566:resolve_secrets
[OK]section_start:1660820566:prepare_executor
[OK][OK]Preparing the "shell" executor
[OK]Using Shell executor...
section_end:1660820566:prepare_executor
[OK]section_start:1660820566:prepare_script
[OK][OK]Preparing environment
Running on LENOVO-TOMASZ...
section_end:1660820567:prepare_script
[OK]section_start:1660820567:get_sources
[OK][OK]Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in D:/Apps/Gitlab-Runner/builds/NHueiFPj/0/squish1/AddressBookGUITests/.git/
Checking out 9f47e370 as main...
git-lfs/3.1.4 (GitHub; windows amd64; go 1.17.8)

Skipping Git submodules setup
section_end:1660820571:get_sources
[OK]section_start:1660820571:step_script
[OK][OK]Executing "step_script" stage of the job script
 $env:Path = 'D:/Squish/Qt/Squish7.0.1-Qt6.2/bin/;' + $env:Path
 echo "TESTCENTER_PROJECT = $TESTCENTER_PROJECT"
TESTCENTER_PROJECT = pawtom-demo
 echo "TESTCENTER_HOST = $TESTCENTER_HOST"
TESTCENTER_HOST = https://eimsbuettel.froglogic.com
 echo "TESTCENTER_BATCH = $TESTCENTER_BATCH"
TESTCENTER_BATCH = 616699827
 echo "Running suite_py ... "
Running suite_py ... 
 squishrunner.exe --testsuite D:\\suites\\examples\\qt\\addressbook\\suite_py  --exitCodeOnFail 13 --reportgen "$REPORTGEN"
2022-08-18T13:02:52	START     	Start 'suite_py'              	Test 'suite_py' started
2022-08-18T13:02:52	START_TEST_CASE	Start 'tst_general'           	Test 'tst_general' started (tst_general)
msg 'Import ID: 130811'
2022-08-18T13:02:55	PASS      	Comparison                    	'125' and '125' are equal
2022-08-18T13:02:56	PASS      	Comparison                    	'126' and '126' are equal
2022-08-18T13:02:57	PASS      	Comparison                    	'Jane' and 'Jane' are equal
2022-08-18T13:02:57	PASS      	Comparison                    	'Doe' and 'Doe' are equal
2022-08-18T13:02:57	PASS      	Comparison                    	'jane.doe@nowhere.com' and 'jane.doe@nowhere.com' are equal
2022-08-18T13:02:57	PASS      	Comparison                    	'123 555 1212' and '123 555 1212' are equal
2022-08-18T13:02:57	PASS      	Comparison                    	'125' and '125' are equal
2022-08-18T13:02:57	END_TEST_CASE	End 'tst_general'             	End of test 'tst_general'
2022-08-18T13:02:59	START_TEST_CASE	Start 'tst_adding'            	Test 'tst_adding' started (tst_adding)
2022-08-18T13:02:59	FATAL     	Evaluation error              	inconsistent use of tabs and spaces in indentation
2022-08-18T13:02:59	END_TEST_CASE	End 'tst_adding'              	End of test 'tst_adding'
2022-08-18T13:02:59	START_TEST_CASE	Start 'tst_adding_data'       	Test 'tst_adding_data' started (tst_adding_data)
2022-08-18T13:03:00	PASS      	Verified                      	True expression
2022-08-18T13:03:00	PASS      	Comparison                    	'Aaiyasha' and 'Aaiyasha' are equal
2022-08-18T13:03:00	PASS      	Comparison                    	'Cheatham' and 'Cheatham' are equal
2022-08-18T13:03:00	PASS      	Comparison                    	'aaiya.cheatha@shearer.net' and 'aaiya.cheatha@shearer.net' are equal
2022-08-18T13:03:00	PASS      	Comparison                    	'543 709 9479' and '543 709 9479' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'Abdul-Wahhab' and 'Abdul-Wahhab' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'Dickie' and 'Dickie' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'abdu.dickie@grissom.com' and 'abdu.dickie@grissom.com' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'591 326 1378' and '591 326 1378' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'Aditya' and 'Aditya' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'Crisp' and 'Crisp' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'ad.crisp@beadsworth.net' and 'ad.crisp@beadsworth.net' are equal
2022-08-18T13:03:01	PASS      	Comparison                    	'565 741 2692' and '565 741 2692' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'Adora' and 'Adora' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'Hay' and 'Hay' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'adora.hay@corless.com' and 'adora.hay@corless.com' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'582 381 4345' and '582 381 4345' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'Ahmet' and 'Ahmet' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'Boardus' and 'Boardus' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'a.board@oldenburg.com' and 'a.board@oldenburg.com' are equal
2022-08-18T13:03:02	PASS      	Comparison                    	'538 161 3394' and '538 161 3394' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'Aillie' and 'Aillie' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'Lant' and 'Lant' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'ailli.lant@harkins.net' and 'ailli.lant@harkins.net' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'579 102 6482' and '579 102 6482' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'Alanah' and 'Alanah' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'Dewar' and 'Dewar' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'a.dewar@handfield.eu' and 'a.dewar@handfield.eu' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'555 137 3609' and '555 137 3609' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'Alea' and 'Alea' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'Cooper' and 'Cooper' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'alea.cooper@breedlove.net' and 'alea.cooper@breedlove.net' are equal
2022-08-18T13:03:03	PASS      	Comparison                    	'529 792 9823' and '529 792 9823' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'Amy' and 'Amy' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'Mcmillan' and 'Mcmillan' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'amy.mcmillan@robb.eu' and 'amy.mcmillan@robb.eu' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'588 226 5176' and '588 226 5176' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'Aron' and 'Aron' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'Grieve' and 'Grieve' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'ar.grieve@mulkey.eu' and 'ar.grieve@mulkey.eu' are equal
2022-08-18T13:03:04	PASS      	Comparison                    	'573 879 8988' and '573 879 8988' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'Aston' and 'Aston' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'Rob' and 'Rob' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'aston.rob@uren.net' and 'aston.rob@uren.net' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'527 859 2438' and '527 859 2438' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'Awais' and 'Awais' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'Rohrbach' and 'Rohrbach' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'aw.rohrbach@bowser.com' and 'aw.rohrbach@bowser.com' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'550 500 3059' and '550 500 3059' are equal
2022-08-18T13:03:05	PASS      	Comparison                    	'12' and '12' are equal (table contains as many rows as added data)
2022-08-18T13:03:05	END_TEST_CASE	End 'tst_adding_data'         	End of test 'tst_adding_data'
2022-08-18T13:03:06	START_TEST_CASE	Start 'tst_addLoggingWrapper' 	Test 'tst_addLoggingWrapper' started (tst_addLoggingWrapper)
2022-08-18T13:03:08	LOG       	Logged function called        	
2022-08-18T13:03:09	LOG       	File Open clicked             	
2022-08-18T13:03:10	PASS      	Verified                      	True expression
2022-08-18T13:03:10	LOG       	Logged function called        	
2022-08-18T13:03:10	PASS      	Verified                      	True expression
2022-08-18T13:03:10	LOG       	Logged function called        	
2022-08-18T13:03:11	PASS      	Comparison                    	'Jane' and 'Jane' are equal
2022-08-18T13:03:11	PASS      	Comparison                    	'Doe' and 'Doe' are equal
2022-08-18T13:03:11	PASS      	Comparison                    	'jane.doe@nowhere.com' and 'jane.doe@nowhere.com' are equal
2022-08-18T13:03:11	PASS      	Comparison                    	'555 123 4567' and '555 123 4567' are equal
2022-08-18T13:03:11	PASS      	Verified                      	True expression
2022-08-18T13:03:11	LOG       	Logged function called        	
2022-08-18T13:03:11	END_TEST_CASE	End 'tst_addLoggingWrapper'   	End of test 'tst_addLoggingWrapper'
2022-08-18T13:03:12	START_TEST_CASE	Start 'tst_clickButton_wrapper'	Test 'tst_clickButton_wrapper' started (tst_clickButton_wrapper)
2022-08-18T13:03:13	LOG       	clickButton() called          	
2022-08-18T13:03:14	LOG       	File Open clicked             	
2022-08-18T13:03:15	PASS      	Verified                      	True expression
2022-08-18T13:03:15	LOG       	clickButton() called          	
2022-08-18T13:03:16	PASS      	Verified                      	True expression
2022-08-18T13:03:16	LOG       	clickButton() called          	
2022-08-18T13:03:16	PASS      	Verified                      	True expression
2022-08-18T13:03:16	PASS      	Comparison                    	'Jane' and 'Jane' are equal
2022-08-18T13:03:16	PASS      	Comparison                    	'Doe' and 'Doe' are equal
2022-08-18T13:03:16	PASS      	Comparison                    	'jane.doe@nowhere.com' and 'jane.doe@nowhere.com' are equal
2022-08-18T13:03:16	PASS      	Comparison                    	'555 123 4567' and '555 123 4567' are equal
2022-08-18T13:03:16	LOG       	clickButton() called          	
2022-08-18T13:03:16	END_TEST_CASE	End 'tst_clickButton_wrapper' 	End of test 'tst_clickButton_wrapper'
2022-08-18T13:03:18	END       	End 'suite_py'                	End of test 'suite_py'
section_end:1660820598:step_script
[OK]section_start:1660820598:after_script
[OK][OK]Running after_script
Running after script...
 echo "${TESTCENTER_HOST}/Testcenter/explore?&project0=${TESTCENTER_PROJECT}&batch0=${TESTCENTER_BATCH}"
https://eimsbuettel.froglogic.com/Testcenter/explore?&project0=pawtom-demo&batch0=616699827
section_end:1660820599:after_script
[OK]section_start:1660820599:cleanup_file_variables
[OK][OK]Cleaning up project directory and file based variables
section_end:1660820599:cleanup_file_variables
[OK]ERROR: Job failed: exit status 1

pipeline output

More articles about Squish and GitLab integration

Introduction¶

GitLab and Test Center¶

Pipeline configuration¶

Executing the Example¶

Related Information¶

Usage Data Processing by Google