Measuring code coverage of ASP.NET Web application using Squish Coco & tests run by Squish for Web¶
This article demonstrates measuring code coverage of an ASP.NET (C#) Web application using Squish Coco. Additionally, automated tests are executed by Squish for Web, measuring the code coverage of each GUI automated test. The code coverage insight enables teams to improve their automated test suite by removing tests that don't increase testing coverage, and identifying areas that remain untested. Similar achievements can be made using other web automation tests, or even through manual testing. The test execution order, which indicates how to best maximize the code coverage can be calculated after each execution. The calculation reveals a test sequence in which tests with the highest code coverage appear first. This information is necessary when picking good test candidates for a subset of automated tests to be run in a limited time (i.e. sanity/brash automated tests).
Web application instrumentation¶
The example application used to demonstrate this solution is a simple ASP.NET MVC 5 Web application designed using C# in Visual Web Developer. MVC stands for model-view-controller, and is a pattern for developing applications containing: models (classes that represent the application data), views (templates which generate an HTML response) and controllers (classes that handle incoming browser requests, retrieve model data, and specify templates view which return a response to the browser). The code coverage analysis will reveal how each part is used during the execution of single automated test.
The application, Frog Directory, is run in Visual Studio on a localhost, but can also be deployed and run on a remote host. Frog Directory maintains a directory of frogs, storing information such as species, date of birth, gender and price. The application offers functionality to create, edit and delete entries and offers, as well as provides filtering and searching capabilities.
During the Squish Coco installation the original C# compiler is replaced by the CoverageScanner. The CoverageScanner instruments the source code, which enables the code coverage measurement capabilities, after which it calls the original compiler to execute the application, leaving the source code untouched. To activate the instrumentation in Visual Studio project, go to Project Properties, click the Build tab and enter
COVERAGESCANNER_COVERAGE_ON in the Conditional compilation symbols box.
After each build, a Squish Coco instrumentation file (
.csmes) is generated. In the example application, Frog Directory, the file is called
FrogDirectory.dll.csmes and is located in the
obj\Release subdirectory. Double click the file to inspect it in the CoverageBrowser. Currently no coverage statistics are available in the CoverageBrowser because the application has yet to be executed.
Squish for Web integration¶
In order to generate any code coverage we need to perform some actions on the running (and previously instrumented) application. Actions can be performed manually or using automation (i.e. running unit tests or automated GUI tests). GUI automated tests for the ASP.NET application can be designed using Squish for Web. To integrate Squish for Web and Squish Coco we need to add a small segment of code to the automated tests and the Web application.
init function, which is called at the beginning of each test. This function passes information to the Web application by calling a loadUrl function indicating the test case has started.
cleanup function, which is called at the end of each test. This function passes information to the Web application indicating the given test case has ended, and with a given status (PASSED or FAILED). Place both functions in the shared test suite resources, ensuring they are called for every test case in the test suite.
An additional C# function needs to be implemented in the Web application to handle requests from the
cleanup functions. The
init function causes the C# CoverageScanner library function
__coveragescanner_clear() to be called, reseting the status of all instrumentations. The
cleanup function causes the
__coveragescanner_testname(id) to be called, which sets the name of the test case currently being executed. After which the
__coveragescanner_teststate is called, and sets the result of executed test to
__coveragescanner_save() executes and saves the execution report using the filename previously set in the
__coveragescanner_filename. Using this method of communication (loading URL's with a defined syntax) between Squish for Web and Squish Coco, allows Squish for Web and an ASP.NET Web server to on different hosts. The Squish Coco execution report is stored on the host (in a TEMP directory) where Web server is running.
Code Coverage Measurement analysis¶
Next we'll run three test cases (tst_addFrog, tst_filterFrog and tst_searchFrog) and load an execution report (
csexe file located in %TEMP% directory) in the CoverageBrowser (select File > Load Execution Report). The CoverageBrowser analyzes what code executed when the automated tests ran. It also offers Line Coverage, Function Coverage and Branch, Decision and Condition Coverage and analysis.
To view the optimal execution order, select Tools > Optimized Execution Order. In the following example, the optimal order is tst_searchFrog and tst_filterFrog. From a code coverage perspective, the third test case, tst_addFrog, is redundant, and therefore is omitted from the report.
Configuring squishserver (see "--config setHardExitTimeout", "--config setSoftExitTimeout")