Measuring code coverage using Squish Coco & Squish GUI Tester

Last edited on

Measuring code coverage of ASP.NET Web application using Squish Coco & tests run by Squish for Web

Introduction

This article demonstrates measuring code coverage of an ASP.NET (C#) Web application using Squish Coco. Additionally, automated tests are executed by Squish for Web, measuring the code coverage of each GUI automated test. The code coverage insight enables teams to improve their automated test suite by removing tests that don't increase testing coverage, and identifying areas that remain untested. Similar achievements can be made using other web automation tests, or even through manual testing. The test execution order, which indicates how to best maximize the code coverage can be calculated after each execution. The calculation reveals a test sequence in which tests with the highest code coverage appear first. This information is necessary when picking good test candidates for a subset of automated tests to be run in a limited time (i.e. sanity/brash automated tests).

Web application instrumentation

The example application used to demonstrate this solution is a simple ASP.NET MVC 5 Web application designed using C# in Visual Web Developer. MVC stands for model-view-controller, and is a pattern for developing applications containing: models (classes that represent the application data), views (templates which generate an HTML response) and controllers (classes that handle incoming browser requests, retrieve model data, and specify templates view which return a response to the browser). The code coverage analysis will reveal how each part is used during the execution of single automated test.

The application, Frog Directory, is run in Visual Studio on a localhost, but can also be deployed and run on a remote host. Frog Directory maintains a directory of frogs, storing information such as species, date of birth, gender and price. The application offers functionality to create, edit and delete entries and offers, as well as provides filtering and searching capabilities.

Application Index Page screenshot

During the Squish Coco installation the original C# compiler is replaced by the CoverageScanner. The CoverageScanner instruments the source code, which enables the code coverage measurement capabilities, after which it calls the original compiler to execute the application, leaving the source code untouched. To activate the instrumentation in Visual Studio project, go to Project Properties, click the Build tab and enter COVERAGESCANNER_COVERAGE_ON in the Conditional compilation symbols box.

Activating the instrumentation in Visual Studio

After each build, a Squish Coco instrumentation file (.csmes) is generated. In the example application, Frog Directory, the file is called FrogDirectory.dll.csmes and is located in the obj\Release subdirectory. Double click the file to inspect it in the CoverageBrowser. Currently no coverage statistics are available in the CoverageBrowser because the application has yet to be executed.

Squish for Web integration

In order to generate any code coverage we need to perform some actions on the running (and previously instrumented) application. Actions can be performed manually or using automation (i.e. running unit tests or automated GUI tests). GUI automated tests for the ASP.NET application can be designed using Squish for Web. To integrate Squish for Web and Squish Coco we need to add a small segment of code to the automated tests and the Web application.

init Function

Implement an init function, which is called at the beginning of each test. This function passes information to the Web application by calling a loadUrl function indicating the test case has started.

function init(){
    testCase = squishinfo.testCase.split("\\").pop();
    loadUrl("http://localhost:49221/Frogs/Coco/"+testCase+"?start=1");
}
init function

cleanup Function

Implement a cleanup function, which is called at the end of each test. This function passes information to the Web application indicating the given test case has ended, and with a given status (PASSED or FAILED). Place both functions in the shared test suite resources, ensuring they are called for every test case in the test suite.

function cleanup(){
    testCase = squishinfo.testCase.split("\\").pop();
    teststate = "PASSED";
    if(test.resultCount("errors")>0 || test.resultCount("fails")>0 || test.resultCount("fatals")>0)
        teststate = "FAILED";

    loadUrl("http://localhost:49221/Frogs/Coco/"+testCase+"?teststate="+teststate);
}
cleanup function

Handling the init and cleanup Functions

An additional C# function needs to be implemented in the Web application to handle requests from the init and cleanup functions. The init function causes the C# CoverageScanner library function __coveragescanner_clear() to be called, reseting the status of all instrumentations. The cleanup function causes the __coveragescanner_testname(id) to be called, which sets the name of the test case currently being executed. After which the __coveragescanner_teststate is called, and sets the result of executed test to PASSED|FAILED|CHECK_MANUALLY. Finally __coveragescanner_save() executes and saves the execution report using the filename previously set in the __coveragescanner_filename. Using this method of communication (loading URL's with a defined syntax) between Squish for Web and Squish Coco, allows Squish for Web and an ASP.NET Web server to on different hosts. The Squish Coco execution report is stored on the host (in a TEMP directory) where Web server is running.

public ActionResult Coco(string id, string teststate, string start)
        {
            if (!String.IsNullOrEmpty(start))
            {
                CoverageScanner.__coveragescanner_clear();
            }

            if (!String.IsNullOrEmpty(teststate))
            {
                CoverageScanner.__coveragescanner_testname(id);
                if (teststate == "PASSED" || teststate == "FAILED")
                {
                    CoverageScanner.__coveragescanner_teststate(teststate);
                }
                else
                {
                    CoverageScanner.__coveragescanner_teststate("CHECK_MANUALLY");
                }

                string tempDir = Environment.GetEnvironmentVariable("temp");
                CoverageScanner.__coveragescanner_filename(tempDir+Path.DirectorySeparatorChar+"coco");
                CoverageScanner.__coveragescanner_save();
            }
            return View();
        }
Coco function inside Controller class

Code Coverage Measurement analysis

Next we'll run three test cases (tst_addFrog, tst_filterFrog and tst_searchFrog) and load an execution report (csexe file located in %TEMP% directory) in the CoverageBrowser (select File > Load Execution Report). The CoverageBrowser analyzes what code executed when the automated tests ran. It also offers Line Coverage, Function Coverage and Branch, Decision and Condition Coverage and analysis.

Loading execution report in CoverageBrowser

To view the optimal execution order, select Tools > Optimized Execution Order. In the following example, the optimal order is tst_searchFrog and tst_filterFrog. From a code coverage perspective, the third test case, tst_addFrog, is redundant, and therefore is omitted from the report.

Optimal Execution Order

Additional information