Measuring code coverage of web application using Squish Coco & TestNG driven tests

Last edited on

Introduction

This article demonstrates measuring code coverage of an ASP.NET (C#) Web application using Squish Coco. Additionally, written in Java automated tests are executed by TestNG framework, measuring the code coverage of each executed test. The integration between TestNG driven tests (run on a client host) and the web server running ASP.NET (C#) Web application is done using HTTP requests. This HTTP GET requests carry information about what test is currently being executed and what is the status of given tests after the execution is finished. Web application (through some simple extension) passes this information to Coverage Scanner. At the end user can analyse what ASP.NET (C#) Web application server code was executed during automated GUI tests execution.

Web application instrumentation

The example application used to demonstrate this solution is a simple ASP.NET MVC 5 Web application designed using C# in Visual Web Developer. MVC stands for model-view-controller, and is a pattern for developing applications containing: models (classes that represent the application data), views (templates which generate an HTML response) and controllers (classes that handle incoming browser requests, retrieve model data, and specify templates view which return a response to the browser). The code coverage analysis will reveal how each part is used during the execution of single automated test.

The application, Frog Directory, is run in Visual Studio on a localhost, but can also be deployed and run on a remote host. Frog Directory maintains a directory of frogs, storing information such as species, date of birth, gender and price. The application offers functionality to create, edit and delete entries and offers, as well as provides filtering and searching capabilities.

Application Index Page screenshot

During the Squish Coco installation the original C# compiler is replaced by the CoverageScanner. The CoverageScanner instruments the source code, which enables the code coverage measurement capabilities, after which it calls the original compiler to execute the application, leaving the source code untouched. To activate the instrumentation in Visual Studio project, go to Project Properties, click the Build tab and enter COVERAGESCANNER_COVERAGE_ON in the Conditional compilation symbols box.

Activating the instrumentation in Visual Studio

After each build, a Squish Coco instrumentation file (.csmes) is generated. In the example application, Frog Directory, the file is called FrogDirectory.dll.csmes and is located in the obj\Release subdirectory. Double click the file to inspect it in the CoverageBrowser. Currently no coverage statistics are available in the CoverageBrowser because the application has yet to be executed.

TestNG integration

In order to generate any code coverage we need to perform some actions on the running (and previously instrumented) application. Actions can be performed manually or using automation (i.e. running unit tests or automated GUI tests). GUI automated tests for the ASP.NET web application can be designed using any web automated testing tool (including Selenium) which allows to develop tests in Java. Later those Java written tests can be run using TestNG framework.

To integrate TestNG driven tests and Squish Coco we need to enhance both Java tests and the Web application. The main purpose is to provide communication channel for information about what test is being executed and what is the status of finished test.

Automated Tests enhancement

We need to implement an ITestListener that will provide handling methods for events like TestStart, TestSuccess and TestFailures. So first, using the @Listeners annotation, we must specify TestNG listeners for our test class.

import org.testng.annotations.Listeners;
[...]
@Listeners({com.froglogic.examples.CocoReporter.class})
public class FrogShopBasicTests {
   [...]
   @Test
   public void testAddDeleteFrog(){
   [...]
   }
   [...]
   @Test
   public void testFiltering(){
   [...]
   }
}
FrogShopBasicTest.java

In CocoReporter class implement an onTestStart function, which is called at the beginning of each test. This function passes information to the Web application by sending HTTP GET request with test case name indicating given test case has started. Additionally implement onTestSuccess and onTestFailure functions, which are called at the end of each test. This function passes information to the Web application indicating the given test case has ended, and with a given status (PASSED or FAILED).

package com.froglogic.examples;

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

import java.net.HttpURLConnection;
import java.net.URL;

public class CocoReporter implements ITestListener {

    private final String USER_AGENT = "Mozilla/5.0";
    private final String urlBaseStr = "http://localhost:49221/Frogs/Coco/";

    private void sendGet(String urlStr){
        URL url;
        try {
            url = new URL(urlStr);
            HttpURLConnection con = (HttpURLConnection) url.openConnection();
            con.setRequestMethod("GET");
            con.setRequestProperty("User-Agent", USER_AGENT);
            int responseCode = con.getResponseCode();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private String getTcName(ITestResult iTestResult){
        String mName = iTestResult.getName();
        String[] classNames = iTestResult.getTestClass().getName().split("\\.");
        String className = classNames[classNames.length-1];
        return "GUITests!"+className+"!"+mName;
    }

    @Override
    public void onTestStart(ITestResult iTestResult) {
        String urlStr = urlBaseStr+getTcName(iTestResult)+"?start=1";
        sendGet(urlStr);
    }

    @Override
    public void onTestSuccess(ITestResult iTestResult) {
        String urlStr = urlBaseStr+getTcName(iTestResult)+"?testState=PASSED";
        sendGet(urlStr);
    }

    @Override
    public void onTestFailure(ITestResult iTestResult) {
        String urlStr = urlBaseStr+getTcName(iTestResult)+"?testState=FAILED";
        sendGet(urlStr);
    }

    @Override
    public void onTestSkipped(ITestResult iTestResult) {
    }

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult iTestResult) {
        String urlStr = urlBaseStr+getTcName(iTestResult)+"?testState=FAILED";
        sendGet(urlStr);
    }

    @Override
    public void onStart(ITestContext iTestContext) {
    }

    @Override
    public void onFinish(ITestContext iTestContext) {
    }
}
CocoReporter.java

Web application enhancement

An additional C# function needs to be implemented in the Web application to handle HTTP requests from the TestNG listeners. The onTestStart handler function causes the C# CoverageScanner library function __coveragescanner_clear() to be called, reseting the status of all instrumentations. The onTestSuccess and onTestFailure functions causes the __coveragescanner_testname(id) to be called, which sets the name of the test case currently being executed. After which the __coveragescanner_teststate is called, and sets the result of executed test to PASSED|FAILED|CHECK_MANUALLY. Finally __coveragescanner_save() executes and saves the execution report using the filename previously set in the __coveragescanner_filename. Last, we shall put our C# enhancement function inside

#region CoverageScanner(cov-off), so we CoverageScanner won't instrument this function, as we are not interested of measuring it's coverage during execution of automated tests.

Using this method of communication (loading URL's with a defined syntax) between Squish for Web and Squish Coco, allows running TestNG tests and an ASP.NET Web server on different hosts. The Squish Coco execution report is stored on the host (in a TEMP directory) where the Web server is running.

#region CoverageScanner(cov-off)
public ActionResult Coco(string id, string teststate, string start)
        {
            if (!String.IsNullOrEmpty(start))
            {
                CoverageScanner.__coveragescanner_clear();
            }

            if (!String.IsNullOrEmpty(teststate))
            {
                CoverageScanner.__coveragescanner_testname(id);
                if (teststate == "PASSED" || teststate == "FAILED")
                {
                    CoverageScanner.__coveragescanner_teststate(teststate);
                }
                else
                {
                    CoverageScanner.__coveragescanner_teststate("CHECK_MANUALLY");
                }

                string tempDir = Environment.GetEnvironmentVariable("temp");
                CoverageScanner.__coveragescanner_filename(tempDir+Path.DirectorySeparatorChar+"coco");
                CoverageScanner.__coveragescanner_save();
            }
            return View();
        }
#endregion
Coco function inside Controller class (C#)

Code Coverage Measurement analysis

Next we'll run two test cases (testAddDeleteFrog and testFilterandView) and load an execution report (csexe file located in %TEMP% directory) in the CoverageBrowser (select File > Load Execution Report). The CoverageBrowser analyzes what code executed when the automated tests ran. It also offers Line Coverage, Function Coverage and Branch, Decision and Condition Coverage and analysis.

Loading execution report in CoverageBrowser

When analysing coverage of source code in frogscontroller.cs, we noticed that we tested filtering by frog gender (see that line 44 was executed), but our tests did not cover filtering by frog species (see that line 39 was not executed). This is very valuable information, hence we shall enhance testFilterandViw test case to cover also this scenario.

Coverage analysis

Additional information

Source Code for examples

Download example source code containing CocoReporter.java (TestNG Listener implementation) and FrogShopBasicTests.java (Selenium Tests driven by TestNG).