This article demonstrates measuring code coverage of an ASP.NET (C#) Web application using Squish Coco. Additionally, written in Java automated tests are executed by TestNG framework, measuring the code coverage of each executed test. The integration between TestNG driven tests (run on a client host) and the web server running ASP.NET (C#) Web application is done using HTTP requests. This HTTP GET requests carry information about what test is currently being executed and what is the status of given tests after the execution is finished. Web application (through some simple extension) passes this information to Coverage Scanner. At the end user can analyse what ASP.NET (C#) Web application server code was executed during automated GUI tests execution.
Web application instrumentation¶
The example application used to demonstrate this solution is a simple ASP.NET MVC 5 Web application designed using C# in Visual Web Developer. MVC stands for model-view-controller, and is a pattern for developing applications containing: models (classes that represent the application data), views (templates which generate an HTML response) and controllers (classes that handle incoming browser requests, retrieve model data, and specify templates view which return a response to the browser). The code coverage analysis will reveal how each part is used during the execution of single automated test.
The application, Frog Directory, is run in Visual Studio on a localhost, but can also be deployed and run on a remote host. Frog Directory maintains a directory of frogs, storing information such as species, date of birth, gender and price. The application offers functionality to create, edit and delete entries and offers, as well as provides filtering and searching capabilities.
During the Squish Coco installation the original C# compiler is replaced by the CoverageScanner. The CoverageScanner instruments the source code, which enables the code coverage measurement capabilities, after which it calls the original compiler to execute the application, leaving the source code untouched. To activate the instrumentation in Visual Studio project, go to Project Properties, click the Build tab and enter
COVERAGESCANNER_COVERAGE_ON in the Conditional compilation symbols box.
After each build, a Squish Coco instrumentation file (
.csmes) is generated. In the example application, Frog Directory, the file is called
FrogDirectory.dll.csmes and is located in the
obj\Release subdirectory. Double click the file to inspect it in the CoverageBrowser. Currently no coverage statistics are available in the CoverageBrowser because the application has yet to be executed.
In order to generate any code coverage we need to perform some actions on the running (and previously instrumented) application. Actions can be performed manually or using automation (i.e. running unit tests or automated GUI tests). GUI automated tests for the ASP.NET web application can be designed using any web automated testing tool (including Selenium) which allows to develop tests in Java. Later those Java written tests can be run using TestNG framework.
To integrate TestNG driven tests and Squish Coco we need to enhance both Java tests and the Web application. The main purpose is to provide communication channel for information about what test is being executed and what is the status of finished test.
Automated Tests enhancement¶
We need to implement an
ITestListener that will provide handling methods for events like TestStart, TestSuccess and TestFailures. So first, using the @Listeners annotation, we must specify TestNG listeners for our test class.
CocoReporter class implement an
onTestStart function, which is called at the beginning of each test. This function passes information to the Web application by sending HTTP GET request with test case name indicating given test case has started. Additionally implement
onTestFailure functions, which are called at the end of each test. This function passes information to the Web application indicating the given test case has ended, and with a given status (PASSED or FAILED).
Web application enhancement¶
An additional C# function needs to be implemented in the Web application to handle HTTP requests from the TestNG listeners. The
onTestStart handler function causes the C# CoverageScanner library function
__coveragescanner_clear() to be called, reseting the status of all instrumentations. The
onTestFailure functions causes the
__coveragescanner_testname(id) to be called, which sets the name of the test case currently being executed. After which the
__coveragescanner_teststate is called, and sets the result of executed test to
__coveragescanner_save() executes and saves the execution report using the filename previously set in the
__coveragescanner_filename. Last, we shall put our C# enhancement function inside
#region CoverageScanner(cov-off), so we CoverageScanner won't instrument this function, as we are not interested of measuring it's coverage during execution of automated tests.
Using this method of communication (loading URL's with a defined syntax) between Squish for Web and Squish Coco, allows running TestNG tests and an ASP.NET Web server on different hosts. The Squish Coco execution report is stored on the host (in a TEMP directory) where the Web server is running.
Code Coverage Measurement analysis¶
Next we'll run two test cases (testAddDeleteFrog and testFilterandView) and load an execution report (
csexe file located in %TEMP% directory) in the CoverageBrowser (select File > Load Execution Report). The CoverageBrowser analyzes what code executed when the automated tests ran. It also offers Line Coverage, Function Coverage and Branch, Decision and Condition Coverage and analysis.
When analysing coverage of source code in
frogscontroller.cs, we noticed that we tested filtering by frog gender (see that line 44 was executed), but our tests did not cover filtering by frog species (see that line 39 was not executed). This is very valuable information, hence we shall enhance testFilterandViw test case to cover also this scenario.
Source Code for examples¶
Download example source code containing CocoReporter.java (TestNG Listener implementation) and FrogShopBasicTests.java (Selenium Tests driven by TestNG).