Better Looking TestNG Reports with ReportNG

We had a set of system tests using Selenium 2 Webdriver and I was not satisfied with the default TestNG reports. I was  looking for a way to make reports look better and provide all the information necessary for analysis of a test failure when is happens. There were two requirements :

1. provide nice, compact overview 
2. include screenshot of a moment of failure

First thing I tried was Allure framework - it creates very nice reports but I had to reject it after some trials because it the way it works it was incompatible with the existing tests and its also quite invasive.

Fortunately I found ReportNG after that. The default design might not be so fancy but it is still very good and it fits well into TestNG and our tests.

First we had to add necessary dependencies to our maven POM:

<dependency>
   <groupId>org.testng</groupId>
   <artifactId>testng</artifactId>
   <version>6.8.8</version>
</dependency>

<dependency>
   <groupId>org.uncommons</groupId>
   <artifactId>reportng</artifactId>
   <version>1.1.4</version>
   <exclusions>
      <exclusion>
         <groupId>org.testng</groupId>
         <artifactId>testng</artifactId>
      </exclusion>
   </exclusions>
</dependency>

<dependency>
   <groupId>com.google.inject</groupId>
   <artifactId>guice</artifactId>
   <version>3.0</version>
</dependency>

TestNG has several interfaces to hook into the test processing, the most interesting probably are ITestListener, IConfigurationListener, and sometimes IMethodInterceptor. ReportNG add class HTMLReporter to that.

To add a screenshot to the report, we need to save it ITestListener.onContextFailure() and pick it up
in a custom ReportNGUtils -- for customization we need to provide custom Velocity context by overriding createContext() and passing custom ReportNGUtils implementation.

import org.apache.commons.io.FileUtils;
import org.apache.velocity.VelocityContext;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.IConfigurationListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.uncommons.reportng.HTMLReporter;

import java.io.File;
import java.net.URL;

public class TestListener extends HTMLReporter
       implements ITestListener, IConfigurationListener
{
    protected static final CustomReportNgUtils REPORT_NG_UTILS = new CustomReportNgUtils();

    @Override 

    protected VelocityContext createContext()
    {
        VelocityContext context = super.createContext();

        // VelocityContext has three properties: meta, utils, messages 
        // - see AbstractReporter.createContext()
        context.put("utils", REPORT_NG_UTILS);

        return context;
    }

    /** Invoked when test method (method with annotation @Test) fails. */
    @Override
    public void onTestFailure(ITestResult testResult)
    {
        if (getWebDriver(testResult) != null)
        {
            File scrFile = ((TakesScreenshot) getWebDriver(testResult))
                                             .getScreenshotAs(OutputType.FILE);
            String screenshotName = createScreenshotName(testResult);

            File targetFile = new File(screenshotName);
            FileUtils.copyFile(scrFile, targetFile); 

            URL scrUrl = new URL(getDriver(testResult).getCurrentUrl()); 

            Screenshot screenshot = new Screenshot(targetFile, srcUrl );
            testResult.setAttribute(Screenshot.KEY, screenshot);
        }

    }

    // ...
}

Class Screenshot is a custom class holding screenshot-related data, bare bones version could looke as this:

class Screenshot
{
    /* Name of {@link ITestResult} attribute for Screenshot. */
    static final String KEY = "screenshot";

    /** File in which is the screenshot stored. */
    File file;

    /** URL of a web application's page the screenshot captures. */
    URL url;
}

Now we need to add the custom ReportNGUtils implementation which picks up contextual information (Screenshot instance in our case) and uses it to modify the report output.

import java.util.List;

import org.testng.ITestResult;
import org.uncommons.reportng.ReportNGUtils;

class CustomReportNgUtils extends ReportNGUtils
{
    public List<String> getTestOutput(ITestResult testResult)
    {
        List<String> output = super.getTestOutput(testResult);

        if ( testResult.getAttribute(Screenshot.KEY) != null )
        {
            Screenshot screenshot = (Screenshot) testResult.getAttribute(Screenshot.KEY);
            String screenshotFileName = screenshot.getFile().getName();

            if (screenshot != null)
            {
                String url = (String) testResult.getAttribute("screenshotUrl");
                output.add(String.format("screenshot for %s  %s <br/><img src='../screenshots/%s'>",
                                         testResult.getName(), url, screenshotFileName)
                );
            }
        }

        return output;
    }
}

The final step is to register test listener in the plugin executing the tests. We use failsafe, the configuration for surefire is similar if you desire to use it.

<build>
        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>

                <configuration>

                    <systemPropertyVariables>
                        <org.uncommons.reportng.escape-output> 

                        false 

                        </org.uncommons.reportng.escape-output>
                    </systemPropertyVariables>
                    <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary.xml</summaryFile>

                    <testClassesDirectory>${project.build.directory}/classes</testClassesDirectory> 
                    <properties>
                        <property>
                            <name>usedefaultlisteners</name>
                            <value>false</value>
                        </property>
                        <property>
                            <name>listener</name>
                            <value>
                                org.bithill.test.testng.TestListener
                            </value>
                        </property>
                    </properties>

                    <suiteXmlFiles>
                        <suiteXmlFile>src/main/resources/suiteX.xml</suiteXmlFile>
                    </suiteXmlFiles>

                </configuration>

                <executions>

                    <execution>
                        <id>integration-test</id> 

                        <phase>integration-test</phase>

                        <goals> <goal>integration-test</goal> </goals>
                    </execution>

                    <execution>
                        <id>verify</id> 

                        <phase>verify</phase>

                        <goals> <goal>verify</goal> </goals>
                    </execution>

                </executions>
            </plugin>

        </plugins>
    </build> 

And that's all - when you run 'mvn failsafe:integration-test' the tests are run, then you follow with 'mvn failsafe:verify',  which processes the results of the integration tests and generates a report and sets proper build result. Note that set the testClassesDirectory si crucial if you have it different than the expected 'test-classes'.