We had a set of system tests using Selenium 2
Webdriver and I was not satisfied with the default
TestNG reports. I was looking for a way to make reports look better and provide all the information necessary for analysis of a test failure when is happens. There were two requirements :
- provide nice, compact overview
- include screenshot of a moment of failure
First thing I tried was Allure framework - it creates very nice reports but I had to reject it after some trials because it the way it works it was incompatible with the existing tests and its also quite invasive.
Fortunately I found
ReportNG after that. The default design might not be so fancy but it is still very good and it fits well into TestNG and our tests.
First we had to
add necessary
dependencies to our
maven POM:
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.8.8</version>
</dependency>
<dependency>
<groupId>org.uncommons</groupId>
<artifactId>reportng</artifactId>
<version>1.1.4</version>
<exclusions>
<exclusion>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
<version>3.0</version>
</dependency>
TestNG has several interfaces to hook into the test processing, the most interesting probably are
ITestListener,
IConfigurationListener, and sometimes
IMethodInterceptor. ReportNG add class HTMLReporter to that.
To add a screenshot to the report, we need to save it
ITestListener.onContextFailure() and pick it up
in a custom
ReportNGUtils -- for customization we need to provide custom Velocity context by overriding
createContext() and passing custom ReportNGUtils implementation.
import org.apache.commons.io.FileUtils;
import org.apache.velocity.VelocityContext;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.IConfigurationListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.uncommons.reportng.HTMLReporter;
import java.io.File;
import java.net.URL;
public class TestListener extends HTMLReporter
implements ITestListener, IConfigurationListener
{
protected static final CustomReportNgUtils REPORT_NG_UTILS = new CustomReportNgUtils();
@Override
protected VelocityContext createContext()
{
VelocityContext context = super.createContext();
// VelocityContext has three properties: meta, utils, messages
// - see AbstractReporter.createContext()
context.put("utils", REPORT_NG_UTILS);
return context;
}
/** Invoked when test method (method with annotation @Test) fails. */
@Override
public void onTestFailure(ITestResult testResult)
{
if (getWebDriver(testResult) != null)
{
File scrFile = ((TakesScreenshot) getWebDriver(testResult))
.getScreenshotAs(OutputType.FILE);
String screenshotName = createScreenshotName(testResult);
File targetFile = new File(screenshotName);
FileUtils.copyFile(scrFile, targetFile);
URL scrUrl = new URL(getDriver(testResult).getCurrentUrl());
Screenshot screenshot = new Screenshot(targetFile, srcUrl );
testResult.setAttribute(Screenshot.KEY, screenshot);
}
}
// ...
}
Class Screenshot is a custom
class holding screenshot-related data, bare bones version could looke as this:
class Screenshot
{
/* Name of {@link ITestResult} attribute for Screenshot. */
static final String KEY = "screenshot";
/** File in which is the screenshot stored. */
File file;
/** URL of a web application's page the screenshot captures. */
URL url;
}
Now we need to add the
custom ReportNGUtils implementation which picks up contextual information (Screenshot instance in our case) and uses it to modify the report output.
import java.util.List;
import org.testng.ITestResult;
import org.uncommons.reportng.ReportNGUtils;
class CustomReportNgUtils extends ReportNGUtils
{
public List<String> getTestOutput(ITestResult testResult)
{
List<String> output = super.getTestOutput(testResult);
if ( testResult.getAttribute(Screenshot.KEY) != null )
{
Screenshot screenshot = (Screenshot) testResult.getAttribute(Screenshot.KEY);
String screenshotFileName = screenshot.getFile().getName();
if (screenshot != null)
{
String url = (String) testResult.getAttribute("screenshotUrl");
output.add(String.format("screenshot for %s %s <br/><img src='../screenshots/%s'>",
testResult.getName(), url, screenshotFileName)
);
}
}
return output;
}
}
The final step is to
register test listener in the plugin executing the tests. We use
failsafe, the configuration for surefire is similar if you desire to use it.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<org.uncommons.reportng.escape-output>
false
</org.uncommons.reportng.escape-output>
</systemPropertyVariables>
<summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary.xml</summaryFile>
<testClassesDirectory>${project.build.directory}/classes</testClassesDirectory>
<properties>
<property>
<name>usedefaultlisteners</name>
<value>false</value>
</property>
<property>
<name>listener</name>
<value>
org.bithill.test.testng.TestListener
</value>
</property>
</properties>
<suiteXmlFiles>
<suiteXmlFile>src/main/resources/suiteX.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<phase>integration-test</phase>
<goals> <goal>integration-test</goal> </goals>
</execution>
<execution>
<id>verify</id>
<phase>verify</phase>
<goals> <goal>verify</goal> </goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
And that's all - when you run '
mvn failsafe:integration-test' the tests are run, then you follow with '
mvn failsafe:verify', which processes the results of the integration tests and generates a report and sets proper build result. Note that set the
testClassesDirectory si crucial if you have it different than the expected 'test-classes'.
