Better Looking TestNG Reports with ReportNG

We had a set of system tests using Selenium 2 Webdriver and I was not satisfied with the default TestNG reports. I was  looking for a way to make reports look better and provide all the information necessary for analysis of a test failure when is happens. There were two requirements :

1. provide nice, compact overview 
2. include screenshot of a moment of failure

First thing I tried was Allure framework - it creates very nice reports but I had to reject it after some trials because it the way it works it was incompatible with the existing tests and its also quite invasive.

Fortunately I found ReportNG after that. The default design might not be so fancy but it is still very good and it fits well into TestNG and our tests.

First we had to add necessary dependencies to our maven POM:

<dependency>
   <groupId>org.testng</groupId>
   <artifactId>testng</artifactId>
   <version>6.8.8</version>
</dependency>

<dependency>
   <groupId>org.uncommons</groupId>
   <artifactId>reportng</artifactId>
   <version>1.1.4</version>
   <exclusions>
      <exclusion>
         <groupId>org.testng</groupId>
         <artifactId>testng</artifactId>
      </exclusion>
   </exclusions>
</dependency>

<dependency>
   <groupId>com.google.inject</groupId>
   <artifactId>guice</artifactId>
   <version>3.0</version>
</dependency>

TestNG has several interfaces to hook into the test processing, the most interesting probably are ITestListener, IConfigurationListener, and sometimes IMethodInterceptor. ReportNG add class HTMLReporter to that.

To add a screenshot to the report, we need to save it ITestListener.onContextFailure() and pick it up
in a custom ReportNGUtils -- for customization we need to provide custom Velocity context by overriding createContext() and passing custom ReportNGUtils implementation.

import org.apache.commons.io.FileUtils;
import org.apache.velocity.VelocityContext;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.IConfigurationListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.uncommons.reportng.HTMLReporter;

import java.io.File;
import java.net.URL;

public class TestListener extends HTMLReporter
       implements ITestListener, IConfigurationListener
{
    protected static final CustomReportNgUtils REPORT_NG_UTILS = new CustomReportNgUtils();

    @Override 

    protected VelocityContext createContext()
    {
        VelocityContext context = super.createContext();

        // VelocityContext has three properties: meta, utils, messages 
        // - see AbstractReporter.createContext()
        context.put("utils", REPORT_NG_UTILS);

        return context;
    }

    /** Invoked when test method (method with annotation @Test) fails. */
    @Override
    public void onTestFailure(ITestResult testResult)
    {
        if (getWebDriver(testResult) != null)
        {
            File scrFile = ((TakesScreenshot) getWebDriver(testResult))
                                             .getScreenshotAs(OutputType.FILE);
            String screenshotName = createScreenshotName(testResult);

            File targetFile = new File(screenshotName);
            FileUtils.copyFile(scrFile, targetFile); 

            URL scrUrl = new URL(getDriver(testResult).getCurrentUrl()); 

            Screenshot screenshot = new Screenshot(targetFile, srcUrl );
            testResult.setAttribute(Screenshot.KEY, screenshot);
        }

    }

    // ...
}

Class Screenshot is a custom class holding screenshot-related data, bare bones version could looke as this:

class Screenshot
{
    /* Name of {@link ITestResult} attribute for Screenshot. */
    static final String KEY = "screenshot";

    /** File in which is the screenshot stored. */
    File file;

    /** URL of a web application's page the screenshot captures. */
    URL url;
}

Now we need to add the custom ReportNGUtils implementation which picks up contextual information (Screenshot instance in our case) and uses it to modify the report output.

import java.util.List;

import org.testng.ITestResult;
import org.uncommons.reportng.ReportNGUtils;

class CustomReportNgUtils extends ReportNGUtils
{
    public List<String> getTestOutput(ITestResult testResult)
    {
        List<String> output = super.getTestOutput(testResult);

        if ( testResult.getAttribute(Screenshot.KEY) != null )
        {
            Screenshot screenshot = (Screenshot) testResult.getAttribute(Screenshot.KEY);
            String screenshotFileName = screenshot.getFile().getName();

            if (screenshot != null)
            {
                String url = (String) testResult.getAttribute("screenshotUrl");
                output.add(String.format("screenshot for %s  %s <br/><img src='../screenshots/%s'>",
                                         testResult.getName(), url, screenshotFileName)
                );
            }
        }

        return output;
    }
}

The final step is to register test listener in the plugin executing the tests. We use failsafe, the configuration for surefire is similar if you desire to use it.

<build>
        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>

                <configuration>

                    <systemPropertyVariables>
                        <org.uncommons.reportng.escape-output> 

                        false 

                        </org.uncommons.reportng.escape-output>
                    </systemPropertyVariables>
                    <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary.xml</summaryFile>

                    <testClassesDirectory>${project.build.directory}/classes</testClassesDirectory> 
                    <properties>
                        <property>
                            <name>usedefaultlisteners</name>
                            <value>false</value>
                        </property>
                        <property>
                            <name>listener</name>
                            <value>
                                org.bithill.test.testng.TestListener
                            </value>
                        </property>
                    </properties>

                    <suiteXmlFiles>
                        <suiteXmlFile>src/main/resources/suiteX.xml</suiteXmlFile>
                    </suiteXmlFiles>

                </configuration>

                <executions>

                    <execution>
                        <id>integration-test</id> 

                        <phase>integration-test</phase>

                        <goals> <goal>integration-test</goal> </goals>
                    </execution>

                    <execution>
                        <id>verify</id> 

                        <phase>verify</phase>

                        <goals> <goal>verify</goal> </goals>
                    </execution>

                </executions>
            </plugin>

        </plugins>
    </build> 

And that's all - when you run 'mvn failsafe:integration-test' the tests are run, then you follow with 'mvn failsafe:verify',  which processes the results of the integration tests and generates a report and sets proper build result. Note that set the testClassesDirectory si crucial if you have it different than the expected 'test-classes'.

Remarkable Changes in Past Versions of Selenium 2 WebDriver

For a long time I kept the code base of our tests running on Selenium 2.0a6. The reasons for not upgrading were different for different Selenium 2 versions - perceived stability problems in InternetExplorer or Firefox, changes being not to enough beneficial for our tests and sometimes even lack of time for the change.

Now it seems Selenium 2.8 is a good candidate for upgrade.  Following enumeration summarizes the changes we waited for and I want to use it as a thanks to all the Selenium 2 developers who participated in the effort. I also hope it wil help to anybody upgrading or considering the upgrade to a newer Selenium 2 .

1/  RenderedWebElement deprecated and removed in 2.0rc3,  method isDisplayed() was moved to WebElement class.

2/ Mouseover works since 2.0 RC2:

import org.openqa.selenium.interactions.Actions;
import org.openqa.selenium.interactions.Action;
Actions builder = new Actions(driver);
Action hoverAction = builder.moveToElement(mouseOverElement).build();
hoverAction.perform();

3/ For some time it was necessary to send Enter to a button in MSIE to press it,
but since version 2.2 clicking on buttons (WebElement.click()) seems to work flawlessly.

4/ Version 2.3 brought the nice Alert class for confirmation and alert dialogs, rendering thus JavaScript workarounds obsolete:

package org.openqa.selenium;
public interface Alert
{
  void dismiss();
  void accept();
  String getText();
  void sendKeys(String keysToSend);
}

the usage:

Alert prompt = driver.switchTo().alert();
// some short sleep here
log.debug( prompt.getText() );
prompt.sendKeys("AAA");
// some short sleep here
prompt.accept();

5/ And finally, since 2.8 important parts of the advanced interactions, double-click and right-click, work both for MSIE end FF (since 2.5for MSIE):

Actions builder = new Actions(driver);
Action doubleClick = builder.doubleClick(element).build();
doubleClick.perform();

Actions builder = new Actions(driver);
Action rightClick = builder.contextClick(element).build();
rightClick.perform();

Setting Firefox Preferences via Selenium 2 (WebDriver API)

I wanted to run Firefox from WebDriver with custom preferences.  So I looked into the well-known about:config for name of the option and, just to be sure, consulted About:config entries section of Mozilla Wiki.

With the name of desired config option to change, the rest is a pieceof cake:

FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("dom.event.contextmenu.enabled",false);
WebDriver webDriver =  new FirefoxDriver(profile);

Moving to Selenium 2 on WebDriver, Part No.5

Sending Mouse Clicks Via Javascript

Sometimes the WebDriver click does not work well,  after all it is still under development and bugs should be expected. For such case the ability to execute JavaScript through JavascriptExecutor comes in handy. Following code can send "click" or "dblclick" when used as eventName to a DOM element addressed by elementRef - JavaScript expression evaluating to a Node.

  public void fireJsEvent(String elementRef, String eventName)
    {

        String script =
        "" +
        " function eventFire(element, eventName)" +
        " {" +
        "  if (element.fireEvent)" +
        "  { element.fireEvent('on' + eventName); }" +
        "  else" +
        "  {" +
        "    var eventObject = document.createEvent('Events');" +    
        // parameters: type, bubbles, cancelable
        "    eventObject.initEvent(eventName, true, false);" +
        "    element.dispatchEvent(eventObject);" +
        "  }" +
        " };";

        String eventCall = String.format( "eventFire(%s, '%s');", elementRef, eventName );
        String exec = script + eventCall;
        js.executeScript(exec);
    }

Possible usage on a web page with dojo:

fireJsEvent("dojo.query('.someColumn:nth-child(1) li:nth-child(2)')[0]", "click");

Moving to Selenium 2 on WebDriver, Part No.4

Conditional Waits

Majority part of web UI tests I fixed and keep fixing has the same flaw - there is either no wait for dynamically loaded parts of page or the wait is unconditional. As driver.get() is blocking this is clearly problem of an AJAX calls.

WebDriver provides two APIs for conditional waits - com.thoughtworks.selenium.Wait and org.openqa.selenium.support.ui.Wait.


Class com.thoughtworks.selenium.Wait

Simpler of the two and IMHO better suited for ad-hoc waits is this abstract class. It has only two methods - wait() and until().  You have to implement method until() and it should return true when the condition is met.  

Example of wait for pop-up of given name and switching to it:

public void waitForPopUp(final String windowName, int timeoutMs)
{
   new Wait()
   {
      @Override
      public boolean until()
      {
         try
         {
            driver.switchTo().window(windowName);
            return true;
         }
         catch (SeleniumException ignored) { }
         return false;
      }
   }.wait(String.format("Timed out waiting for %s. Waited %s",windowName, timeoutMs), timeoutMs);
}

Interfaces and classes from org.openqa.selenium.support.ui

More sophisticated approach is represented by interfaces Wait, implemented by WebDriverWait class, and  ExpectedCondition (extending com.google.common.base). Class implementing the ExpectedCondition interface must define method apply() returning true or not-null when a condition is met.

Example of wait expecting JavaScript expression evaluate to true:

int timeoutSeconds = 10;
Wait wait = new WebDriverWait(driver, timeoutSeconds);
JavascriptExecutor js = (JavascriptExecutor)driver;

public boolean waitForJsCondition(final String script, String timeoutMessage)
{
    boolean result = false;

    try
    {
        result = wait.until
        (
            new ExpectedCondition<Boolean>()
            {
               public Boolean apply(WebDriver driver)
               { return (Boolean)js.executeScript("return " +  script); }
            }
        );
    }
    catch (Throwable t)
    { handleWaitTimeout(t,timeoutMessage); }

    return result;
}

As you can see the advantage is that this Wait returns value. You can wait for an element to become visible and return it, thus keeping you code cleaner and more resilient. Even better, expected conditions returning the same type are interchangable. That not only adds to flexibility, it also helps to fight code duplication.

Moving to Selenium 2 on WebDriver, Part No.3

For following examples we define:
JavaScriptExecutor js = (JavascriptExecutor)driver);


Count elements matching a XPath expression
Selenium 1:
selenium.getXpathCount(location);

Selenium 2:
driver.findElements(By.xpath(location)).size(); 

Set focus to an element


Selenium 1:
selenium.focus( locator );

Selenium 2:

Driver calls "element.focus()" before sending further events as click or keys to the element, so no explicit setting of focus method is necessary. Still, following should work: js.executeScript ("document.getElementById('inputId').focus()");


Handle Confirm or Alert

Selenium 1:
selenium.chooseOkOnNextConfirmation(); // accept
selenium.chooseCancelOnNextConfirmation(); // cancel

Selenium 2:
js.executeScript("window.confirm = function(msg){ return true;};");
js.executeScript("window.confirm = function(msg){ return false;};");

These JavaScript 'hacks' change behaviour of all confirmation dialogs called after their use.

To be a bit more flexible I use set of three methods.
To handle alerts, just just override window.alert instead of window.confirm.

1. Save original handler and sets a custom one.
js.executeScript("window.orig_confirm = window.confirm;");
// always confirm and save a dialog message
js.executeScript("window.confirm = function(msg){ document.last_confirm=msg; return true;};");

2. Retrieve the alert message.
// alternative to selenium.getConfirmation();
String message = (String)js.executeScript("return document.last_confirm");

3. Restore the original handler.
js.executeScript("window.confirm = window.orig_confirm;");


Select a Window
Every window has a name, in our example the name is 'windowName'.

Selenium 1:
selenium.selectWindow("name=windowName");

Selenium 2:
driver.switchTo().window("windowName");

Moving to Selenium 2 on WebDriver, Part No.2

In the following examples the location is always of type org.openqa.selenium.By, while locator is the old Selenium string.

Select a single item in select by label

Selenium 1:
selenium.select(locator,"label=" + optionLabel);

Selenium 2:
Select select = (Select)driver.findElement(location);
select.selectByVisibleText(optionLabel);

Select a single item in select by value

Selenium 1:
selenium.select(location,"value=" + optionValue);

Selenium 2:
select.selectByValue(optionValue);

Write text to an input field

Selenium 1:
selenium.type(locator, text);

Selenium 2:
driver.findElement(location).sendkeys(text);

Get text content of an element

Selenium 1:
selenium.getText(locator);

Selenium 2:
driver.findElement(location).getText();

Setting a checkbox to 'checked' state

Selenium 1:
selenium.check(locator);

Selenium 2:
driver.findElement(location).setSelected();


Javascript in MSIE

When using Javascript (through JavascriptExecutor's executeScript()) you can get unexpected behaviour - MSIE crashes without useful error message when you access some Javascript objects in browser.

To cope with this on Windows 7, check your Protected Mode settings (Tools > Internet Options > Security) and set it to the same value for all zones. I don't know if Windows XP is also affected and how to se it it such case.

Execute Javascript

For all following examples let's define:
JavascriptExecutor js = (JavascriptExecutor) driver;

Selenium 1:
selenium.runScript(script);

Selenium 2:
js.executeScript(script);

Execute Javascript and get result

Selenium 1:
scriptResult = selenium.getEval(scriptReturningString);

Selenium 2:
String scriptResult = (String)js.executeScript(scriptReturningString);

WebDriver requires the script to begin with "return". It's easy to miss it in documentation. Script runs in context of a currently selected window - you do not need any tricks for that as in Selenium 1. Also the result is casted to matching Java type.

Moving to Selenium 2 on WebDriver, Part No.1

I am in the process of trying to move code of some test to the Selenium 2 WebDriver. This is the first part of series of short articles about how to "translate" old Selenium API to the WebDriver-based one.

In following, let's state that xpathLocator is Selenium 1 locator, while xpathFinder is more or less By.xpath(xpathLocator),selenium is instance of Selenium, driver is instance of WebDriver (FirefoxDriver).

Is an element present on a page ?

Selenium 1:
selenium.isElementPresent(xpathLocator);

Selenium 2:
driver.findElements(xpathFinder).size()>0;

Get HTML source of a page

Selenium 1:
selenium.getHtmlSource();

Selenium 2:
driver.getPageSource();

Is a text present on a page ?

Selenium 1:
selenium.isTextPresent(text);

Selenium 2:
driver.getPageSource().contains(text);

Go to relative URL
The relativeUrl is URL relative to initial URL, i.e. without the "http://hostname".

Selenium 1:
selenium.open(relativeUrl);

Selenium 2:
driver.get( initialUrl + relativeUrl );

This call is blocking, i.e. further processing is stopped until the page is loaded.

To open an URL relative to the currently used  one, use  driver.getCurrentUrl():
driver.get( driver.getCurrentUrl() + relativeUrl );

Is an element visible on a page?

Selenium 1:
selenium.isVisible(location)

Selenium 2:
driver.findElement(location) instanceof RenderedWebElement
((RenderedWebElement)driver.findElement(location)).isDisplayed()