Moving TeamCity Build to Remote Agent

As we added more build configuration to our TeamCity server, it was soon too much for the machine hosting it. Having a spare machine we decided to move some build to this new machine to lift the burden from our TeamCity's shoulders.

It was also opportunity to look at TeamCity plugins in general an specifically at so called agent tools.

Pre-requisities

• fresh installation of the Linux distribution of your choice
• JDK
• OpenSSH daemon for remote access  

Agent Push

On the target machine create an account for teamcity, e.g. teamcity - to make the maintenance easier use the same user/group name and id as on the machine hosting TeamCity.

I wanted to use password-based authentication but avoid to disclose root password so I used the same  credentials "push agent" as used for  "run under". It worked up to the "su" point -- see below.

Problem: push fails with "Algorithm negotiation fail"

This is cause by the removal of unsafe algorithms from OpenSSH default configuration. Unfortunately the JSCH library used by TeamCity still tries to use them and is refused.

To make JSCH happy, you can enable weak key-exchage algorithms by adding following line to the /etc/ssh/sshd_config file (diffie-hellman-group1-sha1 stands for 1024 bit DH with SHA1, diffie-hellman-group-exchange-sha1 for custom DH with SHA1):

KexAlgorithms diffie-hellman-group1-sha1, diffie-hellman-group-exchange-sha1

Please enable this line only for the agent push and make sure it is removed after that. It is broken in current TeamCity 9.1.3.

Problem: "su: must be run from a terminal"

The whole error message looks similar to this:

Remote agent installation failed: Command '[./bootstrapper.sh "http://myteamcity:8111" "/home/teamcity/BuildAgent" "some_security_token" "user" "password"]' was executed with error message(s): su: must be run from a terminal.

There are several issues associated with this error and I am not sure what is the proper solution at the time being - addding user teamcity to group sudo did not work.  As I installed only one agent, I "solved" it by logging to the agent machine as the teamcity user, editing name in buildAgent.properties  and executing "agent.sh start".

The last thing to do its to go to the Agents tab in TeamCity, check the agent's status, authorize it if it is not authorized and set compatible configuration so no build is run on the agent until it is really ready.

Agent Tools

TeamCity plugins can have both server and agent side. The agent-side plugins that do not load any classes into the runtime are called agent tools - it is TeamCity way for distribution of binary files to agents.

In your .BuildServer/plugins directory (default value) create directory .tools, if it is not there yet. Each agent plugin then can put either zip file or directory with the tools to distribute to all agents. The distribution starts in about 2 minutes. It is possible that a build configuration asociated with the agent is required to trigger the process.

Create directory .BuildServer/plugins/.tools/my_plugin and put your scripts inside. The files  should be accompanied with plugin descriptor teamcity-plugin.xml . If you do not create it,  an empty one is created on the agent side. The distribution process removes executable bit from all files put into the directory -- to prevent it you have to list your executable in the descriptor under "executable-files":

<?xml version="1.0" encoding="UTF-8"?>
 <teamcity-agent-plugin xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                        xsi:noNamespaceSchemaLocation="urn:schemas-jetbrains-com:teamcity-agent-plugin-v1-xml">
   <tool-deployment>
      <layout>
         <executable-files>
           <include name='path_to_executable'/>
         </executable-files>
      </layout>
   </tool-deployment>
 </teamcity-agent-plugin>

The path is relative to the plugin directorty and you do not have to start it with "./".

Calling Your Agent Tools Script

Create a build step of runner type "Command line" and set working directory of the step to %teamcity.tool.my_plugin% . Then you can execute scripts relative to the plugin directory, .e.g. with "Command executable" set to "./helloWorld.py" .

Better Looking TestNG Reports with ReportNG

We had a set of system tests using Selenium 2 Webdriver and I was not satisfied with the default TestNG reports. I was  looking for a way to make reports look better and provide all the information necessary for analysis of a test failure when is happens. There were two requirements :

1. provide nice, compact overview 
2. include screenshot of a moment of failure

First thing I tried was Allure framework - it creates very nice reports but I had to reject it after some trials because it the way it works it was incompatible with the existing tests and its also quite invasive.

Fortunately I found ReportNG after that. The default design might not be so fancy but it is still very good and it fits well into TestNG and our tests.

First we had to add necessary dependencies to our maven POM:

<dependency>
   <groupId>org.testng</groupId>
   <artifactId>testng</artifactId>
   <version>6.8.8</version>
</dependency>

<dependency>
   <groupId>org.uncommons</groupId>
   <artifactId>reportng</artifactId>
   <version>1.1.4</version>
   <exclusions>
      <exclusion>
         <groupId>org.testng</groupId>
         <artifactId>testng</artifactId>
      </exclusion>
   </exclusions>
</dependency>

<dependency>
   <groupId>com.google.inject</groupId>
   <artifactId>guice</artifactId>
   <version>3.0</version>
</dependency>

TestNG has several interfaces to hook into the test processing, the most interesting probably are ITestListener, IConfigurationListener, and sometimes IMethodInterceptor. ReportNG add class HTMLReporter to that.

To add a screenshot to the report, we need to save it ITestListener.onContextFailure() and pick it up
in a custom ReportNGUtils -- for customization we need to provide custom Velocity context by overriding createContext() and passing custom ReportNGUtils implementation.

import org.apache.commons.io.FileUtils;
import org.apache.velocity.VelocityContext;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.IConfigurationListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.uncommons.reportng.HTMLReporter;

import java.io.File;
import java.net.URL;

public class TestListener extends HTMLReporter
       implements ITestListener, IConfigurationListener
{
    protected static final CustomReportNgUtils REPORT_NG_UTILS = new CustomReportNgUtils();

    @Override 

    protected VelocityContext createContext()
    {
        VelocityContext context = super.createContext();

        // VelocityContext has three properties: meta, utils, messages 
        // - see AbstractReporter.createContext()
        context.put("utils", REPORT_NG_UTILS);

        return context;
    }

    /** Invoked when test method (method with annotation @Test) fails. */
    @Override
    public void onTestFailure(ITestResult testResult)
    {
        if (getWebDriver(testResult) != null)
        {
            File scrFile = ((TakesScreenshot) getWebDriver(testResult))
                                             .getScreenshotAs(OutputType.FILE);
            String screenshotName = createScreenshotName(testResult);

            File targetFile = new File(screenshotName);
            FileUtils.copyFile(scrFile, targetFile); 

            URL scrUrl = new URL(getDriver(testResult).getCurrentUrl()); 

            Screenshot screenshot = new Screenshot(targetFile, srcUrl );
            testResult.setAttribute(Screenshot.KEY, screenshot);
        }

    }

    // ...
}

Class Screenshot is a custom class holding screenshot-related data, bare bones version could looke as this:

class Screenshot
{
    /* Name of {@link ITestResult} attribute for Screenshot. */
    static final String KEY = "screenshot";

    /** File in which is the screenshot stored. */
    File file;

    /** URL of a web application's page the screenshot captures. */
    URL url;
}

Now we need to add the custom ReportNGUtils implementation which picks up contextual information (Screenshot instance in our case) and uses it to modify the report output.

import java.util.List;

import org.testng.ITestResult;
import org.uncommons.reportng.ReportNGUtils;

class CustomReportNgUtils extends ReportNGUtils
{
    public List<String> getTestOutput(ITestResult testResult)
    {
        List<String> output = super.getTestOutput(testResult);

        if ( testResult.getAttribute(Screenshot.KEY) != null )
        {
            Screenshot screenshot = (Screenshot) testResult.getAttribute(Screenshot.KEY);
            String screenshotFileName = screenshot.getFile().getName();

            if (screenshot != null)
            {
                String url = (String) testResult.getAttribute("screenshotUrl");
                output.add(String.format("screenshot for %s  %s <br/><img src='../screenshots/%s'>",
                                         testResult.getName(), url, screenshotFileName)
                );
            }
        }

        return output;
    }
}

The final step is to register test listener in the plugin executing the tests. We use failsafe, the configuration for surefire is similar if you desire to use it.

<build>
        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>

                <configuration>

                    <systemPropertyVariables>
                        <org.uncommons.reportng.escape-output> 

                        false 

                        </org.uncommons.reportng.escape-output>
                    </systemPropertyVariables>
                    <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary.xml</summaryFile>

                    <testClassesDirectory>${project.build.directory}/classes</testClassesDirectory> 
                    <properties>
                        <property>
                            <name>usedefaultlisteners</name>
                            <value>false</value>
                        </property>
                        <property>
                            <name>listener</name>
                            <value>
                                org.bithill.test.testng.TestListener
                            </value>
                        </property>
                    </properties>

                    <suiteXmlFiles>
                        <suiteXmlFile>src/main/resources/suiteX.xml</suiteXmlFile>
                    </suiteXmlFiles>

                </configuration>

                <executions>

                    <execution>
                        <id>integration-test</id> 

                        <phase>integration-test</phase>

                        <goals> <goal>integration-test</goal> </goals>
                    </execution>

                    <execution>
                        <id>verify</id> 

                        <phase>verify</phase>

                        <goals> <goal>verify</goal> </goals>
                    </execution>

                </executions>
            </plugin>

        </plugins>
    </build> 

And that's all - when you run 'mvn failsafe:integration-test' the tests are run, then you follow with 'mvn failsafe:verify',  which processes the results of the integration tests and generates a report and sets proper build result. Note that set the testClassesDirectory si crucial if you have it different than the expected 'test-classes'.

Book Review: Mastering Apache Maven 3

When I got my hands on Mastering Apache Maven 3  by Prabath Siriwardena, I was sceptical at first, thinking it would be another half-cooked product trying to make up for lack of information with funny stories.

I am glad I can say that's not this case at all. The book is very good and covers all aspects of Maven in satisfactory level. Any developer who wants to us Maven efficiently will probably benefit from reading it and it useful also as comprehensive Maven reference.  Yes, as a natural nit-picker I found some things missing but it the number of such things was surprisingly  low.

What is even better it goes directly to the point, not wasting paper and my time on funny stories, jokes, or other filler stuff.


You will find in it description of POM (Project Object Model), various Maven configuration options, life cycle, description of several the most popular plug-ins, assemblies, archetypes, and repository management.  The final chapter about the best practices would have saved from some pain in past if I had it.

I will definitively find it place in my bookcase.

DIY Wi-Fi Access Point on Linux

Hardware

From local vendor I bought PCI Express card with Atheros AR9485 chip complying to IEEE 802.11n. It should be versatile and well-supported via ath9k Linux kernel driver.

In case you are interested in the particular device, it is TP-LINK TL-WN781N.

Kernel 

Naturally you must have some decently recent kernel and enabled support for wireless networking, your Wi-Fi chip and bridging (will be explained later).
The more or less relevant config options on my machine looks this way:

Networking support --->
  Networking options  --->
    <M> 802.1d Ethernet Bridging
  -*- Wireless  --->
        <M> cfg80211 - wireless configuration API
        [*] enable powersave by default
        <M> Generic IEEE 802.11 Networking Stack (mac80211)
        -*- Enable LED triggers
  <M> RF switch subsystem support

Device Drivers  --->
  [*] Network device support  --->
        Wireless LAN  --->
          <M> Atheros Wireless Cards ---->
                      <M> Atheros 802.11n wireless cards support
                      [*] Atheros ath9k PCI/PCIe bus support
  -*- Enable LED triggers -->
        {M} LED Class Support
        -*- LED Trigger support

Bridging Wi-Fi and Ethernet 

To connect clients connecting to our Access Point we need to bridge our existing wired ethernet device eth0 and the wireless one, names wlp1s0 in my case.

Enable IP Forwarding 

Once we have the support in  kernel, we should enable it by default, that is done on Gentoo Linux by editing /etc/sysctl.conf and setting net.ipv4.ip_forward = 1.

You should also install bridge-utils, containing brtctl utility that can be used to query and change setting of ethernet bridges on your machine.

Setup Ethernet Bridge

Then you can adapt your network configuration. I did it by moving all configuration from eth0 to my bridge, called br0, and add eth0 to it at the boot time.  There should also be configuration for the wireless device but we do not want to add it to the bridge on boot - that's what will hostapd take care for.

So my /etc/conf.d/net looks this way after the above described changes:

modules=("ifconfig")

# wired

config_eth0="null"

# wireless

modules_wlp1s0="!iwconfig !wpa_supplicant"
config_wlp1s0="null"
channel_wlp1s0="6"
essid_wlp1s0="MY_SSID"
mode_wlp1s0="master"

# bridge

config_br0="192.168.1.250/24"
routes_br0="default via 192.168.1.254"
dns_servers_br0="192.168.1.250" #pdnsd
bridge_br0="eth0" # adding eth0 to the bridge on boot

Your configuration will probably vary, e.g.  setup of the DNS server, namely using caching DNS server pdnsd, would differ.

Don't forget to add net.br0 to the startup scripts for default run level oor you'll end with machine without network at all.

Access Point

The final piece in the puzzle is hostapd - the utility that attaches to your wireless device and changes it to Wi-Fi AP.  You have to adapt its config (/etc/hostapd/hostapd.conf on my machine) to you needs - make sure  that the settings match what you have in your network configuration:

interface=wlp1s0

bridge=br0

ssid=MY_SSID 

I also set country_code in hope that it will help me to meet wireless regulations and hw_mode. For security I set wpa  (set to 2, to avoid weak WPA 1) and wpa_passphrase. All the configuration options are well-documented so take you time and read it - further details are easily available in wikis on internet.

The final set is to test hostapd setup and to add hostapd to startup scripts for default run level. 

Reboot

If you did everything right, you can reboot. Boot will bring up eth0, then br0 and add eth0 to it, the hostapd kicks in bringing up the wireless device and adding it to the bridge - once done, AP starts to advertise itself and you can connect using predefined pass-phrase.

I hope I did not mess things too much as I am newbie to Wi-Fi, and this how-to will help you a bit.