In the following, I want to show you how to test if your tomcat web application is thread-safe. As an example application, I use Jenkins deployed on an apache tomcat 9.0.

To detect concurrency bugs during our tests we use vmlens. vmlens traces the test execution and analyzes the trace afterward. It detects deadlocks and race conditions during the test run.

Testing

To enable vmlens we add it as java agent to the CATALINA_OPTS in catalina.sh on Linux or catalina.bat on windows:

CATALINA_OPTS="-javaagent:<Path to agent> -Xmx8g"

We also set a high enough heap size. After running Jenkins and executing some build jobs we see the following report in vmlens:

Analyzing

Let us look at one of the races found, the race at accessing the field hudson.UDPBroadcastThread.shutdown.

The thread "Jenkins UDP 33848 monitoring thread" reads the field in the race and the thread "localhost-startStop-2" writes it. Let us look at the class and the reading method run() and writing method shutdown().

public class UDPBroadcastThread extends Thread {
    private boolean shutdown;
public void run() {
        try {
            mcs.joinGroup(MULTICAST);
            ready.signal();
            while(true) {
                byte[] buf = new byte[2048];
                DatagramPacket p = new DatagramPacket(buf,buf.length);
                mcs.receive(p);
                SocketAddress sender = p.getSocketAddress();
                // prepare a response
                TcpSlaveAgentListener tal = jenkins.getTcpSlaveAgentListener();
                StringBuilder rsp = new StringBuilder("");
                tag(rsp,"version", Jenkins.VERSION);
                tag(rsp,"url", jenkins.getRootUrl());
                tag(rsp,"server-id", jenkins.getLegacyInstanceId());
                tag(rsp,"slave-port",tal==null?null:tal.getPort());
                for (UDPBroadcastFragment f : UDPBroadcastFragment.all())
                    f.buildFragment(rsp,sender);
                rsp.append("");
                byte[] response = rsp.toString().getBytes("UTF-8");
                mcs.send(new DatagramPacket(response,response.length,sender));
            }
        } catch (ClosedByInterruptException e) {
            // shut down
        } catch (SocketException e) {
            if (shutdown) { // forcibly closed
                return;
            }            // if we failed to listen to UDP, just silently abandon it, as a stack trace
            // makes people unnecessarily concerned, for a feature that currently does no good.
            LOGGER.log(Level.INFO, "Cannot listen to UDP port {0}, skipping: {1}", new Object[] {PORT, e});
            LOGGER.log(Level.FINE, null, e);
        } catch (IOException e) {
            if (shutdown)   return; // forcibly closed
            LOGGER.log(Level.WARNING, "UDP handling problem",e);
            udpHandlingProblem = true;
        }
    }
      public void shutdown() {
        shutdown = true;
        mcs.close();
        interrupt();
    }
}

The field shutdown is a nonvolatile field. It is read in line 28 and 35 in the method run and written in line 41 in the method shutdown

Since the field hudson.UDPBroadcastThread.shutdown is not volatile, it is not guaranteed that the "Jenkins UDP 33848 monitoring thread" sees the values set by the "localhost-startStop-2" thread.

The "Jenkins UDP 33848 monitoring thread" might for example run on the first core while "localhost-startStop-2" on the second core of a multi-core CPU. The write to a normal field does not invalidate the cache of the cores. Therefore the "Jenkins UDP 33848 monitoring thread" still sees the cached old value.