vmlens Manual

Why vmlens?

Running your tests with multiple threads does not work. Bugs depend on a specific thread interleaving, which is often impossible to reach by simply rerunning your test multiple times. And data races only occur on specific hardware architectures and JVMs.

Therefore vmlens uses the Java Memory Model to execute all possible thread interleavings and to check for data races in the program flow. This blog post describes how vmlens uses the Java Memory Model to test all thread interleavings.

Using vmlens

The following example shows how to write multi-threaded tests with vmlens. All examples are in this GitHub project:

import com.vmlens.api.AllInterleavings;
public class TestUpdateWrong {
    public void update(ConcurrentHashMap<Integer, Integer> map) {
        Integer result = map.get(1);
        if (result == null) {
            map.put(1, 1);
        } else {
            map.put(1, result + 1);
        }
    }
    @Test
    public void testUpdate() throws InterruptedException {
        try (AllInterleavings allInterleavings = 
                new AllInterleavings("TestUpdateWrong");) {
	// surround the test with a while loop, iterationg over
	// the class AllInterleavings
            while (allInterleavings.hasNext()) {
                final ConcurrentHashMap<Integer, Integer> map = 
                        new ConcurrentHashMap<Integer, Integer>();
                Thread first = new Thread(() -> {
                    update(map);
                });
                Thread second = new Thread(() -> {
                    update(map);
                });
                first.start();
                second.start();
                first.join();
                second.join();
                assertEquals(2,map.get(1).intValue());
            }
        }
    }
}

In your test method, you surround the code you want to test with a while loop iterating over the class AllInterleavings. vmlens executes the block inside the while loop multiple times, for each thread interleaving once. If the test fails vmlens shows the thread interleaving which led to the failure. If the test succeeds vmlens shows the last thread interleaving.

The above example test fails, and vmlens reports the interleaving which led to the failed assertion:

In maven, you can see this report by clicking on the link TestUpdateWrong in the file target/interleave/elements.html. In eclipse you can see the report by clicking on the link TestUpdateWrong in the view under Window -> Show View -> Other... -> vmlens -> vmlens Explorer.

The maven reports are described here. The eclipse views are described here.

How to run the test

You can run the test in eclipse using the vmlens run short cut for JUnit. Right click on the JUnit class -> Run As -> JUnit Test traced with vmlens.

To run the test with maven put the vmlens interleave plugin in your maven pom.xml as described here.

Data races

vmlens analyses a trace of the test run to detect data races and deadlocks. A data race happens when reading and writing to a shared variable is not correctly synchronized. A data race means that thread might read stale or inconsistent values.

The following example shows a data race in an inconsistent synchronized test.

public class TestInconsistentSynchronized {
	private static final Object LOCK_1 = new Object();
	private static final Object LOCK_2 = new Object();
	int i = 0;
	@Test
	public void test() throws InterruptedException {
		try (AllInterleavings allInterleavings = 
				new AllInterleavings
				("TestInconsistentSynchronized");) {
			while (allInterleavings.hasNext()) {
				Thread first = new Thread(() -> {
					synchronized (LOCK_1) {
						i++;
					}
				});
				Thread second = new Thread(() -> {
					synchronized (LOCK_2) {
						i++;
					}
				});
				first.start();
				second.start();

				first.join();
				second.join();
			}
		}
	}
}

For the above test, vmlens reports the interleaving which led to the data race:

In maven, you can see this report by clicking on the link TestInconsistentSynchronized in the file target/interleave/elements.html. In eclipse you can see the report by clicking on the link TestInconsistentSynchronized in the view under Window -> Show View -> Other... -> vmlens -> vmlens Explorer.

The maven reports are described here. The eclipse views are described here.

Deadlocks

To detect a deadlock during a test run both threads need to request the locks at the exact same time. To make deadlock detection timing independent, vmlens analysis the order in which the threads request the locks. If this order contains a cycle vmlens has found a deadlock.

public class TestUpdateRecursive {
	private final ConcurrentHashMap&lt;Integer, Integer&gt; 
		map = new ConcurrentHashMap&lt;Integer, Integer&gt;();
	public TestUpdateRecursive() {
		map.put(1, 1);
		map.put(2, 2);
	}
	public void update12() {
		map.compute(1, (key, value) -&gt; {
			map.compute(2, (k, v) -&gt; {
				return 2;
			});
			return 2;
		});
	}
	public void update21() {
		map.compute(2, (key, value) -&gt; {
			map.compute(1, (k, v) -&gt; {
				return 2;
			});
			return 2;
		});
	}
	@Test
	public void testUpdate() throws InterruptedException {
		try (AllInterleavings allInterleavings = 
				 new AllInterleavings(&quot;TestUpdateRecursive&quot;);) {
			while (allInterleavings.hasNext()) {
				Thread first = new Thread(() -&gt; {
					update12();
				});
				Thread second = new Thread(() -&gt; {
					update21();
				});
				first.start();
				first.join();

				second.start();
				second.join();
			}
		}
	}
}

For the above test vmlens generates the following report:

In maven, you can see this report by opening the file target/interleave/issues.html. In eclipse you can see the report by opening the view under Window -> Show View -> Other... -> vmlens -> vmlens Issues.

The maven reports are described here. The eclipse views are described here.

Tips for writing multi-threaded tests

When writing multi-threaded tests using vmlens you should follow the following tips:

  1. Make your tests repeatable: vmlens executes the code inside the while loop multiple times. So your code inside the while loop must be repeatable.
  2. Use few threads: There is no need to use many threads to make the chances higher to execute a specific thread interleaving. Since vmlens executes all thread interleavingfs it is often better to use only a small amount of threads.

The vmlens maven plugin

Installation and running tests

To use vmlens with maven, configure a plugin tag to tell maven that the vmlens plugin should be executed at the test phase. And include the jar com.vmlens.api as test dependency.

<project>
<!-- to include the class AllInterleavings into the test class path.  -->	
<dependency>
  <groupId>com.vmlens</groupId>
  <artifactId>api</artifactId>
  <version>1.1.0</version>
  <scope>test</scope>
</dependency>	
	
<build>
  <plugins>
<!-- to run the vmlens maven plugin during the maven test phase  -->	 
    <plugin>
    <groupId>com.vmlens</groupId>
    <artifactId>interleave</artifactId>
    <version>1.1.0</version>
    <executions>
      <execution>
        <goals>
          <goal>test</goal>
        </goals>
      </execution>
    </executions>
    </plugin>
     ...
    </plugins>
</build>
      ...
</project>

maven plugin configuration

The vmlens interleave plugin is based on the maven surefire plugin. All configs which apply to the maven surefire plugin apply also to the interleave plugin, except the test fork configs.

The following parameters are additions from the vmlens interleave plugin:

doNotTrace

The doNotTrace let you exclude methods from tracing through vmlens. A method matching the doNotTrace tag and all methods called by this method will not be traced by vmlens. In the following example the method om.vmlens.test.TestMethod.signal and all methods of the class com.vmlens.test.TestMavenDoNotTrace will not be traced:

<doNotTrace>
	<method>com.vmlens.test.TestMethod.signal</method>
	<method>com.vmlens.test.TestMavenDoNotTrace.*</method>
</doNotTrace>

vmlens supports the ant directory pattern for method names. So you can use the following wild cards:

  • * Matches zero or more characters (not including the path separator .)
  • ** Matches zero or more path segments.
  • ? Matches one character (any character except the path separator .)

Examples:

  • com.vmlens.** Matches all methods from classes in the package and sub-packages from com.vmlens
  • com.vmlens.test.TestAllRaces.* Matches all methods in the class com.vmlens.test.TestAllRaces

suppress

The suppress tag let you suppress data races. If a data race is suppressed it will no longer show in the issue report and the build will no longer fail because of this data race. The following example shows how to suppress a data race at the field com.vmlens.test.race.TestAllRaces.count and at the array com.vmlens.test.race.ArrayElement[].

<suppress>
    <race>com.vmlens.test.race.TestAllRaces.count</race>
	<race>com.vmlens.test.race.ArrayElement[]</race>
</suppress>

trace

Trace let you narrow the methods which get traced. If trace is configured vmlens will only search for data races inside methods matching the trace elements. Also when showing the stack trace where the data race was found vmlens will start at the first method matching a trace element.

The following example shows how to narrow the search and stack trace to methods from classes to the package com.vmlens.test.

<trace>
	<method>com.vmlens.test.state.report.**</method>
</trace>

vmlens supports the ant directory pattern for method names. So you can use the following wild cards:

  • * Matches zero or more characters (not including the path separator .)
  • ** Matches zero or more path segments.
  • ? Matches one character (any character except the path separator .)

Examples:

  • com.vmlens.** Matches all methods from classes in the package and sub-packages from com.vmlens
  • com.vmlens.test.TestAllRaces.* Matches all methods in the class com.vmlens.test.TestAllRaces

The reports

The issues report

The issues report shows you all the issues found by vmlens. vmlens detects data races and deadlock by analyzing the execution trace of your test. The report is located in the file maven output folder target/interleave/issues.html.

The example report shows one data race, the symbol and one deadlock, the symbol . The data races happened at accessing the field example.TestDataRace.i .The deadlock happened between the method lockOneToLockTwo and lockTwoToLockOne. Click one of the links to see the stack traces of the data race or deadlock.

The data race report

After clicking on the data race you see the following report:

This Html site shows you the stack trace and thread name of the reading thread, the symbol and for the writing thread, the symbol .

The deadlock report

After clicking on the deadlock you see the following report:

This report shows you the thread names of the two threads in the deadlock, the symbol ,and the stack trace. The position in the stack trace in which a monitor was acquired is marked with the symbol .

The interleave report

The interleave report shows you the thread interleavings which led to an assertion error or to a data race. The report is located in the file maven output folder target/interleave/elements.html.

The interleave detail report

The interleave detail report shows you one thread interleaving of an interleave loop. The report shows you all synchronization actions and data races in the order of occurrence. In the above example, first, a thread with thread id 13 wrote to the variable i which led to data race. After that, the same thread entered a synchronized block, and so on. The following symbols are used in this report:

The following symbols are used in this report:

  • A write which led to a data race
  • A read which led to a data race
  • A write to a volatile field
  • A read to a volatile field
  • A monitor or lock enter
  • A monitor or lock exit

The stack trace report

The stack trace report shows the stack trace of a method call:

The vmlens eclipse plugin

Install

Install from marketplace:

  1. Start Eclipse (version 4.4 or greater)
  2. Drag to your running Eclipse* workspace. *Requires Eclipse Marketplace Client

Or install directly from update site:

  1. Start Eclipse
  2. Select Help>Install New Software…
  3. Work with: https://vmlens.com/download/site/

To use the class AllInterleavings you need to include the jar api-1.1.0.jar into your classpath. You can download this jar from maven central here.

Running JUnit tests

To run a unit test use the junit vmlens shortcut:

The vmlens configuration for JUnit runs is described here

Running all other types of application

To run vmlens with another type of application, create a "vmlens All Applications" run configuration. Select Run -> Run Configurations... from the main menu. And create a "vmlens All Applications" run configuration:

After pressing run vmlens shows you the following dialog:

To run vmlens with an application on your local machine either inside or outside of eclipse add the vmlens agent string to the VM arguments of your application. To run on a remote machine export the agent files to a folder by pressing the export button.

Running on a remote machine

Copy this folder to your remote machine and configure your application with an agent string pointing to the agent.jar in this folder:

-javaagent:<Path of agent.jar>/agent.jar

When you start your application the vmlens agent creates a folder vmlens in the working directory of your application. After you stopped your application the agent stops and you can import the folder vmlens in eclipse: Select File -> Import... from the main menu and select the vmlens Agent files wizard as shown below:

Now vmlens will analyze the execution trace of your application and open the vmlens perspective in eclipse showing you the results.

Please set the java heap size for eclipse in the eclipse.ini to a high enough value, for example -Xmx5g

Changing the path of the vmlens folder

In the default configuration, the agent writes the execution trace in a folder called vmlens in the working directory of your application. To change this path, change the property eventDir in the file called run.properties in the folder which contains the agent.jar file. In the default configuration, this property is set to ./vmlens/

eventDir=./vmlens/

Launch Configuration

In the run configuration you can configure the following properties:

trace

Trace let you narrow the methods which get traced. If trace is configured vmlens will only search for data races inside methods matching the trace elements. Also when showing the stack trace where the data race was found vmlens will start at the first method matching a trace element.

do not trace

vmlens does not trace when inside a method which matches a pattern in this list.

exclude from stacktrace

Methods matching the given pattern will not be shown in stack traces, for example in the stack trace for a deadlock or data race.

suppress

A data race name matching this pattern, will not be shown in the issue view.

Pattern matching guide

vmlens supports the ant directory pattern for method names. So you can use the following wild cards:

  • * Matches zero or more characters (not including the path separator .)
  • ** Matches zero or more path segments.
  • ? Matches one character (any character except the path separator .)

Examples:

  • com.vmlens.** Matches all methods from classes in the package and sub-packages from com.vmlens
  • com.vmlens.test.TestAllRaces.* Matches all methods in the class com.vmlens.test.TestAllRaces

The views

The issues view

The issue report shows you all data races and deadlocks. Open the view you by going to Window -> Show View -> Other... -> vmlens -> vmlens Issues.

The example report shows one data race, the symbol and one deadlock, the symbol .

For the data race the view shows you the stack trace and thread name of the reading thread, the symbol and for the writing thread, the symbol .

For the deadlock the view shows you the thread names of the two threads in the deadlock, the symbol ,and the stack trace. The position in the stack trace in which a monitor was acquired is marked with the symbol .

The interleave view

The interleave report shows you the thread interleavings which led to an assertion error or to a data race. Open the view you by going to Window -> Show View -> Other... -> vmlens -> vmlens Explorer.

The above example report shows you the three potential outcomes of the interleave loop:

  • A successful loop, the symbol
  • An interrupted loop, the symbol
  • And a loop containing a data race, the symbol .

By clicking on the successful interleave loop you see the last thread interleaving of this loop. By clicking on the interrupted loop you see the thread interleaving which led to the interrupt. And by clicking on the loop with a data race you see the thread interleaving which led to the data race.

The interleave detail report

The interleave detail report shows you one thread interleaving of an interleave loop. The report shows you all synchronization actions and data races in the order of occurrence. In the above example, first, a thread with thread id 13 wrote to the variable i which led to data race. After that, the same thread entered a synchronized block, and so on. The following symbols are used in this report:

The following symbols are used in this report:

  • A write which led to a data race
  • A read which led to a data race
  • A write to a volatile field
  • A read to a volatile field
  • A monitor or lock enter
  • A monitor or lock exit

The stack trace report

The stack trace report shows the stack trace of a method call:

The vmlens classes and annotations

vmlens provides one class and one annotation to define the behavior of vmlens. You can download those classes from maven central or using the following dependency in your maven pom:

<dependency>
  <groupId>com.vmlens</groupId>
  <artifactId>api</artifactId>
  <version>1.1.0</version>
</dependency>

The JavaDoc is available here.

Class AllInterleavings

The class com.vmlens.api.AllInterleavings let you test all thread interleavings for your test. Enclose your test in a while loop to iterate through all thread interleavings like in the following example:

import com.vmlens.api.AllInterleavings;
public class TestUniqueId {
 long firstId;
 long secondId;
 @Test
 public void testUniqueId() throws InterruptedException {
  try (AllInterleavings allInterleavings = 
    new AllInterleavings("ConcurrencyTestUniqueId");) {
    while (allInterleavings.hasNext()) {
		firstId  = 0L;
		secondId = 0L;
		UniqueId uniqueId = new UniqueId();
		Thread first = new Thread(() -> {
			firstId = uniqueId.nextId();
		});
		first.start();
		secondId = uniqueId.nextId();
		first.join();
		assertTrue(firstId != secondId);
    }
  }
 }
}

Annotation DoNotTrace

The com.vmlens.annotation.DoNotTrace annotation lets you define methods that should not be traced by vmlens. This allows you to enforce a specific ordering of your thread without influencing the data race detection of vmlens. In the following example, the CountDownLatch creates a happens-before relation between the i++ in the methods updateBefore and updateAfter:

import com.vmlens.annotation.DoNotTrace;
public class UpdateWithDoNotTrace {
	private int i = 0;
	final	CountDownLatch countDownLatch = 
		new CountDownLatch(1);	
	@DoNotTrace
	private void signal() {
		countDownLatch.countDown();
	}
	@DoNotTrace
	private void wait4Signal() 
			throws InterruptedException {
		countDownLatch.await();
	}
	public void updateBefore() {
		i++;
		signal();
	}
	public void updateAfter() 
			throws InterruptedException {
		wait4Signal();
		i++;
	}
}

So without DoNotTrace vmlens would not flag this as a data race. With DoNotTrace detects a data race when the two methods are run in parallel.

Errors and warnings

Maximum operation count reached

This error happens when a thread executes more than the configured maximum of synchronization actions. This typical happens when the test contains a not terminating loop.

You can set the maximum by using the method maximumSynchronizationActionsPerThread​ of the class AllInterleavingsBuilder. In the following example the value is set to 1000:

try (AllInterleavings testUpdate = AllInterleavings.
		builder("TestShowSharedMemory").
		maximumSynchronizationActionsPerThread​(1000).
		build();)  {
	while( testUpdate.hasNext()  ) {   
			
	}
}

The default value is 2000. Set the maximum to -1 to never terminate a thread with this mechanism.

© 2020 vmlens Legal Notice Privacy Policy