Arquillian and TestNG

The RHQ Project has historically run on JBoss AS 4.2.3.  That is changing as we leap forward to JBoss AS 7.1.1.Final for our next release.  There are a lot of advantages to this move but it also presented a lot of challenges, one of which was our integration test approach. 

When deployed on AS4 RHQ used the JBoss Embedded Container to perform unit/integration tests on domain (JPA persistence) and server jar (SLSB) classes. Now that we deploy on AS7, the embedded container was no longer an option. The de facto alternative is Arquillian. Arquillian is a tool that integrates multiple test frameworks, namely JUnit and TestNG, against several containers, namely Glassfish, OpenEJB and, most importantly for us, AS7.

For a more detailed account of this change in RHQ, see this wiki page about the testing impact.  Here I will just summarize some lessons learned and tips for using Arquillian with TestNG.
This is written with reference to Arquillian version 1.0.2.


Arquillian runs the TestNG lifecycle for every test

This is critical to understand. It means that BeforeXXX/AfterXXX methods are all called for every test. Effectively this means that the only methods you can effectively use are BeforeMethod and AfterMethod.

Avoid BeforeClass/AfterClass or other flavors of these methods.

If you absolutely must you may be able to use "stand-ins" for these by using @Test methods instead. By using dependencies, or priority, you can sort of simulate BeforeClass and AfterClass. But note that test priority is evaluated only in the scope of the group its test is in. Also realize that BeforeClass and AfterClass often cause unexpected issues due to test class interactions, and stand-ins can present the same problems. Any startup/teardown actions must be impervious to random test orderings amongst all classes.

Do not share data with instance variables.

The test class is effectively new'd for each test. This can make life difficult. Even though dependencies in test order execution are honored, it does not mean that you can set variable 'foo' in testA and use the value of 'foo' in testB. You have a few alternatives:

• Store and fetch data in the DB.
  This only works if the DB data somehow survives the local Before/After methods, and those of any other test class.

• Store and fetch data to a file by serializing data.
  A simple util is available in AbstractEJB3Test. (This is the class that all of the relevant RHQ integration tests extend. Feel free to take a look.)

• Just write a single, big test, that spans all of the tests you were thinking of splitting up and sharing data between.
  This is probably the easiest route. In this case I suggest writing the tests separately, as private methods, as if they were real tests. And then creating a single consolidatedTest that calls them each in order.

Know that BeforeMethod and AfterMethod are actually invoked twice per test.

Arquillian has two modes of testing, client and in-container. Client testing is such that the tests are basically remote clients and hit the deployed server in that fashion. We use the other mode, which is to run the tests in the container, meaning we can inject InitialContext, EntityManager, etc.. into the tests. Due to architectural integration issues, Arquillian calls BeforeMethod twice, once for each mode. We need to execute our code only for the in-container invocation.

The preferred mechanism for a BeforeMethod or AfterMethod is now to extend the base class methods offered in AbstractEJB3Test. So:

// NOT recommended
@BeforeMethod
public void myBeforeMethodName();

// Recommended
// By overriding the method you'll get built in in-container invocation only, as well as standard error handling.  Note that there are a few flavors
// of the method, some taking parameters.
// IMPORTANT: use protected, not public.  Public methods are considered test methods by testng if the class is annotated with @Test.
@Override
protected void beforeMethod();

// If you must use the annotation ensure you protect the invocation
@BeforeMethod( some attributes you just have to have )
public void myBeforeMethodName() {
  if ( !inContainer()) {
    return;
  }
  ...
}

Know that false positives result if exceptions are thrown in BeforeMethod

Warning! Currently if an exception is thrown from your BeforeMethod it will be swallowed and the test will appear to pass. I'm not sure if there is a way to get the test to fail, currently I am wrapping the logic in a try/catch and printing blatant warnings in the output.

Know that there is an extension for running several test classes against a single deployment

By default Arquillian with testng will for each class in the test suite:

1. Start container
2. Deploy test deployment
3. Undeploy test deployment
4. Shutdown container

Although good for test exclusivity, it is prohibitively slow for a project with many test classes that doesn't need that level of test isolation. Working with the Arquillian developers we've put in place an Arquillian extension that allows us a single container start and deploy. Our extension has only nominal changes applied to the extension provided at https://gist.github.com/3975179.

It should be noted that this extension uses Arquillian's (not testng) BeforeClass/AfterClass hooks, so those become unavailable for other use, but that is not an issue for us as we don't have any other extensions and observing those (or any other) events.

Hope this helps.