Occasionally I’ll see flickering tests. Sometimes they’re green, sometimes they’re red and this can happen without any code changes. What bugs me the most is that when trying to fix the problem, I can never be sure that I haven’t just been lucky and the green I’m seeing isn’t really a false positive. I’ll have to manually run the test several times before my confidence grows.
In an attempt to ease the situation, I created an
@Intermittent annotation with a corresponding JUnit
Runner. Now, I can mark up a suspect test and get JUnit to do the repetition. Joy.
You can then use the
IntermittentRule to run the test method repeatedly.
Or use the
@RunWith annotation to run the test using the
What’s interesting here is the way in which the
Runner interact with
JUnit. Newer versions of JUnit have introduced the idea of
Statements. Using a
Rule allows access to the underlying
Statement which in
our case is the action to run the test method. So we’re able to run the
underlying statement again and again. Nice.
Runner however can hook into JUnit’s framework in a different way. It can
access more than one statement and so can position itself slightly differently.
What this means for us is that when using the
Rule above, any
methods will only be run once but the test method will run multiple times.
Runner above however, will run any
once for each test repetition.
The code is available as part of the tempus-fugit library, feel free to look around.