A JUnit 5 test class passes locally but fails intermittently in CI — diagnose it
This test class is green on every developer machine and green when either test is run alone, but roughly one CI build in four fails on expiresOldOrder with expected: <true> but was: <false> — and re-running the same build usually makes it pass.
Constraints:
- the production code is correct and must not change
- "just retry it" is not an acceptable answer; name what makes the outcome vary
class OrderServiceTest {
private static final List<Order> ORDERS = new ArrayList<>();
@Test
void createsOrder() {
ORDERS.add(new Order("A-1", LocalDate.now()));
assertEquals(1, ORDERS.size());
}
@Test
void expiresOldOrder() {
ORDERS.add(new Order("A-2", LocalDate.now().minusDays(30)));
Order first = ORDERS.get(0);
assertTrue(first.isExpired()); // ← fails here, sometimes
}
}Diagnose the cause.
The tests share mutable state through the static list and depend on execution order, which JUnit 5 does not guarantee. When createsOrder runs first, ORDERS.get(0) is today's order, which is not expired — so the assertion fails. Run alone or in the other order, it passes. Fix by making the fixture per-test (a non-static field, reset in @BeforeEach) and asserting on the order the test itself created, not on index 0.
- ✗Assuming JUnit runs test methods in declaration order
- ✗Calling a retry rule a fix — it hides the shared state instead of removing it
- ✗Sharing a fixture through a
staticfield to 'save time' in setup
- →Why is a wall-clock read such a common source of flakiness, and how do you remove it?
- →Why does
@TestMethodOrdermake the suite pass without making it correct?
What makes the outcome vary
Two independent defects, and both come down to the same thing: the test depends on something it does not control.
- Shared mutable state.
ORDERSisstatic, so one list belongs to the whole class and survives between tests. JUnit 5 builds a fresh test-class instance per method — but that does not reset a static field. - Order dependence. JUnit does not guarantee method order: with no
MethodOrdererconfigured it is deterministic-but-opaque (it depends on reflection and can differ between JDKs and builds). Hence "one build in four".
If createsOrder runs first, today's order is already in the list by the time expiresOldOrder runs, and ORDERS.get(0) returns that one, not the order the test created itself:
OrderServiceTest > expiresOldOrder() FAILED
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at OrderServiceTest.expiresOldOrder(OrderServiceTest.java:18)ORDERS.get(0) is order A-1, dated LocalDate.now(). It is of course not expired.
The fix
class OrderServiceTest {
private List<Order> orders; // ✅ instance field, not static
@BeforeEach
void setUp() {
orders = new ArrayList<>(); // ✅ fresh fixture for every test
}
@Test
void expiresOldOrder() {
Order old = new Order("A-2", LocalDate.now().minusDays(30));
orders.add(old);
assertTrue(old.isExpired()); // ✅ assert on OUR order, not on get(0)
}
}Each test now sees only what it created itself: the order stops mattering, and the failure stops reproducing.
⚠️ @TestMethodOrder(OrderAnnotation.class) would "fix" the suite — and that is the trap. It only pins one order while leaving the tests coupled through shared state: the first new test inserted in the middle brings the failure straight back. A retry rule hides the problem in exactly the same way.
⚠️ The second, quieter dependency is the wall clock: LocalDate.now() makes the test depend on the day it runs. Where a boundary matters in production code ("expired after 30 days"), inject a Clock and pass Clock.fixed(...) — then the result depends on neither the date nor the agent's time zone.
