Why RecoveryAfterSoftCommitTest went flaky

The symptom: clean for months, then flaky from June 10

"Flaky" here means a run failed and then passed when Gradle retried it. No run ever failed for good. Data: the test's Develocity test history (all branches) and its Jenkins failure history in Hossman's reports on fucit.org.

Flaky runs per month, all branches

Develocity has no September runs of this test. Jenkins shows it ran all month (1–2% failures), so that gap is on Develocity's side.

Jenkins weekly failure rate (Solr Jenkins jobs, all branches)

Source: the test's Jenkins failure history on fucit.org (data: failure-rate-history.csv). Two steps line up with two commits: up the week #4451 merged, and down to ~2% right after SOLR-18401 started retrying requests that were never sent. What's left is the committed case. The two most recent main failures (#15202, #15239) both show it: IOException occurred when talking to server on the cut replica.

Since June 10, by branch

The client stack: who sends the update

The test calls one method, cloudClient.add(doc). Under it sit five layers. Click a layer to see what it does here. The orange edge marks what #4451 changed.

The race, step by step

Step through what happens in the microseconds after the test cuts the replica off. The path splits at step 5, depending on whether Jetty has already noticed the cut. Switch between the two outcomes to compare them.

Code

Why LBSolrClient won't retry it

When a request to one replica fails, LBSolrClient can try the next one (here, the leader). For queries it does that freely. For updates it's strict, because replaying an update the server already applied could apply it twice. Pick a failure to see where it goes:

The experiment: same test, macOS vs Linux

A purpose-built test, RecoveryAfterSoftCommitProxyRaceTest, cuts the proxy, waits a fixed delay, and sends one update straight to the cut replica. Each round is classified as committed (not retryable, the flaky failure) or not sent (safe to retry). The chart shows the share of rounds that failed committed.

pause-then-close freezes the proxy first, so the request is fully written (committed) but never forwarded, then closes it. It fails committed every time on both systems: a deterministic repro, proving Solr never even received a request the client refuses to retry.

The fix: don't send updates to the replica you just cut off

First: what this test is actually for

It helps to know the three states a document goes through in Lucene:

The test came in with SOLR-6640, "Replication can cause index corruption" (fix by Shalin Shekhar Mangar, Jan 2015). It checks that a replica holding soft-commit searchers can still recover by copying the whole index after a partition:

maxBufferedDocs=2    3 adds force a flush: segment files on disk, but not committed
soft commit          a searcher opens on those flushed-but-uncommitted files
proxy.close()        the replica is cut off; the leader keeps indexing
numRecordsToKeep=2   the replica falls too far behind to replay updates → it needs a full index copy
proxy.reopen()       recovery must discard the uncommitted files the searcher holds open,
                     copy the leader's index, and come back ACTIVE without corruption

The 2014 bug. Leader and replica each name their own segment files (_0.cfs, _1.cfs…), so the same name can hold different documents on each node. During the full copy, the replica rolled back its writer to discard its uncommitted segments, but its soft-commit searcher still held them open, so the stale files survived. The fetcher then skipped any leader file whose name already existed locally:

// SnapPuller.downloadIndexFiles(), 2014
if (!slowFileExists(indexDir, file.get(NAME)) || downloadCompleteIndex) {
  ... fetch from the leader ...
} else {
  LOG.info("Skipping download for " + file.get(NAME) + " because it already exists");
}

Its only guard, isIndexStale(), compared sizes. A leftover file with the leader's name and size was kept, the replica took the leader's commit point, and that commit point now pointed at the replica's own different documents: "shard2 is not consistent. Got 62 … and got 24". It needed all of: a soft-commit searcher on uncommitted segments, a full copy rather than an update replay, and a same-named, same-sized leftover. The leader was never affected. The fix closes the searcher first, rolls back, and waits until the stale files are really deleted, falling back to a copy into a fresh directory. (The name-and-size mechanism is our reading of the 2014 code and the fix's comments, not reproduced.)

The adds during the cut exist only to push the leader past numRecordsToKeep; which replica the client sends them to is not what the test checks.

Leaders-first client, or a retry inside the test?

#5098 asks exactly this. Leaders-first is the right call: it delivers the adds to the leader, which is all the test needs. A retry loop inside the test would replay updates that the client deliberately refuses to replay (CheckRetryUnrollTest asserts that ClosedChannelException is not retried). And what the client does while a replica is unreachable is a separate concern, already covered by SOLR-18401's tests; if anyone wants more coverage of it, it belongs in its own test.

Where the failure rate came from, and where it goes

Measured: 100 runs of each variant

Each variant starts from the real RecoveryAfterSoftCommitTest and changes exactly one thing. Two of them change wasCommError(): when a request fails, CloudSolrClient asks it whether this was a communication problem, and if so refreshes cluster state and retries the whole request, possibly on another replica. Otherwise the error goes straight back to the caller.

Results. Every variant forces the risky routing (updates may go to any replica) on every run, so these numbers are the worst case, not the Jenkins rate. Linux in Docker, 100 iterations each.

No client-side retry can be safe here: "committed" means the HEADERS left the client, and in general the server may have applied the update. So the fix belongs in the test. The test is about how the leader handles an unreachable replica, so its own updates should go to the leader, which is never the cut replica. That's what #5098 (SOLR-18530) does, and what Hossman suggested on SOLR-18188 in June:

    proxy.close();
    try (CloudSolrClient leaderClient =
        createNewCloudSolrClient(zkServer.getZkAddress(), DEFAULT_COLLECTION, true, 30000, 120000)) {
      for (; i < MAX_DOCS; i++) {
        ...
        leaderClient.add(document);   // shardLeadersOnly = true → directUpdate → leader
      }
    }

What happened to the other tests #4451 touched

Why branch_9x never flaked

branch_9x's test uses the old Apache HTTP/1.1 client. You might expect it to handle the cut replica better. It doesn't: it was protected by a retry one layer up, and it reached the cut replica less often in the first place.

The old client fails more on a fresh connection, until CloudSolrClient retries it

Old Apache client on branch_9x, Linux (Docker). The first three bars count single updates in the experiment harness; the last two count runs of the real test.

Where the retry lives

protected boolean wasCommError(Throwable rootCause) {
  return super.wasCommError(rootCause)
      || rootCause instanceof ConnectTimeoutException
      || rootCause instanceof NoHttpResponseException;
}

protected boolean wasCommError(Throwable t) {
  return SolrException.hasCause(t, SocketException.class)
      || SolrException.hasCause(t, UnknownHostException.class)
      || SolrException.hasCause(t, RequestNotSentException.class);
}

#4451 changed three things at once

A minor factor: Apache re-checks a pooled connection after 3s idle. It then gets refused, and its load balancer fails over to the leader (5/5 in the idle-gap rounds). In the real test the connection is usually only 14–123ms old, so this rarely applies.

How we found it

1. When and where. Develocity history by month, then week, then day: the first flaky runs were on Jun 10–11. main and branch_10x went flaky; branch_9x stayed clean.
2. Which commit. Five candidates landed on main around Jun 8–10, with different 10x backport dates. 10x went flaky the day #4451 was backported (Jun 17), when the other four were already there.
3. How it fails. The Jenkins stack trace: the add right after proxy.close(), sent to the cut replica's port, failing with Broken pipe after the request had started.
4. Down the stack. The routing coin flip, then LBSolrClient's rule that updates are retried only if provably not sent, then HttpJettySolrClient's commit flag, then the Jetty source: "committed" means written to the local socket buffer.
5. Prior work. Hossman and David analysed the flakiness on SOLR-18188 in June. In SOLR-18401 (Aug 24) David traced the regression to the lost wasCommError retry, used the same fucit.org rates, and shipped the "not sent" fix. Nick's #5098 has the test fix.
6. Prove the race. An experiment test on main: never on macOS, ~20% on Linux, 100% under load, plus a deterministic pause-then-close repro.
7. Cross-check with Jenkins. Hossman's fucit.org reports matched the Jun 8 week onset, showed SOLR-18401's step down to ~2%, filled Develocity's September gap, and confirmed that today's failures are the committed kind.
8. Measure the fix. 100 runs per variant, all forcing the risky routing: 16/100 failures before, 0/100 with #5098's leaders-only client, with recovery exercised every time.
9. Why did 9x pass? A 9x experiment comparing both clients, then the real 9x test instrumented and pinned to "any replica", which revealed the cloud-level retry.

What this investigation adds to SOLR-18188 and SOLR-18401

• The residual ~2%, explained: the old client also replayed committed requests (NoHttpResponseException fires after the write). The new rule refuses those, and only routing to the leader removes them.
• The routing difference: on 9x the helper's "any replica" was a no-op setter (~25% exposure); #4451 made it real (~50%).
• The race is Linux-only (never on macOS), with a deterministic pause-then-close repro.
• The 8% → 2% drop, quantified: about 4 in 5 hits fail "not sent", which is exactly what SOLR-18401 retries.

Theories that turned out wrong

Epilogue: what one flaky test revealed

The test was written in 2015 as the regression test for one bug. Eleven years later, its flakiness pointed at two more things, each one layer deeper:

• 2015, why it exists: an index-corruption bug in recovery with soft-commit searchers (SOLR-6640). That bug was found through ChaosMonkeySafeLeaderTest; this test came in with the fix, to keep it from coming back.
• Aug 2026: a real SolrJ regression. Production clients behind idle-timeout load balancers or firewalls would get hard failures on requests that never reached Solr. SOLR-18401's own write-up says how it was found: "RecoveryAfterSoftCommitTest went from zero CI failures to a sustained 3-10% failure rate."
• Today: its last ~2% marks the edge of what a client can safely replay, and shows the test's own routing was the remaining problem (#5098).

Treat a flaky test as a signal, not noise.

Method lessons

• Use branch timing as a natural experiment to pick one commit out of several.
• Measure before explaining. Several plausible explanations here were wrong.
• Make the failure deterministic (pause-then-close) so it can be studied at will.
• Test where CI runs. The race doesn't show up on macOS.
• Instrument the real path. The harness said the old client fails 100%; only the real test showed the retry that rescued it.
• Expect harness bugs. A wrong doc id, load-balancer state leaking between rounds, and overwritten results could each have misled us.

Glossary

SocketProxy

A test helper sitting between clients and a Solr node. close() closes every connection through it and stops accepting new ones: a fake network partition.

HTTP/2 session vs stream

A session is one long-lived TCP connection; each request is a stream on it. Jetty keeps sessions pooled (up to 4 per host, never closed for idleness here) and reuses them.

HEADERS / DATA frames

An HTTP/2 request is a HEADERS frame (method, path, headers) followed by DATA frames (the body). Solr streams the JavaBin body, so DATA goes out after HEADERS.

Committed

Jetty's event that the request's HEADERS were written. It means written into the local socket buffer, not received by the server.

Selector

Jetty's I/O thread that reads from sockets. Jetty only learns the peer closed the connection when the selector reads end-of-stream (filled < 0).

FIN / RST / EPIPE

TCP close signals. After a FIN the local kernel still accepts a write; the peer answers it with an RST, and the next write fails with EPIPE, "Broken pipe".

RequestNotSentException

Solr's marker that a request failed before anything was sent, so it's safe to replay elsewhere (SOLR-18401).

Zombie

LBSolrClient's list of endpoints that failed retryably; it skips them and pings them every 60s to see if they're back.

wasCommError

CloudSolrClient's check for a "communication error". If true, it retries the whole request, possibly on another replica. The old CloudLegacySolrClient added Apache's NoHttpResponseException to it.

NoHttpResponseException

Apache HttpClient's error when a request was written but the connection closed before any response arrived.

Flush / hard commit / soft commit

A flush writes buffered documents as a new segment on disk, not yet committed. A hard commit writes a commit point (segments_N) and fsyncs. A soft commit opens a searcher straight on the writer's current state, so flushed but uncommitted segments become searchable without being durable.

numRecordsToKeep / PeerSync / full replication

A replica that missed only a few updates catches up by replaying them from the leader's update log (PeerSync). If it missed more than numRecordsToKeep, it has to copy the whole index from the leader (full replication, via IndexFetcher).

shardLeadersOnly

A CloudSolrClient setting. True: updates go straight to shard leaders. False: any active replica, shuffled. The test framework picks it at random.