Phi failure detectors: suspicion is not a failure probability

A score and a decision are separate

The 2004 JAIST report separates a continuously valued suspicion signal from an application's action threshold. It defines phi as the negative base-10 logarithm of the model's probability of a later heartbeat. Its implementation fits a normal distribution to recent arrival intervals. [1, section IV]

phi(d) = -log10 S(d)

Here d is time since the last observed heartbeat, not a disk command's latency; S(d) is the assumed inter-arrival survival function. A modeled tail of 0.001 gives phi 3. This identity does not supply a failure prior, an application-health test or a measured false-alarm rate.

In Cassandra's documented architecture, UP/DOWN decisions are local to each observer. A conviction changes request routing; it is not automatic removal of the endpoint from membership. Different observers can therefore require different incident explanations. [3]

Cassandra 5.0.0 does not use that normal tail

In the pinned source, ArrivalWindow.phi() returns elapsed time divided by the sampled mean. The threshold comparison multiplies that raw ratio by 1 / ln(10); the JMX PHI values apply the same factor. The resulting score is d / (mean * ln(10)), algebraically the negative log-tail of an exponential model. [2]

The implementation also seeds its window, filters arrival intervals and suppresses convictions around detected local pauses. Those details matter: the formula alone is not the running detector. Raw diagnostic values and scaled JMX scores should not be compared as if they had identical units. [2, report, interpret, ArrivalWindow and getPhiValues]

This is a version-pinned source reading, not a statement that all Cassandra releases or all phi detectors behave identically. We did not run a Cassandra cluster.

The same silence can produce very different scores

Hesela's calculation fixes an invented mean interval of one second. The normal approximation has an invented standard deviation of 0.1 seconds; the exponential model has the same mean but a different spread. We do not fit either model to observations or claim equal variances.

At 1.5 seconds the normal example exceeds phi 6, while the exponential example remains below 1. This is not evidence that one detector is better. It is evidence that a threshold has meaning only with its model, estimated parameters and implementation. Copying a numeric threshold between systems does not preserve an alert policy.

The calculation uses an untruncated normal approximation, which mathematically permits negative intervals. With these illustrative parameters its negative tail is tiny, but this is still an approximation, not a physical arrival generator. The code computes the upper tail directly with erfc to avoid subtracting a near-one cumulative probability.

The missing input is the failure prior

Consider a separate hypothetical experiment: at the start of an observation interval a node is either failed or alive. Let its prior failure probability be p; assume a failed node cannot produce the next observed heartbeat. Let s = 0.001 be the probability of no heartbeat by the evaluation time when alive. Bayes' rule gives:

P(failed | no heartbeat) = p / (p + (1 - p) * s)

Exact conditional probabilities, in row order: 1000/10999 and 1000/1099.

One phi score is compatible with both results because it did not encode the prior. Real systems also need to account for when failures can occur and whether already-sent heartbeats can arrive after a failure. Our deliberately simple example omits those mechanisms. Neither row is an estimate for a production fleet.

Suspicion does not fence a stale writer

Hesela's design implication: record detection, recovery policy and enforcement as separate events. A membership observer can stop selecting a replica while an old request still exists elsewhere. A high score neither cancels that request nor proves that its write was absent.

Use the existing analyses of storage fencing and stale writes and timeouts and uncertain completion for those different boundaries. A healthy heartbeat is also narrower than a healthy storage workload; see observer-relative failures.

What was checked

The report's evaluation replayed a week of UDP heartbeat observations between Switzerland and Japan. Neither monitored machine crashed. Detection time was estimated under an assumed crash timing, not measured from a population of failed storage nodes. This limits extrapolation to today's clusters. [1, section V]

Our contribution is the explicit model comparison and Bayesian counterexample above. There are no sampled devices, benchmark runs or confidence intervals: these are deterministic calculations with stated assumptions. A real evaluation would need labeled incidents, observation intervals, pauses, losses, model calibration and costs of incorrect suspicions.

Reproduce the numbers with model.py, run the tests, or inspect full-precision JSON. In one directory run python3 -B model.py, then python3 -B -m unittest discover -v. Original code and synthetic results are CC0; cited publications retain their own rights.

Definitions: failure detector and phi accrual failure detector.

Primary sources

1. Hayashibara, Defago, Yared and Katayama, The phi accrual failure detector, JAIST IS-RR-2004-010 (10 May 2004), sections IV-V.
2. Apache Cassandra 5.0.0: FailureDetector.java, ArrivalWindow.phi, interpret and getPhiValues.
3. Apache Cassandra documentation: Dynamo, Ring Membership and Failure Detection (reviewed 17 October 2026).

Reviewed 17 October 2026. Source [1] is the JAIST technical report, not an asserted identical conference edition. No unverified DOI, cluster experiment, universal threshold or measured SEO benefit is claimed.