Lost Update on a Shared Counter

Code

class ProcessedRecordCounter {
    private volatile int count;

    void increment() {
        count++;
    }

    int value() {
        return count;
    }
}

Prod Symptoms

A Spring singleton, Kafka consumer, retry tracker, or in-process aggregator keeps a custom counter in a field. Multiple request or worker threads increment it inside the same JVM.

• The admin endpoint or custom metric reports fewer completed events than audit rows, Kafka commits, request logs, or traces prove happened
• The gap grows as request rate, consumer concurrency, or worker count increases
• No exception is thrown; CPU, GC, and thread pools can look normal
• Repeated load tests produce different totals
• Restart resets the counter but does not fix the race

Run Locally

• Expected count is THREADS multiplied by INCREMENTS_PER_THREAD
• Actual count is often lower
• The amount lost changes from run to run
• volatile does not save the increment because counter++ still reads, computes, and writes
• There is usually no useful stack trace because nothing crashes
• The final read after join() is intentionally safe enough for the demo; the mismatch is caused by non-atomic updates

Inspect hints

• Increase THREADS or INCREMENTS_PER_THREAD if your machine happens to produce the expected value
• Thread dumps rarely prove this race; the mismatch between expected and actual is the useful signal

javac LostUpdateCounterDemo.java
java LostUpdateCounterDemo
for i in {1..5}; do java LostUpdateCounterDemo; done

public class LostUpdateCounterDemo {
    private static final int THREADS = 8;
    private static final int INCREMENTS_PER_THREAD = 500_000;

    private static volatile int counter = 0;

    public static void main(String[] args) throws InterruptedException {
        Thread[] workers = new Thread[THREADS];

        for (int i = 0; i < THREADS; i++) {
            workers[i] = new Thread(() -> {
                for (int j = 0; j < INCREMENTS_PER_THREAD; j++) {
                    counter++;
                }
            }, "incrementer-" + i);
        }

        for (Thread worker : workers) {
            worker.start();
        }
        for (Thread worker : workers) {
            worker.join();
        }

        int expected = THREADS * INCREMENTS_PER_THREAD;
        System.out.println("expected = " + expected);
        System.out.println("actual   = " + counter);
        System.out.println("lost     = " + (expected - counter));
    }
}

Note: The result is probabilistic. volatile makes the final value visible, but increments are still not atomic.

Explanation

The ++ operator is not a single atomic action. It is a read, a calculation, and a write.

Key signal: Lost updates happen when shared read-modify-write state has no atomicity boundary.

• Two threads can read the same old value at the same time
• Both threads compute the same next value
• Both threads write that same next value back
• One completed increment is overwritten by the other
• volatile makes reads and writes visible, but it does not make counter++ atomic

How to Diagnose

Use evidence, not thread dumps, as the primary signal.

• Compare the custom aggregate with an authoritative source: audit table, Kafka offsets, committed records, request logs, or traces
• Repeat under load and vary thread count, request concurrency, or consumer concurrency
• Search for shared fields updated with ++, --, +=, get-then-set, or AtomicLong.get() followed by set()
• A thread dump rarely proves the race after the fact

expected = 4000000
actual   = 2867314
lost     = 1132686

Note: Exact numbers vary. The important evidence is actual being lower than expected.

How to Fix

• Use AtomicLong or AtomicInteger for simple exact per-JVM counters
• Use incrementAndGet(), getAndIncrement(), addAndGet(), or updateAndGet(); do not split read and write
• For cross-pod counters, use an external atomic operation: Redis INCR/INCRBY, an atomic SQL update, optimistic locking, or a durable state store
• Keep the shared counter private so all updates go through the safe API

import java.util.concurrent.atomic.AtomicLong;

public class LostUpdateCounterFixed {
    private static final int THREADS = 8;
    private static final int INCREMENTS_PER_THREAD = 500_000;

    private static final AtomicLong counter = new AtomicLong();

    public static void main(String[] args) throws InterruptedException {
        Thread[] workers = new Thread[THREADS];

        for (int i = 0; i < THREADS; i++) {
            workers[i] = new Thread(() -> {
                for (int j = 0; j < INCREMENTS_PER_THREAD; j++) {
                    counter.incrementAndGet();
                }
            }, "incrementer-" + i);
        }

        for (Thread worker : workers) {
            worker.start();
        }
        for (Thread worker : workers) {
            worker.join();
        }

        int expected = THREADS * INCREMENTS_PER_THREAD;
        System.out.println("expected = " + expected);
        System.out.println("actual   = " + counter.get());
    }
}

Note: AtomicLong makes the increment indivisible inside this JVM. For a multi-pod service, decide whether the counter is intentionally per-instance or must be stored in an external atomic system.