Who Killed My Java Process?

It usually starts the same way. A Java service that ran fine for days is suddenly gone. No exception in the logs. No OutOfMemoryError. No heap dump. No graceful shutdown hook. The last log line is something completely ordinary — and then nothing.

The first instinct is to blame the application. But the application never got a say. If you check the kernel log, the real story is right there:

$ dmesg -T | grep -i "killed process"

[Tue Jul 7 03:12:44] Out of memory: Killed process 21847 (java)
    total-vm:115343360kB, anon-rss:100663296kB, file-rss:0kB

This is the OOM Killer, and the most confusing thing about it is that it has nothing to do with Java's OutOfMemoryError. Those are two entirely different failures:

Understanding why the kernel does this — and why it so often picks your Java process — requires understanding a small lie Linux tells every program.

When a process asks Linux for memory, Linux says yes. Almost always. Even when it doesn't have that much memory.

This is called memory overcommit, and it's the default behavior (vm.overcommit_memory = 0). The kernel's reasoning is actually sound: most programs allocate far more memory than they ever touch. Allocation is just bookkeeping — a promise. Physical RAM pages are only assigned when the process actually writes to a page for the first time.

This is why a JVM can start with -Xmx100g on a 128 GB box that's already running other services and scheduled jobs — and nothing complains. The kernel granted a 100 GB promisewithout checking whether, together with everyone else's promises, it can actually be kept. The bill only comes due as the heap actually grows and pages get touched. And when the machine truly runs out of physical memory and swap, the kernel cannot honor its promises anymore.

At that point it has two options: freeze the entire machine, or kill somebody. It kills somebody.

The OOM Killer isn't random. Every process gets a badness score from 0 to 1000, and the highest scorer dies. The dominant factor is simple: how much physical memory (RSS) the process is using. Roughly, the score is the fraction of total memory the process occupies, in thousandths.

badness ≈ (process RSS + swap used) / total memory × 1000
        + oom_score_adj   (a per-process tunable, -1000 to +1000)

-1000  →  never kill this process
+1000  →  kill this one first

You can inspect any process's current score at /proc/<pid>/oom_score, and nudge it via /proc/<pid>/oom_score_adj.

Now here's the punchline: on a typical server, which process has the largest RSS? The JVM. Always the JVM. A Java service with a multi-gigabyte heap towers over every sshd, cron, and monitoring agent on the box. So even if some rogue script is what pushed the machine over the edge, the kernel looks around, sees the biggest thing in the room, and shoots it.

Which brings me to a configuration I keep seeing in the wild, and the one that motivated this post:

java -Xms512m -Xmx100g -jar service.jar

The thinking goes: "Start small, and let the heap grow only if it's ever needed. 100 GB is just a safety ceiling." It feels conservative. It is actually one of the most dangerous JVM configurations you can run — and to see why, you need to understand what the garbage collector is actually promising you.

The GC's only contract is: keep heap usage under -Xmx.That's it. It makes no promise whatsoever about physical machine memory. The GC doesn't know your machine has 128 GB. It doesn't know three other services and a nightly batch job share the box. Its entire universe is the heap, and its ceiling is the number you gave it.

Here's the failure sequence, step by step:

Machine: 128 GB RAM, shared with other services + nightly jobs.
JVM: -Xms512m -Xmx100g

1. JVM starts. Committed heap: 512 MB. Everyone is happy.

2. Load arrives. Heap fills. Does GC panic? No — usage is
   a fraction of Xmx. The cheapest move is to GROW the heap,
   not to collect aggressively. Committed: 8 GB... 40 GB...

3. GC runs occasionally, frees objects INSIDE the heap.
   But freed heap space stays committed to the JVM —
   most collectors return memory to the OS reluctantly,
   slowly, or (historically) never. RSS only ratchets UP.

4. Committed heap: 85 GB. Plus metaspace, thread stacks,
   code cache, direct buffers... JVM RSS: ~95 GB.
   Other services on the box: ~25 GB. Total: ~120 GB.

5. The nightly batch job fires. It needs 15 GB.
   Kernel: out of physical pages. Badness scores computed.
   Biggest RSS on the machine? The JVM. SIGKILL.

6. The GC never saw it coming. From its perspective, it was
   at 85% of Xmx — under its limit, no OutOfMemoryError to
   throw. The JVM never hit ITS limit. It hit the MACHINE's
   limit — a limit it never knew existed.

This is the cruel part. The garbage collector was working exactly as designed the whole time. With a 100 GB ceiling, it had no reason to work hard — growing the heap is always cheaper than a major collection, so that's what it chose, every time. An absurdly high -Xmx doesn't just permit high memory usage — it actively reduces GC pressure, which means garbage accumulates in ever-growing heap regions instead of being collected.

And note what the process was killed with: anon-rss:100663296kB — about 96 GB resident — while the application's liveobjects might have been a fraction of that. The rest was garbage sitting in committed regions the GC hadn't bothered to collect, plus freed regions it hadn't returned to the OS.

One more thing bites people even when the heap math looks right: the JVM uses significant memory outside the heap. If you size -Xmxto exactly your container or machine limit, you've already lost.

A good working rule: total JVM RSS ≈ heap + 25–50% extra, more if you use a lot of threads or off-heap buffers. In containers, this is exactly why processes with -Xmx set to the full container limit get OOM-killed by the cgroup even though the heap "never filled up." You can see the real breakdown with -XX:NativeMemoryTracking=summary and jcmd <pid> VM.native_memory summary.

The classic advice — repeated in a decade of tuning guides — is to set -Xms equal to -Xmx. And the reasoning is legitimate:

The case for Xms = Xmx. The heap never resizes, so you never pay resize pauses. Combined with -XX:+AlwaysPreTouch, every page is committed and touched at startup — so your memory footprint is fully known from second one. And that has a beautiful failure property: if the machine can't give you the memory, you find out at startup, in a deploy window, with someone watching — not at 3 AM three weeks later when the heap finally grew into a page the kernel couldn't deliver. Fixed footprint also means capacity planning is honest: the machine either fits its workload or it visibly doesn't.

But there's a real counterargument, and it applies to any machine that isn't dedicated to a single always-on service. Say your service genuinely needs 8 GB at peak but idles at 2 GB, and the same box runs scheduled jobs — nightly batch processing, report generation, backfills — that need a few gigabytes when they fire. With Xms = Xmx = 8g, the JVM squats on 8 GB of committed memory around the clock, including the 20 idle hours a day. That memory is reserved but mostly garbage or empty— and your nightly job now can't get the memory it needs, even though the machine is, in any meaningful sense, mostly free.

So the honest answer is: it depends on who else lives on the box.

The "uncommitting GC" part matters for the shared-host case, because it fixes the ratchet problem from earlier. Modern collectors can return freed memory to the OS — but you have to pick one that does, and sometimes ask nicely:

So a reasonable shared-host configuration looks like: -Xms2g -Xmx8g with G1 or ZGC on a modern JDK — the heap grows under load, shrinks when idle, and the nightly batch job gets its memory back.

Putting it all together, the defense has four layers:

1. Size honestly. Set -Xmx so that heap + ~30% non-heap overhead + everything else on the machine fits in physical RAM. In containers, prefer -XX:MaxRAMPercentage=75 over a hardcoded Xmx — it adapts to the container limit automatically. The single most common root cause of Java OOM kills is simply an Xmx the machine could never honor.

2. Monitor RSS, not just heap. Every Java dashboard shows heap usage. Almost none show process RSS — and RSS is the only number the kernel cares about. Alert when RSS approaches the machine or cgroup limit. A widening gap between RSS and live heap is your early warning.

3. Tell the kernel who matters. If one process on the box is critical, lower its oom_score_adj so the kernel kills the batch job and not the database:

4. Contain the blast radius. Run scheduled jobs under their own cgroup memory limits (systemd-run --scope -p MemoryMax=4G ./nightly-job.sh). Then a misbehaving job hits its ownwall and dies alone, instead of driving the whole machine into a global OOM where the kernel picks the victim — which, as we've established, will be your JVM.

The Linux OOM Killer and the JVM garbage collector are both memory managers — but they manage different things and never talk to each other. The GC keeps heap under -Xmx; the kernel keeps the machine alive. When you set an Xmx the machine can't deliver, you create a gap between those two contracts, and your process lives in that gap until the kernel closes it with a SIGKILL.

Give the JVM a ceiling the machine can honor, watch RSS and not just heap, pin Xms = Xmx only when the machine is yours alone — and on shared boxes, use a collector that gives memory back.The OOM Killer only shoots when someone wrote a check the kernel couldn't cash. Don't let it be you.

• ↗Taming the OOM KillerLWN.net — the classic deep dive on badness scoring
• ↗JEP 346: Promptly Return Unused Committed Memory from G1OpenJDK — why G1 finally learned to give memory back
• ↗Linux Memory Overcommitment and the OOM KillerBaeldung — overcommit modes explained