Fixing an OOM Only to Create Zombie Batch Pods: The JVM Non-Daemon Thread Trap

1. The Outage: Running but Completed

Our batch system schedules daily maintenance tasks (such as sending bulk notification messages or compiling records) using Kubernetes CronJobs.

A few days after shipping a performance improvement, our database administrator flagged a critical alarm: "Database connections for the batch user have surged to 100% capacity."

We checked the Kubernetes cluster and found that multiple pods from the previous days' runs were still in the Running state. Looking at the logs of those pods, the batch job had actually finished hours or days ago:

The business logic had finished, but the process was refused to exit. Because the pods remained active, they kept holding onto their database connection pools. As new CronJobs spawned daily, connection leaks accumulated until the database connection pool was fully exhausted.

2. The Trigger: Bounding the Thread Pool

The trigger was a fix we had deployed to solve a thread leak.

To send bulk notifications asynchronously, the batch job relied on Spring's @Async annotation. By default, if you do not declare a custom executor, Spring falls back to using SimpleAsyncTaskExecutor.

SimpleAsyncTaskExecutor does not reuse threads. It spawns a new thread for every single task.

During peak notification windows, this created thousands of concurrent threads, resulting in the dreaded java.lang.OutOfMemoryError: unable to create native thread.

To resolve this, we configured a custom, bounded ThreadPoolTaskExecutor:

@Bean
fun asyncExecutor(): Executor {
    val executor = ThreadPoolTaskExecutor()
    executor.corePoolSize = 10
    executor.maxPoolSize = 20
    executor.queueCapacity = 1000
    executor.initialize()
    return executor
}

This successfully solved the OOM. But it introduced the zombie pod problem.

3. The Root Cause: Non-Daemon Threads and JVM Lifecycle

To understand why the process refused to die, we must look at how the Java Virtual Machine (JVM) decides to terminate.

The JVM terminates only when all active threads are **daemon** threads. If even a single **non-daemon** thread remains active, the JVM will stay alive.

Comparing our two executors explained the behavioral shift:

• SimpleAsyncTaskExecutor (Legacy): Spawns threads on-demand, but those threads terminate immediately after their short task completes. Once the batch job finished, all spawned threads died, the main thread returned, and the JVM exited cleanly.
• ThreadPoolTaskExecutor (New): Maintains its core pool of 10 threads permanently active in memory, waiting for new work. In Java, threads created by a thread pool default to being **non-daemon** threads. Because these 10 core threads remained alive, the JVM never met its termination condition, causing the pod to stay in the Running state.

We confirmed this by taking a thread dump (jstack) of a zombie pod. The dump showed our async executor threads sitting in a waiting on condition state, blocking JVM shutdown:

"asyncExecutor-1" #24 prio=5 os_prio=0 cpu=12.5ms elapsed=86400s tid=0x00007f... nid=0x1e waiting on condition
  java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007...> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

4. The Fix: Thread Lifecycle Management

For batch and CLI-like short-lived processes, you have two primary ways to resolve this thread pool lifecycle issue.

Option A: Enable Daemon Threads

If your asynchronous tasks are safe to abort during JVM exit, configure the thread pool to create daemon threads. Since daemon threads do not block JVM termination, the process will exit cleanly when the main thread completes:

@Bean
fun asyncExecutor(): Executor {
    val executor = ThreadPoolTaskExecutor()
    executor.corePoolSize = 10
    executor.maxPoolSize = 20
    executor.setDaemon(true)  // Threads will not block JVM exit
    executor.initialize()
    return executor
}

Option B: Graceful Shutdown (Recommended)

If your tasks must complete before the application shuts down to avoid data corruption, enable graceful shutdown on the executor. This tells Spring to shut down the executor when the context closes, waiting for active tasks to finish before terminating:

@Bean
fun asyncExecutor(): Executor {
    val executor = ThreadPoolTaskExecutor()
    executor.corePoolSize = 10
    executor.maxPoolSize = 20
    executor.setWaitForTasksToCompleteOnShutdown(true)
    executor.setAwaitTerminationSeconds(60)
    executor.initialize()
    return executor
}

Alternatively, you can manage this globally in Spring Boot using configuration properties: