Preventing OOM in Large Data Rendering Workers: Chunking and PDF Merging

Batch workers that process and render documents often run into memory limits when dealing with power users. When our document generation worker suffered repeated crashes on large accounts, the root cause was a combination of open-ended lists and rendering-library limitations. This post walks through the diagnostics, the chunk-based rendering solution, and a secondary consumer idempotency issue it exposed.

1. The Symptom: Unhealthy Readiness Probes

Our document generation worker (a Kotlin/JVM service running on Kubernetes) began restarting repeatedly. The K8s events showed:

Warning  Unhealthy  Readiness probe failed: /actuator/health/readiness: context deadline exceeded
Normal   Killing    Container failed liveness probe, will be restarted

The pod logs showed that fetching the dataset was taking an enormous amount of time, while the actual rendering step was relatively fast:

create-template-context: success (220765 ms)   ← 3 minutes and 40 seconds
render-html: success (6038 ms)                  ← 6 seconds

Because the JVM was configured with -XX:+ExitOnOutOfMemoryError, the container was terminating immediately when it ran out of heap space. During the crash, the JVM became unresponsive, causing the Kubernetes liveness probe to timeout and trigger a container kill.

2. The Root Cause: Triple In-Memory Copies

Our worker allows users to export up to 4 years of transaction history. For highly active accounts, this can include hundreds of thousands of transactions.

The service code was performing the following steps:

It split the query into 90-day chunks, fetched the records, and accumulated them into a single, open-ended MutableList (Copy 1).
It mapped these records to UI display items (Copy 2).
It split the items into pages for Thymeleaf layout templates (Copy 3).
Finally, the PDF rendering library (openhtmltopdf) converted the entire HTML DOM into a single, massive PDF document.

Having three distinct copies of hundreds of thousands of objects in the JVM heap, combined with the heavy memory footprint of the HTML-to-PDF rendering engine (which builds the entire layout DOM in memory), easily exceeded the 10GB JVM heap limit (-Xmx10240m) and crashed the process.

3. The Solution: Chunked PDF Rendering and Merging

Because the PDF rendering library does not support streaming layout output, we could not simply stream the HTML to a file. We had to keep the memory footprint bounded.

The solution was to render the document in paginated chunks and merge them using PDFBox.

New Generation Pipeline:

Fetch the transaction history using a database cursor or Kotlin Sequence (avoiding loading everything into a list).
Group the items into chunks of N pages (e.g., 50 pages per chunk).
Render each chunk to a separate, temporary PDF file on disk. This keeps the in-memory HTML DOM size small and predictable.
Use Apache PDFBox's PDFMergerUtility to merge the temporary PDF files into the final output document.
Clean up the temporary files.

This pattern capped our memory usage. The memory complexity dropped from O(N) where N is the total history length, to O(1) in terms of heap space, as the maximum heap usage was limited to the size of a single 50-page chunk.

4. Secondary Discovery: The Duplicate Event Trap

During our investigation of the OOM crashes, we uncovered an interesting user complaint: "Sometimes, I receive a success email for my document, followed immediately by a failure email, and my document download status is marked as failed."

The log analysis revealed the following sequence:

02:19:35.263 worker [Attempt 1] EVENT_RECV (messageId=a303feee)
02:19:35.270 worker [Attempt 1] calls /api/fetch -> returns 200 OK
02:22:45.582 worker [Attempt 1] PDF generated successfully
02:22:45.644 worker [Attempt 1] calls /api/complete -> returns 200 OK (Sends Success Email)
02:22:45.650 worker [Attempt 2] EVENT_RECV (messageId=a303feee) *Re-delivered*
02:22:45.654 worker [Attempt 2] calls /api/fetch -> returns 500 (Status is already COMPLETED)
02:22:45.665 worker [Attempt 2] fails with FeignException
02:22:45.688 worker [Attempt 2] calls /api/fail -> returns 200 OK (Sends Failure Email & marks failed)

What happened: Because the generation took over 3 minutes, the Kafka broker sometimes assumed the consumer was dead and initiated a consumer group rebalance. This caused the message to be re-delivered to another worker.

The fix: We updated the worker and the server API to be idempotent:

If the API returns a status showing the job is already completed, the worker skips processing and immediately acknowledges the Kafka offset.
The server refuses to overwrite a COMPLETED status with a FAILED status.

Tip: To prevent Kafka rebalances on long-running operations, you should also increase max.poll.interval.ms or decouple the Kafka consumer thread from the actual processing thread by passing the task to a background thread pool while keeping the consumer thread polling.

5. Conclusion

When building batch workers, memory constraints require careful layout design. Avoid loading entire datasets into memory. Splitting documents into logical chunks, rendering them individually, and merging them at the file system level is a highly effective way to keep memory usage flat. Furthermore, because long-running tasks are prone to network timeouts and re-deliveries, ensuring that your task processor is idempotent is critical to preventing race conditions.