Funnel Data Collection: Log vs. Kafka — An Engineering Decision Record

1. The Trigger: A Funnel Data Gap

We recently launched a new authentication flow that migrated users away from our legacy system. While the feature worked perfectly, our analytics dashboards went blank because the legacy logs—which our analytics engine ingested to track user drop-offs and funnel steps—were no longer being printed.

The first reaction was to replicate the old logging pattern in the new authentication services. However, the data team proposed a different approach: "Do not use logs; publish these events directly to Kafka instead."

2. The Fallacy: "Kafka is More Reliable for Analytics"

Kafka is a fantastic tool for event streaming. It guarantees at-least-once delivery, maintains event order, and protects against data loss.

But does funnel analytics data need that level of guarantee?

Funnel data is statistical. We calculate aggregate percentages and conversion rates (e.g., "What percentage of users dropped off between entering their phone number and verifying their OTP?"). Losing a few events during a network hiccup might change a conversion rate from 84.12% to 84.11%, which has zero business impact. Unlike ledger entries, financial transactions, or audit logs, funnel metrics do not require strict, transaction-level consistency.

From a data reliability standpoint, Kafka was overkill.

3. The Real Driver: Consumer Convenience

When we discussed this with the data team, we discovered that their preference for Kafka was not actually about transport-level reliability. It was about operational convenience:

• No Log Parsing: Text logs are difficult to parse, and formats frequently break. Kafka topics enforce a structured JSON schema, ensuring consistent data ingestion.
• Replayability: Kafka offsets make it easy to restart consumers and reprocess events in case of downstream database failures.
• Resource Isolation: Running Elasticsearch queries against production databases for analytics reports introduces operational risk. Consuming from Kafka isolates the analytics workload entirely.

The data team did not necessarily need Kafka as the transport; they needed a **structured, replayable event channel that avoided text log parsing**.

4. The Cost: Producer-Consumer Asymmetry

While Kafka provided massive benefits for the consumer (the data team), it introduced high costs for the producers (the backend services):

• Diverse Technology Stacks: The authentication flows spanned multiple backend services written in different languages (Java, Kotlin, Python). Adding Kafka producers to all of them meant building and maintaining multiple client configurations.
• Legacy and Fade-out Services: Some services in the flow were scheduled for deprecation. Adding new infrastructure dependencies to a dying service is a clear anti-pattern.
• Network Management: Connecting some services to the data team's separate Kafka cluster required setting up new VPC peering, security groups, and SSL certificate management.

5. The Solution: Decoupling at the Infrastructure Layer

Instead of forcing the backend applications to publish directly to Kafka, we moved the integration to our logging infrastructure:

This pattern achieved zero application-level cost. The backend services printed standard logs, while the logging agent handled Kafka publishing. The data team received their structured JSON events on their Kafka topic, and the backend team avoided managing additional client dependencies in legacy services.

Important: Relying on stdout logging under high traffic can cause performance bottlenecks due to synchronous I/O blocking. Ensure you configure your application to use asynchronous logging (such as Logback's AsyncAppender or Log4j2's LMAX Disruptor-based AsyncLogger) to prevent logging from slowing down application throughput.