Monster Scale Summit

Summary

The talk explains replacing embedded RocksDB state in Kafka Streams with ScyllaDB. Kafka Streams originally depends only on Kafka and stores state locally, which causes slow recovery, scaling limits, and operational pain. Using ScyllaDB externalizes state, decouples compute and storage, and enables stateless stream processors that scale cleanly in Kubernetes. The speaker walks through production use at Metronome, details the ScyllaDB data model, fencing with lightweight transactions, and lessons learned around latency, consistency, and cluster sizing. Kafka Streams to

Topics discussed

• What Kafka and Kafka Streams are and how embedded RocksDB is used for state
• Why RocksDB causes availability, scaling, and performance problems in stateful Kafka Streams applications
• When cloud databases like DynamoDB and orchestration with Kubernetes changed original Kafka Streams assumptions
• How replacing RocksDB with ScyllaDB decouples state from compute and enables stateless stream processing
• How a production Kafka Streams pipeline at Metronome uses ScyllaDB for real-time spend limits
• How the ScyllaDB data model maps Kafka partition keys, offsets, and epochs for state management
• How lightweight transactions in ScyllaDB fence zombie Kafka Streams tasks
• Why disk behavior, Bloom filters, and node sizing strongly affect latency in ScyllaDB
• How ScyllaDB consistency levels (quorum, all, one) affect correctness and latency

Takeaways

• Externalizing Kafka Streams state to ScyllaDB removes long restore times and makes failures routine instead of disruptive. Recovery no longer requires replaying large changelog topics into local storage, which improves availability under real load.
• Decoupling compute from state enables true elasticity. Kafka Streams applications become stateless, which fits Kubernetes well and allows independent scaling of stream processors and the database.
• Lightweight transactions are essential for correctness when multiple processors may write the same key, but they are expensive. Use them only where fencing is required, and remove them when write conflicts are impossible.
• ScyllaDB performance issues often surface as disk problems first. High read latency and Bloom filter misbehavior can dominate end-to-end stream latency, even when application code looks fine.
• Consistency settings matter operationally, not just theoretically. Moving between quorum and faster read modes requires careful sequencing and repair to avoid serving stale data during incidents.

Top takeaway

Moving Kafka Streams state out of embedded RocksDB and into ScyllaDB fundamentally changes what is possible. ScyllaDB lets you keep Kafka Streams fast, stateless, and elastic while still maintaining correctness at scale, which is hard to achieve with local state under real production failures.