Building a Faster (Rust-Based) Python Driver for ScyllaDB

Binding Rust to Python with PyO3, zero-copy deserialization with yoke

Since Python is one of the most commonly used languages for interacting with a database, having a robust ScyllaDB Python driver is essential. The first ScyllaDB Python driver was created ages ago, back when ScyllaDB was in the 2.x stage, and it’s been used ever since.

As part of the ZPP Project at the University of Warsaw, we decided to create a new Python driver for ScyllaDB. Why build a new Python driver when ScyllaDB already has one? Well, there are several key reasons:

1. “One driver to rule them all” – writing a new Python driver gave us a chance to align it with ScyllaDB’s current strategy: implementing official drivers on top of the high-performance Rust Driver. Since this has already been successfully done for other languages (you can read more about them in their blogposts: CPP RS Driver, Node.js RS Driver, and C# RS Driver), following this strategy was a natural step forward.
2. Maintainability – one of the main advantages of this approach is improving maintainability by reusing the Rust Driver’s implementation. Instead of duplicating complex driver logic across several different languages, we can use the existing one.
3. Performance boost – it’s no secret that Python is significantly slower than Rust. The CPython execution model introduces inherent constraints that limit CPU-bound operations implemented directly in Python. Issues such as dynamic dispatch, high object allocation overhead, and the Global Interpreter Lock (GIL) severely limit performance. Consequently, delegating most of the work to Rust in the new driver was expected to substantially reduce request execution times (spoiler: it did :)).
4. Code quality and typing – in the legacy Python Driver, many invariants are checked at runtime rather than at compile time. That leads to limited compile-time guarantees, as expected in a dynamically typed language. Transitioning to Rust with its static typing drastically improves the driver’s internal code readability, safety, and reliability.

Looking at all these advantages, there was no doubt that this approach could work well. One question remained, though: how do we connect Rust with Python?

Binding technology

We considered two approaches for integrating Rust and Python:

1. Writing our own binding layer – this approach would give us maximum flexibility to adapt the integration to our exact needs. On the other hand, it would introduce significant complexity and boilerplate code, especially when robust, pre-existing solutions are already available.
2. Using PyO3 – PyO3 is a library designed specifically for creating Rust bindings for Python. The main advantage is that it automatically generates the necessary boilerplate to expose our Rust structures and methods directly to Python, and that makes driver development significantly easier.

Because PyO3 is highly convenient, we ran benchmarks – hoping it wouldn’t introduce significant overhead.

PyO3 benchmarks

PyO3 overhead for small values is negligible

Benchmark Scenario	Python Throughput	Rust via PyO3 Throughput	Speedup / Overhead
Empty Function Call (0 args, returns None)	12.45M ops/s	15.60M ops/s	~1.25x faster
Pass Small Arguments (5x i32 inputs)	8.47M ops/s	7.65M ops/s	~0.90x (slight overhead)
Return Small Argument (0 args, returns i32)	8.94M ops/s	10.23M ops/s	~1.14x faster

Function calls and small returns are faster in Rust. Passing small arguments to a function is slightly slower using PyO3, but this is expected due to (minimal but present) integer conversion. Overall, PyO3 overhead for small values is no problem.

Rust efficiently constructs Python objects and passing Python objects into Rust is extremely cheap

Benchmark Scenario	Python Throughput	Rust via PyO3 Throughput	Speedup / Overhead
Empty Class Instantiation	9.25M ops/s	12.91M ops/s	~1.40x faster
One-Field Class Creation	6.10M ops/s	12.22M ops/s	~2.00x faster
Field Lookup & Arithmetic (1 lookup + 2 additions)	7.34M ops/s	6.77M ops/s	~0.92x (slight overhead)
Passing Object (PyAny reference in Rust)	8.76M ops/s	9.72M ops/s	~1.11x faster

Rust #[pyclass] instantiation can be 2 times faster than Python’s own class creation.

Attribute lookup and list indexing are efficient

Benchmark Scenario	Python Throughput	Rust via PyO3 Throughput	Speedup / Overhead
List Indexing & Arithmetic (Read element, 2 additions)	6.67M ops/s	8.83M ops/s	~1.33x faster

Python to Rust conversion of large data is a major slowdown

Benchmark Scenario	Python Throughput	Rust via PyO3 Throughput	Speedup / Overhead
Passing Large Arguments	7.78M ops/s	824 ops/s	~9444x slower

In this benchmark, Python constructs a large payload once outside the timed loop: a list of integers, a string, and a bytes object. The Python version simply receives these objects and does nothing (just binds them to a local tuple), while the Rust version receives the same objects and converts them into native Rust types:

• Python list → Vec<i32> (copy + per-element conversion)
• Python string → &str (borrow)
• Python bytes → Vec<u8> (copy)

This benchmark is intentionally one-sided. It essentially measures the upper bound on the Python-to-Rust conversion cost for large containers, since the Python baseline does almost no work.

This significant overhead comes from the fact that Rust must allocate new buffers, copy all data, and validate element types across the FFI boundary.

When writing the driver, this gave us the insight that we should avoid passing large Python objects into Rust, and rather construct large value buffers entirely in Rust.

Returning large data

Benchmark Scenario	Python Throughput	Rust via PyO3 Throughput	Speedup / Overhead
Returning Large Data (List, String, & Bytes)	1,316 ops/s	1,253 ops/s	~0.95x (slight overhead)

In this benchmark, functions with no arguments create a large list, a long string, and a byte sequence, then return them. This measures the cost of creating large data in Rust and returning it across the boundary to Python (versus native Python doing the creation and returning by itself).

Returning large lists, strings, and byte payloads from Rust achieves nearly identical throughput to native Python, introducing minimal FFI overhead. This behavior perfectly fits database driver workloads, which typically process small inputs and return large outputs.

Overall, the results of these basic, synchronous benchmarks were incredibly encouraging. In almost all cases, the overhead was negligible. The one scenario where the overhead was significant can easily be avoided by keeping large data buffers entirely on the Rust side.

In addition to these basic synchronous benchmarks, we also needed to evaluate asynchronous performance.

PyO3 Async Runtime Benchmark

We benchmarked different approaches to integrating Rust async code using PyO3 with Python’s asyncio.

The benchmark compares three different implementations of the same async task:

1. Tokio with Async Runtimes – Rust implementation using pyo3_async_runtimes::tokio::future_into_py to bridge Tokio futures to Python’s asyncio,
2. Tokio with PyO3 – Rust implementation using a static Tokio runtime with manual spawning,
3. Asyncio Python – Pure Python implementation using asyncio.

Each implementation spawns multiple concurrent tasks that perform simple integer addition, simulating some workload (sufficient to prevent optimization).

The benchmark runs three distinct scenarios to test different usage patterns:

Scenario	Calls	List Size	Total Operations
Many small tasks	150,000	8	1,200,000
Medium tasks	2,500	1,000	2,500,000
Few large tasks	5	200,000	1,000,000

Scenario 1 – many small tasks

Implementation	Mean	Std Dev	vs Best
Asyncio Python	16.2918s	0.9967s	Baseline
Tokio with Async Runtimes	16.7730s	0.4385s	+2.96% slower
Tokio with PyO3	23.5543s	0.2557s	+44.60% slower

Scenario 2 – medium tasks

Implementation	Mean	Std Dev	vs Best
Tokio with Async Runtimes	0.4937s	0.0311s	Baseline
Tokio with PyO3	0.5631s	0.0204s	+14.10% slower
Asyncio Python	17.9199s	1.2237s	+3630% slower

Scenario 3 – few large tasks

Implementation	Mean	Std Dev	vs Best
Tokio with Async Runtimes	0.2275s	0.0069s	Baseline
Tokio with PyO3	0.2543s	0.0085s	+11.80% slower
Asyncio Python	6.6552s	0.5734s	+2926% slower

Conclusions

1. In Scenario 1, the overhead of spawning tasks in Rust dominates and all implementations perform similarly, mainly due to contention of GIL in multithreaded environments like the Tokio runtime.
2. When task count is moderate to low (Scenarios 2 & 3), Rust implementations are up to ~35x faster than Python. In the context of Python Rs Driver, this suggests it’s better to limit the number of calls if possible and submit larger tasks.
3. When we don’t cross the language gap frequently, Rust outperforms Python significantly.

Based on both the synchronous and asynchronous benchmark results, PyO3 was the clear choice for our driver. The key takeaways were: avoid passing large data buffers from Python to Rust and avoid spawning many small tasks.

To see how convenient it is to create a Python class from Rust using PyO3, look at this example of creating the Session class:

We then create a session module containing this class:

In many cases, using PyO3 greatly simplified our work, but it did not solve all our problems. Rust and Python are fundamentally different languages, making their integration quite complex. Here are the three main challenges we encountered:

1. References and lifetimes – one major issue was the inability to expose Rust references to Python. The Rust compiler checks reference lifetimes statically, while Python operates at runtime and manages objects completely differently. Therefore, every time we wanted to expose data based on a Rust reference to Python, we had to analyze the case individually and find a safe solution.
2. Mutability and thread safety – in Rust, only one mutable reference to a given structure can exist at a time. Python has no such restrictions. Since we wanted to support various Python versions, including Free-Threaded Python, we had to be very careful to ensure that the state shared between Python and Rust was properly protected.
3. GIL and Python object access – another challenge was the GIL. To create or access a Python object, the Rust code must acquire the GIL. Therefore, during implementation, we had to consciously decide when to acquire it, and – for longer operations on the Rust side – when to release it to avoid unnecessarily blocking Python code.

Choosing serialization approach

After selecting PyO3 as a binding layer, our first task was to decide on the serialization approach. We came up with three ideas on how it can be done:

1. Rust-side serialization – in this approach, the entire serialization process is performed on the Rust side. The main advantage here is that we can reuse serialization logic from the Rust Driver. On the other hand, Python objects are passed to Rust as Py<PyAny>, which is a smart pointer from the PyO3 library. Consequently, we do not know the type of the value being serialized in advance. An additional call to Python is therefore necessary to determine the object’s type for proper serialization. This introduces additional overhead due to the FFI calls.
2. Python-side serialization – in contrast to the previous approach, this method performs serialization entirely on the Python side and passes a bytearray of the serialized values to Rust. This reduces the number of FFI calls to just one. It doesn’t eliminate the problem of crossing the language boundary, because, as we described in PyO3 benchmarks, passing a large object from Python to Rust introduces a big overhead. Another problem with this solution is that the serialization logic can’t be reused; we have to implement it again in Python.
3. Cross-language serialization – this method combines the two previous approaches. The main idea is to expose structures used for serialization from the Rust Driver and use them in Python. The buffer with serialized values is kept on the Rust side. This is an obvious advantage over the Python-side approach because we avoid passing a big object from Python to Rust. Compared to the Rust-side approach, it enables implementing custom serialization, but at the cost of having some logic written on the Python side. However, we encountered the following problem: Python objects cannot hold Rust values with lifetimes. To work around this, each Python-exposed method with a lifetime acts as a wrapper/facade. Rust creates a fresh owned value during the call instead of exposing it.

To evaluate which methods worked best, we implemented each approach for a subset of data types and performed benchmarks. We divided them into four categories:

List benchmarks

In these benchmarks, the Rust-side implementation consistently outperforms the other two approaches. It achieves roughly 10 times better performance than the Rust-Python approach and yields a 20 to over 40 times improvement compared to the Python-side implementation.

String benchmarks

For small strings, the performance differences are similar to those observed for lists, with the Rust-side implementation significantly outperforming the alternatives. However, for medium and large strings, the Rust and Rust-Python approaches have comparable execution times. This behavior was initially caused by unnecessary copying during string serialization in the Rust-side approach. In the final implementation, this overhead was reduced by avoiding redundant allocations and copies.

Nested list benchmarks

Nested depth indicates the number of nested list dimensions present. Lists are constructed so that at each depth the list contains two copies of the previous level, creating a binary-tree-like structure. Depth 15 produces approximately 2¹⁵ leaf elements.

In all cases, the Rust implementation again achieves substantially lower execution times than both alternatives.

Mixed table benchmarks

A mixed table consists of repeated groups of heterogeneous data types, including integer, text, double, boolean, and list values. The width parameter indicates how many groups of such columns are included.

These results once again demonstrate that the Rust-side approach is the most performant option, while the Rust-Python implementation consistently outperforms the Python-side approach.

Based on these results, there was no doubt that we should select the Rust-side approach for our implementation. However, the comparison of the alternative methods remains valuable for future development. Since the Rust-side approach does not support custom serialization, an alternative solution will need to be selected to support this functionality later on.

Deserialization

After agreeing on the serialization approach, it was time to consider another essential component of the driver: deserialization. We decided that the most beneficial approach would be to perform deserialization on the Rust side. There were two main reasons for this:

1. Efficient Python object creation – The PyO3 benchmarks described earlier demonstrated that creating Python objects directly from Rust is efficient.
2. FFI calls vs logic in Python – The serialization benchmarks revealed that having logic on the Python side yields much worse performance than having it on the Rust side and crossing the languages boundary multiple times.

However, one question remained: should we deserialize directly into Python objects or should we first create intermediate Rust types and then convert them into Python objects?

For complex CQL types, the choice was straightforward. Constructing large structures in Rust first would be highly inefficient, therefore they are constructed directly as Python objects. The same approach was also considered for primitive CQL types, but ultimately rejected. Implementing this would require replicating the deserialization logic already present in the Rust Driver, which contradicted one of our main project goals: ensuring the new driver is nicely maintainable. In the final implementation, intermediate Rust values were used for primitive CQL types. The overhead introduced by this two-step conversion was expected to be minimal, since the intermediate Rust values were either stored on the stack or deserialized only to borrowed structures.

Yoke for zero-copy deserialization

When request results arrive from the database, they come as raw bytes that must be deserialized. To make this as fast as possible, the driver uses a zero-copy design. Instead of making unnecessary memory copies, the iterators borrow directly from the response buffer returned by the database.

Because these iterators borrow their underlying data, they have Rust lifetimes. This creates a challenge when exposing them to Python. PyO3 does not allow exposing Rust structs with non-static lifetimes to Python (and, in general, this is not possible to do safely).

We had to find a way to keep a zero-copy design from the Rust Driver while exposing the iterator over the result to Python. This is where the yoke crate came to the rescue.

What is yoke?

The primary goal of yoke is to safely erase non-static lifetimes by tying a data owner together with a data borrower into a single, self-contained unit. It uses the following model:

1. The Cart – the struct that owns the actual underlying data.
2. The Yokeable View – the borrower struct holding references that look inside the cart.

By managing the self-referential relationship between the cart and the yokeable view under the hood, yoke allows us to erase the explicit lifetime parameter from the outer view, transforming a temporary borrow into a static one that PyO3 can safely expose.

Using yoke in our driver

In our driver, the cart is a wrapper struct named QueryResultCart:

For yoke to work safely, the cart must implement the StableDeref trait. This is a safety guarantee promising that even if the cart wrapper moves, the memory address of the target data will never move.

We achieve this by wrapping QueryResult from the Rust Driver inside an Arc. The request result lives at a fixed location on the heap, while only the Arc pointer lives on the stack. When the QueryResultCart struct moves, only the stack pointer moves and the underlying QueryResult memory block stays in the same place.

The borrower view in our driver is the Cursor<'a> struct:

By deriving Yokeable, we tell yoke that this struct contains internal lifetime references. Cursor holds both the row and column iterators from the Rust Driver so they can borrow directly from the exact same response buffer without making any allocations.

Finally, we tie the cart and yokeable view together into struct exposed as a Python class:

The key detail here is that Cursor<'a> becomes Cursor<'static> inside the Yoke. This way, we eliminate the non-static lifetime from the structure definition. It can now be safely exposed to Python and used to iterate over the request result.

Current state and capabilities

Note: This represents the state as of June 2026.

The driver already supports most of the functionality required by client applications. It can handle both prepared and unprepared statements, batches, and serialization and deserialization of all CQL types.

Serialization

Both serialization and deserialization are performed on the Rust side. For serialization, this allows us to reuse the Rust driver’s existing logic. It was also the fastest approach in our benchmarks. Deserialization uses two strategies depending on the type being decoded. Native CQL values are first deserialized using the Rust driver’s existing implementations and then converted into Python objects through PyO3. This keeps the CQL decoding logic in one place, while the additional conversion is inexpensive for these small values. Complex types such as collections, tuples, vectors, and UDTs are instead constructed directly as Python objects, since creating an intermediate Rust structure only to convert it immediately afterwards would introduce unnecessary allocations and work.

Statement Execution API

Request execution and connection management is also powered by Rust, although the API is designed to mostly follow Python conventions. You connect to the database via a session builder that allows for method chaining rather than having one long constructor, which is an improvement compared to the legacy driver. The builder provides methods mostly corresponding to those available in Rust driver, sometimes adapted to be more Python-friendly. For example, prepared and unprepared statements are both executed through a single Session.execute() method; the Rust layer determines the statement type and calls the appropriate underlying Rust driver method.

Paging

The driver supports both automatic and manual paging, but we wanted the API to make the two modes difficult to accidentally mix. Query execution returns an immutable RequestResult representing the current page together with the state required to continue the query. Calling fetch_next_page() does not modify this object. Instead, it returns a new RequestResult containing the next page. This means that two iterators created from the same result cannot unexpectedly advance each other’s paging state, which was possible with mutable result objects used by the legacy Python driver.

Internally, both modes reuse the same paging machinery. A Pager is responsible for requesting subsequent pages, while a row iterator consumes and deserializes rows from the current response buffer. Manual paging exposes these operations separately: iter_current_page() only reads the page already in memory, while fetch_next_page() performs the next database request. For automatic paging, RequestResult implements Python’s asynchronous iteration protocol and combines the two operations: once the row iterator reaches the end of a page, the pager fetches another one and iteration continues transparently. Because PyO3 does not directly provide the asynchronous __anext__ abstraction needed here, the implementation bridges Rust futures to Python futures using pyo3-async-runtimes.

Error handling

Error handling turned out to be another place where simply exposing the Rust driver was not enough. Rust represents failures using strongly typed enums, while Python users expect an exception hierarchy that they can catch at different levels. Converting every Rust error to a string as soon as it occurs would be simple, but it would throw away information that may become useful later. We therefore keep errors structured while they travel through the Rust implementation and convert them into Python exceptions only when they cross the public API boundary.

The two hierarchies intentionally do not map one-to-one. A Python exception such as SessionConfigError may correspond internally to several Rust variants describing different failures. For example, an invalid item in the contact-points sequence can retain both the index of that item and the original Python exception that caused extraction to fail, while passing an entirely wrong object can instead retain its type name. This lets the Rust code keep precisely the information available at the place where the error occurred without requiring Python users to understand every internal variant.

The same mechanism is especially useful during serialization and deserialization. As an error propagates through nested values, the driver can gradually attach context such as the affected parameter, column name, column index, or the path inside a tuple, map, vector, or UDT. At the Python boundary, From<...> for PyErr conversions create the appropriate exception object. Additional information is exposed as exception attributes, and when the original failure came from Python code, it is attached as the exception’s cause. The result is still a familiar Python exception API, but without flattening the much richer diagnostic information available on the Rust side.

Metadata cache

Cluster metadata presented another cross-language problem. The Rust driver already maintains information about nodes, keyspaces, tables, columns, and replica placement, so copying the complete metadata tree into Python every time it is accessed would be both wasteful and difficult to keep consistent. Instead, ClusterState acts as a lightweight Python-facing view over the metadata snapshot maintained by Rust and keeps a cache of Python wrapper objects.

Not everything is cached in the same way. Node objects are created eagerly because they are accessed frequently, particularly by routing and custom load-balancing policies. Schema objects are created lazily: a Python Keyspace, Table, or related object is constructed only when the user actually requests it. Once created, these objects are stored through PyO3’s reference-counted Py pointers. That means subsequent accesses can return the same Python object with little more than a reference-count increment instead of rebuilding the metadata representation.

This also gives the API useful identity semantics. Asking for the same cached node or schema element repeatedly returns the same Python object, so ordinary Python is comparisons behave predictably. At the same time, the underlying Rust metadata is treated as an immutable snapshot. When the Rust driver observes a schema or topology change, the snapshot changes and the corresponding Python-side cache can be invalidated rather than trying to mutate a large tree of existing objects in place.

The cache is also used on a performance-sensitive path: replica lookup. ReplicaLocator can work directly with the reference-counted Rust metadata instead of copying it, and nodes returned by the Rust locator can be translated back into already cached Python Node objects. In the common case, exposing a replica to a Python load-balancing policy therefore requires only a cheap reference-count increment rather than allocating another wrapper. This matters because routing decisions may be performed for every query.

Benchmarks

The main goal of the design was to improve performance in comparison to the previous Python driver. Therefore, that legacy Python driver serves as a baseline in the conducted benchmarks.

One basic scenario was to insert simple key-value records using prepared statements. In each run the operation was performed for a different number of times (horizontal ax) – and every time it vastly surpassed the old driver with regard to execution time (vertical ax).

In case of SELECT operations, we ran simple benchmarks in which a result of 10 rows was retrieved from the database a different number of times for each run. The performance gain in comparison to the old driver is shown below.

The most notable gains, however, can be observed in concurrent execution workloads, where the driver benefits from realizing the GIL during long-running operations. That allows the Rust tokio runtime to schedule and execute requests concurrently in a more efficient manner.

The same benchmarks for insertion and selection were performed, each using 100 concurrent workers.

Those results show that delegating performance-critical operations to Rust is a great way to make the database communication more efficient for ScyllaDB users.

Future plans

At the time of writing, the core of the Python RS driver is functional, but the project has not yet reached production-ready status. Development is still taking place in a separate project repository while features are gradually reviewed and integrated upstream. Some important components – including TLS configuration, query metadata, retry policies, and load balancing support – have already been implemented as prototypes, but were still undergoing review as of September 2026. Further work will focus on integrating and stabilizing these features, extending test coverage, and filling the remaining gaps required for production use.

Another important goal is compatibility with the legacy Python driver API. The new driver intentionally introduces a redesigned, asynchronous API, which is easier to maintain and better matches modern Python. However, many existing ScyllaDB applications are written against the old cassandra-driver interface. A compatibility layer would allow such applications to migrate to the new Rust-backed implementation with significantly fewer code changes.

Providing this compatibility is more involved than simply preserving method names. The legacy driver relies heavily on its own ResponseFuture abstraction and callback-based execution model, while the new driver is built around Python’s asyncio ecosystem and await. Supporting the old API would therefore require adapting the new result and execution structures to reproduce legacy behavior, including callbacks and future-like objects, on top of the new asynchronous implementation. This adds complexity, but would provide a practical migration path for existing users who want the performance benefits of the new driver without immediately rewriting their applications.

Note: This post covers the progress made up to June 2026, which was when the ZPP Project at the University of Warsaw was completed. Since then, ScyllaDB has taken over development of the driver. Updates from this point forward will be noted in release notes on the ScyllaDB community forum.