TLDR; We decided to move on from Redis for queue and in-memory cache management and migrate it to the Cube Store. That means we’ll depreciate Redis for cache and queue management later this year and fully remove its support in new versions of Cube in 2023.
Cube’s mission is to use headless BI to power your data applications with consistent, secure, and performant data. One of the ways we’ve been guaranteeing data performance has been with Cube’s queue and in-memory caching layers. Until now, these components were based on Redis, an open source, in-memory data store.
However, as Cube has been scaling and seeing significant expansion in its use cases, we’ve realized using Redis doesn’t make sense anymore. Instead, we’ve come to need an internal, more specialized solution to avoid unnecessary dependencies and execute operations smoothly. And so, we’re beginning to sunset its support soon—and are excited to migrate queue management and in-memory cache to Cube Store.
How Cube’s in-memory cache and queue management work
Let’s start with some context. Cube has a two-level caching system: pre-aggregations and in-memory cache. There are also Cube’s queues, which sit in front of databases to provide a concurrency burst safety net in case there are too many simultaneous requests. Cube’s in-memory cache and queues allow you to scale stateless API instances horizontally and make APIs idempotent; they are currently powered by Redis.
Cube’s queues act as a buffer for your database when there's a burst of requests hitting the same data from multiple concurrent users. Its job is to ensure that the data sources are not overloaded and that multi-stage queries are executed and refreshed in the correct order. Cube’s in-memory cache solves a pretty similar problem, offloading query workloads from databases that shouldn’t be refreshed yet by
refreshKey rules. The in-memory cache also offloads cluster coordination for query execution costs as its overhead is noticeable in deployments with high concurrency.
Cube currently maintains in-memory cache, query execution queues for pre-aggregations, and data queries in Redis. The architecture built with Cube and Redis enables the queues to be idempotent—meaning that if multiple identical queries come in, only one will run against the data source.
Why is Redis not a fit anymore?
Redis requires too many complex instructions for simple operations like atomically pushing to a queue, executing mutually exclusive queries, and idempotent result waiting on client nodes. It also yields an excess of optimistic concurrency clashes. As a result, Redis cluster coordination consumes a lot of resouces under the high load, which creates a bottleneck to scaling the API concurrency.
Finally, even with a lot of community effort, we couldn’t provide stable support for clustered Redis. All of these issues made it clear to our users and us that the current queue management and in-memory caching systems need simplification.
How will Cube Store replace Redis?
The Cube Store implementation will closely mimick Redis’ functionality. However, we’re going to replace lengthy command batches with single atomic domain-specific instructions. This change will drastically reduce clashes. Additionally, it’ll significantly simplify cache and queue access flow, which should drive a significant throughput boost.
Cube Store will also employ similar distributed LRU caching semantics as Redis. Unlike Redis, however, it’ll store data in binary format utilizing columnar compression strategies in cases where it makes sense to do so.
Most importantly to you—in-memory cache and queue management in Cube Store will use the same SQL protocol as pre-aggregation management, which is already occurring in Cube Store. So, you’ll be able to use Cube Store for in-memory cache and queue management without any reconfiguration or additional setup.
We’re planning to release initial support for queue management in Cube Store and issue a deprecation notice for using Redis with Cube in November 2022. We’ll fully remove Redis support in Cube around Q2 2023.
Goodbye, Redis—and hello, Cube Store
And so, we’re gradually saying goodbye to Redis support.
It’s been a great run, but bigger and better performance is coming for Cube users: in-memory cache and queue management with Cube Store is on the horizon.
Need help migration to Cube Store-based queue management? Contact our team via our contact page.