Setting up Redis as a cache and message broker

Redis runs in memory and forgets data as soon as the process stops — that is why the configuration determines whether you can use Redis as a cache or as a message broker. This article explains how to set up Redis 7.2 for both roles, including memory policy, persistence, and isolation between functions.

Prerequisites

• Redis 7.2 or newer, installed on a Linux server (Debian 12 or Ubuntu 22.04 recommended)
• Root access or a user with `sudo` privileges
• Basic knowledge of the Redis CLI (`redis-cli`)
• For the message broker setup: an application that supports Redis as a broker, such as Celery 5.3 or a custom client

Steps

1. Check the Redis version and status

redis-server --version
redis-cli ping

Redis 7.2 includes all the features you need: `maxmemory`, eviction policies, RDB snapshots, AOF, and Redis Streams. `redis-cli ping` should return `PONG`; if it does not, start the service first with `sudo systemctl start redis-server`.

2. Configure Redis as a cache with a memory limit and LRU

Open the configuration file:

sudo nano /etc/redis/redis.conf

Set a maximum memory size and choose the eviction policy. For caching, use `allkeys-lru`, which removes the least recently used keys as soon as memory is full:

maxmemory 4096mb
maxmemory-policy allkeys-lru

`4096mb` is a realistic limit for production. For testing purposes, you can use `2mb`, but expect aggressive eviction then. The LRU policy makes TTL per key unnecessary: outdated data disappears automatically when memory fills up. TTL remains useful for session data or authentication tokens, but not for generic cache objects.

3. Enable RDB snapshots for cache recovery

Redis can periodically write a snapshot of all data to disk. This is sufficient for a cache: after a restart, the data is temporarily gone, but the backend database refills the cache.

save 60 1

This command writes a snapshot if there is at least one change within 60 seconds. The file `dump.rdb` appears in the Redis working directory. For cache use, this is sufficient; you do not need AOF, because loss of cache data does not cause lasting damage.

4. Restart Redis and test the cache configuration

sudo systemctl restart redis-server
redis-cli CONFIG GET maxmemory
redis-cli CONFIG GET maxmemory-policy

Both commands should show the configured values. Now test whether eviction works by writing more data than the limit allows:

redis-cli DEBUG POPULATE 1000000
redis-cli INFO memory | grep used_memory_human

If `used_memory_human` stays around the limit, Redis removes old keys correctly.

5. Configure Redis as a message broker with AOF persistence

For messaging, messages must not be lost. Redis Streams and lists provide the queue functionality, but without AOF, no message survives a restart. Open the configuration file again:

sudo nano /etc/redis/redis.conf

Replace the snapshot settings with AOF:

appendonly yes
appendfsync everysec

`appendfsync everysec` writes all write commands to the AOF file every second. This provides a good balance between performance and safety: you lose at most one second of messages in a crash. Disable the `save` rules, because RDB is not reliable enough for messaging.

6. Use Redis Streams for reliable messaging

Redis Streams is the native solution for pub/sub and task queues. A stream is a time-ordered log that supports multiple consumers, including acknowledgements and backpressure.

Create a stream and add a message:

redis-cli XADD task_queue '*' task send_email user 42

Read messages as a consumer:

redis-cli XREADGROUP GROUP workers consumer1 COUNT 10 STREAMS task_queue >

This command reads new messages that have not yet been processed by another consumer. Combine this with AOF: if Redis restarts, both the stream and the consumer groups remain intact.

7. Use separate Redis instances for cache, broker, and result store

A single Redis instance for both cache and messaging causes interference: LRU eviction can remove active messages, and a full cache slows down the queue. Therefore, use separate instances.

Create a second Redis service:

sudo cp /etc/redis/redis.conf /etc/redis/redis-broker.conf
sudo nano /etc/redis/redis-broker.conf

Change at least these settings:

port 6380
maxmemory 0
maxmemory-policy noeviction
appendonly yes
pidfile /var/run/redis/redis-server-broker.pid
logfile /var/log/redis/redis-server-broker.log

`maxmemory 0` means unlimited memory, and `noeviction` ensures that Redis does not remove messages when memory fills up — it then refuses new write operations instead of removing existing ones. Start the second instance:

sudo redis-server /etc/redis/redis-broker.conf --daemonize yes

Repeat this pattern for a third instance for the result store, for example on port 6381.

8. Point applications to the correct instance

Configure your application with separate URLs for each function. OpenForms uses these variables:

CACHE_REDIS_URL=redis://localhost:6379/0
CELERY_BROKER_URL=redis://localhost:6380/0
CELERY_RESULT_BACKEND=redis://localhost:6381/0

The cache instance on port 6379 has LRU and a maximum of 4 GB. The broker on port 6380 has AOF and no eviction. The result store on port 6381 stores task results with AOF — that data must never be lost.

Verification

Check that all three instances are running and have their own configuration:

redis-cli -p 6379 INFO memory | grep maxmemory
redis-cli -p 6380 CONFIG GET appendonly
redis-cli -p 6381 CONFIG GET maxmemory-policy

The first must show `maxmemory: 4294967296` (4 GB), the second `appendonly: yes`, and the third `maxmemory-policy: noeviction`.

Test the broker by sending and reading a message:

redis-cli -p 6380 XADD test_stream '*' message hello
redis-cli -p 6380 XREAD COUNT 1 STREAMS test_stream 0

The second command must return the message you just added. Then restart the broker instance and read again: the message must still exist thanks to AOF.

Frequently asked problems

Redis does not start after changing `maxmemory-policy`

Check that the configuration file does not contain duplicate lines. Redis uses the last value, but a syntax error in another line can make the entire service refuse to start. Test with `redis-server /etc/redis/redis.conf --test-memory 1` and check the log: `sudo journalctl -u redis-server -n 50`.

Messages disappear after a restart despite AOF

Check that `appendonly` is set to `yes` in the active configuration: `redis-cli CONFIG GET appendonly`. If this returns `no`, Redis used a different configuration file. Also check the AOF file: `ls -lh /var/lib/redis/appendonly.aof`. An empty file means Redis did not activate AOF during the restart.

LRU removes active messages in the queue

This happens when cache and broker share the same instance. Set `maxmemory-policy` to `noeviction` for the broker instance, or better: use separate instances as described in step 7. `noeviction` ensures that Redis refuses to write new data instead of removing existing data.

The choice between RDB and AOF is not a performance issue, but a reliability issue — a cache that loses data is an annoyance, a broker that loses messages is an outage. By separating the roles across separate instances you keep both systems predictable, and that is exactly why Redis holds up in cloud-native architectures as both cache and message broker.