Building a Queue with Redis: Lists, Blocking Pops, and Streams

This article walks through implementing a queue on Redis 7.x, from a plain FIFO list to a reliable pattern that survives worker crashes, and shows where Redis Streams fit in when lists stop being enough.

Requirements

• Redis 7.2 or newer (7.4 recommended; Streams require 5.0+, consumer groups with `XAUTOCLAIM` require 6.2+)
• A client library for your language: `redis-py` 5.x (Python), `ioredis` 5.x (Node.js), or `redis-rb` 5.x (Ruby)
• Shell access to the Redis server for verification, or `redis-cli` installed locally
• Basic familiarity with `LPUSH`, `RPOP`, and how a worker process is started

If you are on Redis 6.x, everything in the list-based sections works. The Streams section needs 6.2+ for `XAUTOCLAIM`, which is what makes failed-message recovery practical.

Step 1 — Start Redis and confirm the version

You need a running instance before anything else. On a package install, the service is usually already up.

```bash redis-server --version redis-cli ping ```

The first command prints the version. The second should return `PONG`. If it returns `Could not connect`, the server is not running — start it with `sudo systemctl start redis-server` on Debian/Ubuntu, or `brew services start redis` on macOS.

For anything beyond a test, bind Redis to a private interface and set a password. An open Redis on a public IP is scanned and compromised within hours. That is not a hypothetical.

Step 2 — Push a task onto a FIFO list

The producer side is one command. `LPUSH` adds to the head of the list; combined with `RPOP` on the consumer side, you get First In, First Out.

```bash redis-cli LPUSH myqueue "send_email:user123" redis-cli LPUSH myqueue "send_email:user124" redis-cli LLEN myqueue ```

`LLEN` returns `2`. The order matters: `LPUSH` puts each new item at the left, `RPOP` removes from the right, so `user123` comes out first. If you swap to `RPUSH` + `LPOP`, you get the same FIFO behavior with the ends mirrored. Pick one convention and stay with it — mixing them in the same codebase is how you end up with a queue that runs backwards.

A list has no schema. Whatever string you push is what the worker receives. Serialize structured payloads as JSON yourself; Redis will not do it for you.

Step 3 — Pop tasks on the consumer side

The naive consumer polls in a loop.

```bash redis-cli RPOP myqueue ```

This returns `send_email:user123` and removes it from the list. When the list is empty, `RPOP` returns `nil` immediately. A worker that calls `RPOP` in a tight loop with no delay will hammer the CPU and the network for nothing.

You can add a sleep, but that adds latency on every task and still wastes cycles when the queue is idle. The next step fixes both.

Step 4 — Block instead of poll with BRPOP

`BRPOP` blocks the connection until an item is available or the timeout expires. This is what you want in production.

```bash redis-cli BRPOP myqueue 0 ```

The `0` means block forever. A non-zero value is the timeout in seconds — use `BRPOP myqueue 5` if you want the worker to wake up periodically for health checks or shutdown signals.

The trade-off: a blocked connection occupies a Redis client slot. With hundreds of workers, you need to raise `maxclients` and watch memory per connection. For most workloads this is fine. For thousands of idle workers, Streams with consumer groups scale better because the blocking happens at the group level, not per list.

`BLPOP` is the mirror command for popping from the left. Use whichever pairs with your push side.

Step 5 — Make the queue reliable with a processing list

Here is the problem with everything above: `RPOP` removes the item before the worker finishes the job. If the worker crashes mid-task, that task is gone. No error, no retry, no trace.

The reliable queue pattern moves the item to a second list instead of deleting it.

```bash

Worker atomically moves the item to a processing list

redis-cli BRPOPLPUSH myqueue myqueue:processing 0 ```

`BRPOPLPUSH` blocks on `myqueue`, pops the rightmost element, and pushes it to `myqueue:processing` in one atomic operation. The worker now holds the task and the task is still in Redis.

When the job succeeds, remove it from the processing list:

```bash redis-cli LREM myqueue:processing 1 "send_email:user123" ```

If the worker dies, the item stays in `myqueue:processing`. A separate watchdog process scans that list for entries older than a threshold and moves them back:

```bash redis-cli RPOPLPUSH myqueue:processing myqueue ```

`BRPOPLPUSH` is deprecated in Redis 6.2 and later. The replacement is `BLMOVE`, which takes an explicit direction for both ends:

```bash redis-cli BLMOVE myqueue myqueue:processing RIGHT LEFT 0 ```

Use `BLMOVE` on 6.2+. It does the same thing with clearer semantics, and it is what the Redis documentation now points to.

The cost of this pattern is a second list to monitor and a watchdog to write. If your tasks are idempotent and losing one occasionally is acceptable, skip it. If losing one means a customer never gets their password reset email, do not.

Step 6 — Move to Streams when you need history and consumer groups

Lists give you a queue. Streams give you a log with acknowledgment, multiple consumers, and replay.

```bash redis-cli XADD mystream '*' task "send_email" user "user123" ```

The `*` tells Redis to generate the message ID from the current timestamp. The reply is the ID, something like `1718000000000-0`.

Create a consumer group:

```bash redis-cli XGROUP CREATE mystream workers 0 ```

The `0` means the group starts reading from the beginning of the stream. Use `$` to start from new messages only.

Consumers read with `XREADGROUP`:

```bash redis-cli XREADGROUP GROUP workers worker-1 COUNT 10 BLOCK 5000 STREAMS mystream '>' ```

The `>` means "messages never delivered to any consumer in this group." Redis tracks which messages each consumer has pending. After processing, acknowledge:

```bash redis-cli XACK mystream workers 1718000000000-0 ```

Unacknowledged messages stay in the group's Pending Entries List (PEL). If `worker-1` dies, another consumer claims its messages:

```bash redis-cli XAUTOCLAIM mystream workers worker-2 60000 0-0 COUNT 10 ```

The `60000` is the minimum idle time in milliseconds before a message can be claimed. Set it longer than your slowest task, or you will have two workers processing the same job.

Streams are heavier than lists. Every message stays in the stream until you trim it with `XADD ... MAXLEN ~ 10000` or `XTRIM`. Without trimming, memory grows without bound. Lists have no such problem because the item is gone once popped.

Step 7 — Turn on persistence

Redis is in-memory. A restart without persistence means an empty queue.

Enable AOF in `redis.conf`:

``` appendonly yes appendfsync everysec ```

`everysec` flushes to disk once per second. You can lose up to one second of writes on a crash. `always` is slower but loses nothing. `no` lets the OS decide, which is fast and unpredictable.

RDB snapshots alone are not enough for a queue. They run on an interval, and everything since the last snapshot is gone on restart. AOF logs every write, which is what you want when the queue holds real work.

Restart Redis after editing the config:

```bash sudo systemctl restart redis-server ```

Even with AOF, a single Redis instance is a single point of failure. For queues that matter, run a replica with Sentinel for automatic failover, or use Redis Cluster. Both add operational cost. Decide whether your queue justifies it before you need it.

Verification

Push a task and confirm it round-trips through the reliable pattern.

```bash redis-cli LPUSH myqueue "test:task" redis-cli BLMOVE myqueue myqueue:processing RIGHT LEFT 0 redis-cli LRANGE myqueue:processing 0 -1 ```

The third command should show `test:task`. Now simulate success:

```bash redis-cli LREM myqueue:processing 1 "test:task" redis-cli LLEN myqueue:processing ```

`LLEN` returns `0`. The task moved through the queue, landed in processing, and was cleared.

For Streams, check the pending list:

```bash redis-cli XPENDING mystream workers ```

The summary line shows how many messages are pending, the smallest and largest ID, and the consumers involved. If nothing is pending after a batch, every message was acknowledged.

Common problems

`BRPOP` returns `nil` immediately instead of blocking. You passed a timeout of `0` to a client that treats it as "no wait." Check your client library's documentation — some require `timeout=None` or a negative value for infinite blocking. The Redis protocol itself treats `0` as block forever, but wrappers do not always pass it through.

Tasks disappear after a worker restart. You are using plain `RPOP` without a processing list. Switch to `BLMOVE` as in Step 5. The task is removed from the queue the moment it is popped, so a crash between pop and completion loses it permanently.

Stream memory grows without limit. You never trim. Add `MAXLEN ~ 100000` to your `XADD` calls, or run `XTRIM mystream MAXLEN ~ 100000` on a schedule. The `~` makes trimming approximate and cheaper; Redis removes whole nodes instead of individual entries. Exact trimming with `=` is slower and rarely worth it.

The queue you build on Redis is only as reliable as the pattern you choose. A list with `RPOP` is three lines of code and loses work. A list with `BLMOVE` and a watchdog is thirty lines and mostly does not. Streams with consumer groups are a hundred lines and give you replay, at the cost of memory you have to manage yourself. Pick the one that matches what happens when a task vanishes — because on a long enough timeline, one will.