Celery and Redis for async tasks in Python

sudo apt update && sudo apt install redis-server

Start the Redis service and check that it is active:

sudo systemctl start redis-server
sudo systemctl enable redis-server

Redis listens on TCP port 6379 by default. Check this with:

redis-cli ping

The output should be `PONG`.

2. Install Celery with Redis support

Celery needs a separate package for Redis integration. Install `celery[redis]` so that all dependencies for Redis support are included:

pip install "celery[redis]"

Without this extra package, Celery cannot connect to Redis, even if the Redis server is running.

3. Configure the Celery application

Create a Celery application that uses Redis as the message broker and result backend. Create a file `celery_app.py`:

from celery import Celery</p><p>app = Celery(
    'tasks',
    broker='redis://localhost:6379/10',
    backend='redis://localhost:6379/11'
)</p><p>app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    timezone='Europe/Amsterdam',
    enable_utc=True,
)

The broker and backend point to Redis database 10 and 11 respectively. This is not a mandatory configuration but a common way to isolate the data streams. You can also use database 0 with `redis://localhost:6379/0`.

4. Define an asynchronous task

Celery tasks are ordinary Python functions with the `@app.task` decorator. Create `tasks.py`:

from celery_app import app</p><p>@app.task
def send_email(recipient: str, subject: str, body: str) -> dict:
    # Simulate a long-running operation
    import time
    time.sleep(5)
    
    # Here you would actually send an email
    return {
        'recipient': recipient,
        'subject': subject,
        'status': 'sent'
    }</p><p>@app.task
def process_upload(file_path: str) -> dict:
    # Process an uploaded file
    with open(file_path, 'rb') as f:
        data = f.read()
    
    # Do something with the data
    result = {'bytes_processed': len(data)}
    
    return result

The decorator ensures that Celery recognizes the function as an asynchronous task that can be executed by a worker.

5. Start the Celery worker

The worker runs in a separate process, independent of the main application. Start the worker with:

celery -A tasks worker --loglevel=info

The worker listens for messages in the Redis queue and processes tasks as they arrive. This is essential for asynchronous operation: the web server keeps responding to user requests, while the worker handles heavy tasks in the background.

6. Send tasks from your application

Call tasks with `.delay()` or `.apply_async()`. Create `main.py`:

from tasks import send_email, process_upload
<h2>Simple call with .delay()</h2>
result = send_email.delay(
    'user@example.com',
    'Welcome',
    'Thank you for registering'
)
<h2>Advanced call with .apply_async()</h2>
result = process_upload.apply_async(
    args=['/tmp/file.pdf'],
    countdown=10,  # start after 10 seconds
    retry=True,
    retry_policy={
        'max_retries': 3,
        'interval_start': 0,
        'interval_step': 0.2,
        'interval_max': 0.2,
    }
)

`.delay()` is a shorthand for `.apply_async()` and only accepts positional arguments. `.apply_async()` offers more control such as delays, retries, callbacks, and queue selection.

7. Retrieve task results

Use the result object to query the status and result of a task. Redis acts here as the result backend where results are temporarily stored:

from tasks import send_email</p><p>result = send_email.delay('user@example.com', 'Test', 'Body')
<h2>Check if the task is done</h2>
print(f"Task done: {result.ready()}")
<h2>Wait for the result (blocks until the task is completed)</h2>
result_data = result.get(timeout=10)
print(f"Result: {result_data}")
<h2>Get the task ID</h2>
task_id = result.id
print(f"Task ID: {task_id}")

This functionality is useful when a user needs to be informed about the progress of a long-running operation.

8. Schedule periodic tasks

Celery can also run periodic jobs via `celery beat`. Add this to `celery_app.py`:

from celery.schedules import crontab</p><p>app.conf.beat_schedule = {
    'send-daily-reports': {
        'task': 'tasks.send_email',
        'schedule': crontab(hour=8, minute=0),
        'args': ('admin@example.com', 'Daily report', 'Here is the report'),
    },
}

Start the beat scheduler along with the worker:

celery -A tasks beat --loglevel=info

Verification

Check whether the setup works by sending a test task and retrieving the result. Run this in a Python session:

from tasks import send_email</p><p>result = send_email.delay('test@example.com', 'Test', 'Body')
print(result.ready())  # False if the task is still running
print(result.get(timeout=10))  # Wait for the result

The output should show the result defined in the task, for example:

{'recipient': 'test@example.com', 'subject': 'Test', 'status': 'sent'}

Also check the Redis queues:

redis-cli -n 10 llen celery

This command shows the number of tasks in the queue. After processing, this should be 0.

Common problems

1. Celery cannot connect to Redis

Check whether Redis is running and reachable on port 6379:

redis-cli ping

If this returns `PONG`, check whether you installed `celery[redis]` instead of just `celery`. Without the extra package, Celery cannot set up a Redis connection.

2. Tasks are not executed

The worker must be active. Start the worker with:

celery -A tasks worker --loglevel=info

Check the log output for error messages. If the worker is running but tasks are not picked up, check whether the broker URL in `celery_app.py` matches the Redis database you are using.

3. Results are not available

If `result.get()` gives a `TimeoutError`, check whether you configured the backend correctly. The backend URL must point to a Redis database that differs from the broker database. For example, use database 10 for the broker and database 11 for the backend:

broker='redis://localhost:6379/10',
backend='redis://localhost:6379/11'

Celery itself is not a message queue but a framework that depends on an external broker such as Redis. Redis acts as the transport layer between the application (producer) and the worker (consumer). This distinction determines the architecture of your asynchronous processing. The Celery documentation for version 5.6.2 was updated on August 24, 2026, and the Redis-specific page in version 5.6.3 on July 27, 2026 — the project is actively maintained and the integration with Redis continues to evolve, so it pays to keep following the official documentation for new features and optimizations.