Task Queues & Scheduling

kombu

The universal translator that lets Python talk to a dozen different messaging systems in one consistent language.

Install it: pip install kombu

What does it do?

Messaging systems like Redis, RabbitMQ, and Amazon SQS all let programs pass messages to each other, but each one speaks its own dialect. Kombu abstracts over all of them, so code written against Kombu’s interface can switch from RabbitMQ to Redis without a rewrite, the way a universal remote works across different TV brands. It’s less something people use directly and more the plumbing other tools, most notably Celery, are built on top of.

See it in action

This sends a small message announcing a job (resizing an image) into a message queue that another program can later pick up and act on.

from kombu import Connection, Exchange, Queue

exchange = Exchange("tasks", type="direct")
queue = Queue("task_queue", exchange, routing_key="tasks")

with Connection("redis://localhost:6379/") as conn:
    with conn.Producer() as producer:
        producer.publish(
            {"job": "resize_image"},
            exchange=exchange,
            routing_key="tasks",
            declare=[queue],
        )

Why would a non-developer care?

Most people will never touch Kombu directly, but if they’ve ever used a Python application with background tasks, Kombu was very likely the layer quietly moving messages between the parts of that system. Its existence is why Celery can support so many different messaging backends without duplicating a mountain of code for each one.

Real-world examples

Kombu was created by Ask Solem, the same person behind Celery, specifically to give Celery a flexible, swappable messaging foundation. Any Celery-based application, which includes a huge swath of Python’s web ecosystem, is running Kombu underneath whether the developers realize it or not.

Who uses it

Mostly used indirectly by developers running Celery; occasionally used directly by engineers who need broker-agnostic messaging in a custom system.

How it compares to alternatives

It doesn’t really have direct competitors in the same sense — it’s a compatibility layer rather than a queue itself, though lower-level libraries like pika and redis-py are what it wraps for specific brokers.

Fun fact

Kombu is the Japanese word for edible kelp, used to make dashi stock — a fitting name for a library that’s meant to be the flavorless base other things, like Celery, are built on top of.

New to Python and want to actually try libraries like this yourself?

Find a beginner-friendly course

Related libraries