I think it's really interesting to know that it's possible to run Django on a single server, no Kubernetes required, while still being able to run services like Celery.

What is Celery?

Celery is an asynchronous task runner, meaning it allows you to run tasks outside of the normal request-response cycle, which has a time limit on how long it should take. For instance, let's says we're running a widget factory, but we need to reticulate our splines, but that takes longer than one request-response cycle can handle. Let's offload that to celery.

The Details

Let's say we have a project called widget_factory with an app called reticulator. Let's add a celery task:

reticulator/tasks.py

import time

from celery import shared_task


@shared_task
def reticulate_splines():
    time.sleep(5)  # let's pretend that this could be longer
    return "reticulated!"

And we follow along with [First steps with Django](https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html) to get our project setup.

widget_factory/celery.py

import os

from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'widget_factory.settings')

app = Celery('widget_factory')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django apps.
app.autodiscover_tasks()

widget_factory/__init__.py

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

And we have django-celery-results installed:

pip install django-celery-results

widget_factory/settings.py

INSTALLED_APPS = (
    ...,
    'django_celery_results',
)

CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'

Don't forget to migrate to make sure we have the backend tables in the database:

./manage.py migrate

Celery SQLite3 Database Broker

Here's where it gets interesting. We'd like to use the database as our broker. We could probably use Redis or RabbitMQ still as our broker since the requirement is that we are running on a single server, and this would still be meeting that requirement, but let's make it interesting and use our default SQLite3 database.

To do that, let's follow the instructions from the old [Using SQLAlchemy](https://docs.celeryq.dev/en/3.1/getting-started/brokers/sqlalchemy.html) page from the docs.

widget_factory/settings.py

CELERY_BROKER_URL = 'sqla+sqlite:///celerydb.sqlite'

And get sqlalchemy installed

pip install sqlalchemy

Now we can run celery:

celery -A widget_factory worker -l INFO

Let's open up a separate terminal now and try reticulating splines

./manage.py shell

...

>>> from reticulator import tasks
>>> result = tasks.reticulate_splines.delay()
>>> result.ready()
False
>>> result.ready()
True
>>> result.get()
'reticulated!'
>>>

Yay! To summarize, we can run a task and save the results all without the need for Redis or RabbitMQ as a broker.

Stretch Goals

• We are using a separate SQLite3 file (celerydb.sqlite) for brokering tasks. If we really wanted to, we could fold that into our main db.sqlite3 database file.

• It would be great to be able to use Django's ORM instead of using SQLAlchemy. Celery used to have support for the Django ORM, but it got removed because it did not have a maintainer. It would be fun to resurrect that code.

• We could figure out how to run celery and Django using supervisord. We can't get around the fact that we need at least two processes, one each Django and Celery.

• We could setup a cookiecutter for all of this. I think it would be really cool to have one for a minimal setup on a VPS.

Drop me any questions, comments, or concerns or if you are interested in my implementing any of the stretch goals or already did so and want to share. Maybe we can build out a cookiecutter together. Thanks for reading!