I think it's really interesting to know that it's possible to run Django on a single server, no Kubernetes required, while still being able to run services like Celery.
What is Celery?
Celery is an asynchronous task runner, meaning it allows you to run tasks outside of the normal request-response cycle, which has a time limit on how long it should take. For instance, let's says we're running a widget factory, but we need to reticulate our splines, but that takes longer than one request-response cycle can handle. Let's offload that to celery.
The Details
Let's say we have a project called widget_factory
with an app called reticulator
. Let's add a celery task:
reticulator/tasks.py
import time
from celery import shared_task
@shared_task
def reticulate_splines():
time.sleep(5) # let's pretend that this could be longer
return "reticulated!"
And we follow along with [First steps with Django](https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html) to get our project setup.
widget_factory/celery.py
import os
from celery import Celery
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'widget_factory.settings')
app = Celery('widget_factory')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django apps.
app.autodiscover_tasks()
widget_factory/__init__.py
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ('celery_app',)
And we have django-celery-results
installed:
pip install django-celery-results
widget_factory/settings.py
INSTALLED_APPS = (
...,
'django_celery_results',
)
CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'
Don't forget to migrate to make sure we have the backend tables in the database:
./manage.py migrate
Celery SQLite3 Database Broker
Here's where it gets interesting. We'd like to use the database as our broker. We could probably use Redis or RabbitMQ still as our broker since the requirement is that we are running on a single server, and this would still be meeting that requirement, but let's make it interesting and use our default SQLite3 database.
To do that, let's follow the instructions from the old [Using SQLAlchemy](https://docs.celeryq.dev/en/3.1/getting-started/brokers/sqlalchemy.html) page from the docs.
widget_factory/settings.py
CELERY_BROKER_URL = 'sqla+sqlite:///celerydb.sqlite'
And get sqlalchemy
installed
pip install sqlalchemy
Now we can run celery:
celery -A widget_factory worker -l INFO
Let's open up a separate terminal now and try reticulating splines
./manage.py shell
...
>>> from reticulator import tasks
>>> result = tasks.reticulate_splines.delay()
>>> result.ready()
False
>>> result.ready()
True
>>> result.get()
'reticulated!'
>>>
Yay! To summarize, we can run a task and save the results all without the need for Redis or RabbitMQ as a broker.
Stretch Goals
We are using a separate SQLite3 file (
celerydb.sqlite
) for brokering tasks. If we really wanted to, we could fold that into our maindb.sqlite3
database file.It would be great to be able to use Django's ORM instead of using SQLAlchemy. Celery used to have support for the Django ORM, but it got removed because it did not have a maintainer. It would be fun to resurrect that code.
We could figure out how to run celery and Django using
supervisord
. We can't get around the fact that we need at least two processes, one each Django and Celery.We could setup a
cookiecutter
for all of this. I think it would be really cool to have one for a minimal setup on a VPS.
Drop me any questions, comments, or concerns or if you are interested in my implementing any of the stretch goals or already did so and want to share. Maybe we can build out a cookiecutter
together. Thanks for reading!