Celery is a simple, flexible and reliable asynchronous task queue/job queue. Task queue is a mechanism used to distribute work across threads or machines. Celery communicates via messages and is focused on real-time processing, while supporting task scheduling at the same time. Tasks are executed either asynchronously (in the background) or synchronously. Celery supports concurrent task execution, prefork (multiprocessing), Eventlet, and gevent.
Celery is easy to set and maintain, you can get started without any configuration files or need to learn the full complexities of the problem Celery solves. It is flexible, though it provides all the tools and support you need to run such a system in production, it can be easily extended, or integrated with other languages. This highly available and fast system was designed around best practices and allows to process huge amounts of messages.
Although Celery is written in Python, the protocol can be implemented in any language and can be easily integrated with a vast number of web frameworks, e.g. Django, Pyramid, Flask, Pylons, web3py, Tornado, etc. Celery is actively used in production systems such as Instagram, Mozilla Add-ons, Adroll, etc. and helps them to process millions of tasks every day.
Since Celery is built on distributed message passing it usually uses a separate service called a message broker to send and receive messages between clients and workers. It is recommended to use RabbitMQ or Redis broker because they are feature complete. There are other experimental solutions for developers to try: MongoDB, ZeroMQ, CouchDB, SQLAlchemy, Django ORM, Amazon SQS.
Main Celery features:
- Scheduling. You can specify the time when to run a task in seconds/datetime, or provide simple interval or crontab expressions for periodic tasks.
- Monitoring. Stream of monitoring events is used to provide real-time information about the cluster.
- High availability and horizontal scaling. Celery system has multiple broker and multiple workers support, it can run on a single machine, on multiple machines, or even across data centers.
- Autoreloading and Resource Leak Protection.
- Workflows. To optimize your work you can create simple and complex workflows using a set of powerful primitives, including grouping, chaining, chunking, etc.
- Time & Rate Limits. Number of tasks executed per second/minute/hour, as well as task processing duration can be controlled and/or be set as a default.
- Autoscaling. Celery dynamically resizes the worker pool depending on load or custom metrics, that is applied to limit memory usage or to enforce a certain service quality.
- User Components. Celery enables fine-grained and tuned control of the worker. User can customize and redefine any worker component.
Asynchronous processing is a method that allows to separate the execution of certain tasks from the main flow of a program. By implementing a job queue into your program's environment, you can offload tasks, increase responsiveness and handle interactions from your users more quickly. Celery is one of the fastest and most reliable tools available. To know more visit celeryproject.org.