Workers, Cron Jobs and Private Services

Background workers

Long tasks (emails, image processing, exports) should run outside web requests. A worker uses the same code with a different start command:

Start command: rq worker --url $REDIS_URL default

or for Celery:

celery -A app.tasks worker --loglevel=info

Workers do not receive HTTP traffic. Share environment variables with the web service through an environment group.

Cron jobs

A Cron Job service runs a command on a schedule (cron syntax, in UTC) and exits:

Schedule: 0 3 * * *
Command:  python scripts/nightly_cleanup.py

Use them for cleanups, reports, backups and reminders. Make jobs idempotent and log clearly; add a heartbeat ping to a monitor so you know when a job did not run.

Private services

Internal APIs or tools that only your other services call can run as private services with no public URL, reachable over the private network by their internal hostname.

Scaling the web service

• Vertical: move to a larger instance type and raise workers/threads with environment variables (WEB_CONCURRENCY, GUNICORN_THREADS), so no code changes are needed.
• Horizontal: add instances (manually or with autoscaling on supported plans). Requires stateless apps: sessions in signed cookies or Redis, uploads in object storage, rate limits in Redis.

A rough sizing guide for a Flask app:

Measure memory per worker in your own metrics and adjust.

Exercise

Add an RQ worker that sends welcome emails queued by the web service, and a nightly cron job that deletes expired tokens. Share configuration through an environment group.

Check yourself

• How does a worker differ from a web service?
• What timezone do cron schedules use?
• What must be true before you add a second web instance?