Skip to content

Watching it

@app.on_job_failure
def alert(job, error, will_retry):
    if not will_retry:
        sentry_sdk.capture_exception(error)


@app.on_job_success
def timing(job, result, duration):
    statsd.timing(job.task, duration)

on_job_start(job) completes the set. Hooks run in the worker and can be def or async def. One that raises is logged and ignored, so a broken metrics client never fails a job. A def hook runs on the worker's event loop, so keep it quick.

tinyjobs stats --app tasks:app
tinyjobs check --app tasks:app --max-wait 5m      # exits 1 on any problem
QUEUE    QUEUED  HELD  WAITING  RUNNING  SUCCESS  FAILED  CANCELLED  OLDEST DUE
-------  ------  ----  -------  -------  -------  ------  ---------  ----------
default  2       0     0        2        0        0       0          16s
mail     1       0     0        0        0        0       0          16s

running past their lease: 2

A job past its lease means the worker holding it stopped reporting, usually because it died. check fails on that, on an app that registers no tasks, on queued or held jobs for a task the app doesn't register (a missing --import), on a due job, queued or held, left longer than --max-wait, and on a cron schedule more than a minute overdue, which means no worker or scheduler is running. A job waiting on another is not due, so it never counts as a backlog. Both are app.stats() and app.check() in code.

Every log record about a job carries the job as fields, so a log pipeline can filter on them without parsing the message. --log-format json writes one object per line, on every command (wrapped here); a tinyjobs logger your settings configured, as under manage.py tinyjobs, or a root logger that already has handlers in a standalone run, is left alone, with a warning when the flag was given:

tinyjobs worker --app tasks:app --log-format json
{"time": "2026-11-05T09:12:44.118Z", "level": "INFO", "logger": "tinyjobs.worker",
 "message": "job 01a1ab2e-... succeeded on attempt 1", "job_id": "01a1ab2e-...",
 "task": "tasks.generate_pdf", "queue": "default", "attempt": 1,
 "worker_id": "laptop/8821/9389541d", "duration_ms": 12.4}

A failure adds error, error_message and will_retry, with the traceback in traceback on the last attempt, which the text format prints under the line; a record about a schedule carries schedule. The fields are extra on the standard logging record, so your own handlers see them too, and the last attempt also carries the exception as exc_info, for Sentry and plain formatters. tinyjobs.logs.JSONFormatter and TextFormatter are what the flag installs, for a logging config of your own.

Prometheus can scrape the worker:

tinyjobs worker --app tasks:app --metrics-port 9464        # --metrics-host 0.0.0.0 to expose it
curl -s localhost:9464/metrics
tinyjobs_jobs_total{task="tasks.generate_pdf",outcome="retry"} 3
tinyjobs_jobs_total{task="tasks.generate_pdf",outcome="success"} 1842
tinyjobs_job_duration_seconds_bucket{task="tasks.generate_pdf",le="0.5"} 1790
tinyjobs_jobs_running{task="tasks.generate_pdf"} 2
tinyjobs_store_jobs{queue="default",status="queued"} 14
tinyjobs_store_queued_jobs{task="tasks.generate_pdf"} 14
tinyjobs_store_oldest_due_seconds{queue="default"} 0.8
tinyjobs_store_expired_leases 0
tinyjobs_info{version="0.18.0"} 1

The counters, the duration histogram and the running gauge are this process's, fed by the job events behind the hooks above. The tinyjobs_store_* gauges are the store's, read with one stats query per scrape that gets five seconds to answer, so a slow store costs the scrape its gauges and not the target; they are the same from every worker, so scrape them from one, or take max() across instances rather than sum(). An embedded worker has no port of its own: return app.metrics.render(await app.astats()) from a route, with tinyjobs.metrics.CONTENT_TYPE as the content type, and give the stats query the budget and the fallback to app.metrics.render() that your route gives any store query.