harshaBackground workers make a request path simpler by moving long-running work elsewhere. They also force a question into the open: what happens when the same logical operation is attempted again?
Celery's task documentation discusses retries, acknowledgment choices, and the importance of idempotent tasks. Those mechanisms need to be matched to the application rather than treated as a universal safety switch. The task guide explains the relevant worker behavior.
Identify the operation
Consider an illustrative ingestion job that extracts a document, creates chunks, and stores embeddings. If the worker retries after the embeddings were written but before completion was recorded, a second execution could create duplicates.
A queue message ID identifies a delivery. A business key identifies the intended operation. Depending on the product, that key might include tenant, document version, and processing configuration. It should distinguish “retry the same work” from “process a genuinely new version.”
The database can then enforce the intended uniqueness instead of asking every worker to remember it perfectly.
Split the side effects
External calls complicate idempotency. A transaction inside one database cannot roll back a request already accepted by another service.
I would make the stages explicit: acquire or inspect the operation record, perform a bounded unit of work, record the result, then advance. Each stage needs a recovery story for a worker disappearing immediately before or after its side effect.
This does not eliminate every race. It creates places where races can be reasoned about and tested.
A task that performs five unrelated effects and returns one success flag leaves too much ambiguity when it stops halfway through.
Retry with purpose
Retries should target transient failures. Repeating a malformed request may add load without making progress. Repeating an unauthorized operation is worse. An operator should be able to see the failure classification and the attempt history.
Backoff, retry limits, and deadlines serve different purposes. A task can stay within its retry limit and still take longer than the workflow can tolerate.
The useful question is not “does the worker retry?” It is “can another attempt finish this operation without corrupting its meaning?” Once that answer is clear, the queue's delivery settings become implementation choices in a coherent design.