Skip to content
T.
Back to writings
Distributed Systems

When One Worker Is Not Enough

Learn how PostgreSQL coordinates multiple task workers with atomic claims, leases, retries, and safe state changes.

8 min read

One worker is sufficient for many small systems. It reads a task, runs the task, and then reads the next task. The design is easy to understand. It is also easy to operate.

The limits become clear when the task count increases. One slow task can delay all other tasks. A stopped worker can stop all work. More powerful hardware can help, but it does not remove the single point of failure.

At this point, you can add more workers. However, more workers introduce a new problem: they must share work safely.

This article explains the design of a distributed task scheduler. The example uses Python workers and PostgreSQL. It supports scheduled tasks, priorities, retries, cancellation, and recovery after a worker failure.

The first problem is ownership

Assume that two workers read the same pending task. Both workers see that the task is available. Both workers start it. The system now runs the task two times.

This duplicate execution can have serious results. The task can send two emails, charge a customer two times, or write the same data more than one time.

A worker must claim a task before it runs the task. The claim must be atomic. This means that the database completes the full claim as one operation.

PostgreSQL can select and lock one task in a transaction:

WITH claimable_task AS (
    SELECT id
    FROM tasks
    WHERE scheduled_time <= now()
      AND (
        status = 'PENDING'
        OR (
          status = 'RUNNING'
          AND (lease_until IS NULL OR lease_until <= now())
        )
      )
    ORDER BY priority DESC, scheduled_time ASC, created_at ASC
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
UPDATE tasks
SET
    status = 'RUNNING',
    claimed_by = %(worker_id)s,
    lease_until = now() + (%(lease_seconds)s * interval '1 second'),
    updated_at = now()
WHERE id = (SELECT id FROM claimable_task)
RETURNING *;

FOR UPDATE locks the selected task. SKIP LOCKED lets a second worker skip that task. The second worker can select a different task without a wait.

The query also defines a clear order. A task with high priority comes first. For tasks with the same priority, the oldest scheduled task comes first.

PostgreSQL is the shared source of state

Each worker has its own memory. A worker cannot use its memory to know which tasks other workers own. The workers need one shared source of state.

Store each task in PostgreSQL:

CREATE TABLE tasks (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    task_type TEXT NOT NULL,
    payload JSONB NOT NULL DEFAULT '{}'::jsonb,
    priority INTEGER NOT NULL,
    scheduled_time TIMESTAMPTZ NOT NULL,
    status TEXT NOT NULL,
    retry_count INTEGER NOT NULL DEFAULT 0,
    max_retries INTEGER NOT NULL DEFAULT 3,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL,
    claimed_by TEXT,
    lease_until TIMESTAMPTZ,
    error_message TEXT
);

The task state must have a small set of permitted values:

  • PENDING: The task waits for a worker.
  • RUNNING: A worker owns the task.
  • COMPLETED: The task finished successfully.
  • FAILED: The task cannot run again.
  • CANCELLED: A user stopped the task.

The claimed_by value identifies the owner. The lease_until value specifies how long the ownership is valid.

Use TIMESTAMPTZ for all times. This type gives consistent comparisons when workers operate in different time zones.

A claim must have a time limit

A permanent claim is not safe. A worker can stop after it claims a task. The task can then stay in the RUNNING state permanently.

Use a lease to give ownership for a limited time. For example, a worker can own a task for 30 seconds. The worker extends the lease while the task runs.

UPDATE tasks
SET
    lease_until = now() + (%(lease_seconds)s * interval '1 second'),
    updated_at = now()
WHERE id = %(task_id)s
  AND status = 'RUNNING'
  AND claimed_by = %(worker_id)s;

The worker ID in the WHERE clause is important. Only the current owner can extend the lease.

Renew the lease at an interval that is shorter than the lease duration. For a 30-second lease, a 10-second renewal interval is a good initial value.

If a worker stops, its lease expires. Another worker can then claim the task. The system recovers without a manual database change.

The worker loop must stay small

A worker does not need complex coordination logic. PostgreSQL does most of the coordination.

Use this worker cycle:

  1. Claim one due task.
  2. Commit the database transaction.
  3. Start lease renewal.
  4. Run the task handler.
  5. Stop lease renewal.
  6. Save the final state.

Do not run the task handler inside the claim transaction. A handler can take a long time. A long transaction keeps locks and database resources for too long.

When no task is available, wait for a short time before the next query. This wait reduces database load. Start with a wait of 0.5 seconds. Increase it if quick task pickup is not necessary.

All state changes must check ownership

A worker can become an old owner while it still runs. For example, its network connection can fail. The lease can expire, and a new worker can claim the task. The first worker can then reconnect.

The first worker must not change the task after ownership changes. Completion, failure, and retry updates must check the worker ID:

UPDATE tasks
SET
    status = 'COMPLETED',
    claimed_by = NULL,
    lease_until = NULL,
    updated_at = now()
WHERE id = %(task_id)s
  AND status = 'RUNNING'
  AND claimed_by = %(worker_id)s;

If the update changes zero rows, the worker is not the current owner. It must discard its result.

This rule protects the task state. It cannot undo an external operation that the old worker already completed.

Temporary failures need persistent retries

Some task failures are temporary. A service can be unavailable. A connection can reach its time limit. The task must run again after a delay.

Keep the retry count in PostgreSQL. The count must remain available after a worker stops.

Use exponential backoff for the retry delay:

retry_delay = min(
    retry_base_delay * (2**task.retry_count),
    retry_max_delay,
)

After a failure, increase retry_count. If retries remain, set the state to PENDING and set scheduled_time to a future time. If the task reaches max_retries, set the state to FAILED.

This design uses the normal task table for retries. A separate retry queue is not necessary.

For a large system, add a small random value to the delay. This change prevents many failed tasks from starting again at the same time.

Duplicate execution is still possible

Atomic claims prevent two workers from owning a task at the same time. They do not give exactly-once execution.

A worker can complete an external operation and stop before it updates PostgreSQL. After the lease expires, another worker runs the task again.

The scheduler therefore gives at-least-once execution. Task handlers must be idempotent. An idempotent handler produces the same result when it runs more than one time.

Use the task ID as an idempotency key for an external service. You can also store operation IDs in a table with a unique constraint.

This rule is necessary for payments, messages, file creation, and other external effects.

Cancellation needs cooperation

A pending task is easy to cancel. Change its state from PENDING to CANCELLED.

A running task is different. A database update cannot stop Python code that already runs. The handler must check for cancellation at safe points if you need a quick stop.

Without this check, cancellation only prevents the worker from saving a successful final state. It does not undo work that the handler already did.

Document this limit in the scheduler interface.

Observe the complete system

Multiple workers are difficult to inspect without good operations data. Use structured logs and include these values:

  • Task ID
  • Worker ID
  • Task type
  • Retry count
  • Retry delay
  • Event name

Record metrics for task claims, completions, failures, retries, and lease renewals. Also record scheduler loop errors.

A process health check is not sufficient. Check that the worker can connect to PostgreSQL. Check the number of old pending tasks. Check the number of tasks with expired leases.

These checks show problems that a process status cannot show.

Test worker coordination

Use a real PostgreSQL database for coordination tests. A mock database cannot fully reproduce row locks and concurrent transactions.

Test these conditions:

  • Many workers claim tasks at the same time.
  • One task has only one current owner.
  • A task becomes available after its lease expires.
  • An old owner cannot save a final state.
  • Retry delays increase to the configured limit.
  • A cancelled task does not become pending again.
  • High-priority tasks run before low-priority tasks.

Also run a load test with many tasks and workers. Look for duplicate claims, tasks that stay in one state, and long database waits.

Add workers without losing control

One worker gives a simple system. Multiple workers give more capacity and better availability. The additional workers also require strict ownership rules.

Keep the durable task state in PostgreSQL. Claim tasks with row locks. Give each claim a renewable lease. Check ownership during every state change. Store retries in the database. Make task handlers idempotent.

With these rules, workers can share tasks safely. The scheduler can also recover when a worker stops. This design gives a practical base that you can extend as the workload increases.

Back to writings