Stack guide

Background jobs and recovery actions: Sidekiq and friends

Every background job system shares one property that makes payment work dangerous: it retries. Sidekiq, Celery, BullMQ, Temporal — all assume tasks are safe to repeat, because most tasks are. Reversal jobs are not. The worker contract below converts repetition-hostile operations into safe ones through four steps any framework can express, with Sidekiq as the worked example.

The four-step worker contract

Read live state → derive key from intent → act → persist response. Reading live state guards the 24-hour key-expiry tail. Intent-scoped keys collapse legitimate retries. Persisting Stripe’s original response answers every later "did this happen?" forensically instead of archaeologically.

class RecoveryWorker
  include Sidekiq::Job

  def perform(finding_id)
    finding = Finding.find(finding_id)
    transfer = Stripe::Transfer.retrieve(finding.transfer_id)
    return if (transfer.amount_reversed || 0) >= finding.expected_reversal

    resp = Stripe::Transfer.create_reversal(
      transfer.id,
      { amount: finding.amount_owed },
      { idempotency_key: "clawback-#{finding.id}" },
    )
    finding.update!(stripe_response: resp.to_json, state: :resolved)
  end
end

Dead jobs and orphaned findings

Jobs exhaust retries eventually; findings must outlive their workers’ optimism. States like clawback-failed carrying verbatim errors keep queues truthful, and deadset monitoring pages humans before sellers discover silent failures.

Backpressure during mass events

Failed campaigns and cancellations flood queues; unthrottled execution trips rate limits mid-correction. Concurrency caps per account plus global throughput limits convert floods into orderly batches — the difference between unwinding gracefully and compounding the incident.

Framework portability

StepFunctions, Temporal, Oban — identical contract, different syntax. The contract is the portable asset; implementations are interchangeable details.

Common questions

Does this require a FeeGuard integration?

No — the page stands alone as stack guidance. FeeGuard observes your event stream externally rather than embedding in it.

Should reversal jobs be idempotent at business level too?

Yes — keys handle transport retries; live-state reads handle logic retries; persisted responses make both auditable. Layers, not substitutes.