Async Systems Decision Guide — Sync vs. Queue vs. Webhook vs. Batch

This guide explains how to choose between synchronous and asynchronous data processing patterns and how to implement whichever one you choose correctly, with retries, error handling, and audit trails that government programs require.

Engineers building or reviewing systems that integrate with external services, process data in the background, or need to handle spikes in volume without blocking users or failing silently will find it most useful.

TL;DR

Synchronous: The user waits for the result. Use this when the operation completes in under 2–3 seconds and the result is needed to continue.

Background job: The operation takes longer or can be retried. The user gets a confirmation immediately; work happens separately.

Message queue / event bus: Multiple services need to react to the same event, or you need guaranteed delivery with backpressure handling.

Webhook: An external system notifies your application when something changes. You receive it, verify it, and act on it.

Batch: Large volumes of data, non-real-time, often from a legacy partner system that only supports file exchange.

Every async pattern needs: a retry policy, a dead-letter queue, idempotency, and logging that serves as an audit trail.

The Core Question

Every time an operation needs to happen in your system, ask one question:

Does the user need to see the result of this operation before they can continue?

If yes, the operation must be synchronous — the user waits, you process, you return a result.

If no, the operation can be asynchronous — you accept the request, confirm receipt, and do the work separately.

A concrete government example: a citizen submits a benefits application online.

The user needs to see a confirmation number immediately. That part is synchronous.
The eligibility check against three state databases takes 8–12 seconds and requires a human caseworker to review the output. That part is asynchronous.
A nightly report of all applications received that day goes to a federal reporting system. That part is a batch job.

Designing the whole application as synchronous means the user stares at a spinner for 12 seconds and risks a timeout. Designing it correctly means the user gets a confirmation in under a second, and the slow work happens without any impact on the user experience.

The Five Patterns

Synchronous API Call (Request–Response)

Your application calls an external service and waits for the response before doing anything else. The user is blocked until the response arrives.

When to Use

The user needs the result to continue (login check, real-time validation, price lookup).
The operation reliably completes in under 2–3 seconds under normal load.
Only one service needs the result.

Problems

Tight coupling. If the external service is slow, your application is slow. If it is unavailable, your application cannot serve that function at all.
No automatic retry. If the request fails, the user must retry manually — or you must add retry logic at the call site.
Cascading failures. A downstream service that takes 30 seconds to respond ties up your web server threads, which can cascade into your entire application becoming unresponsive.

Mitigations

Set a short, explicit timeout (3–5 seconds) on every outbound HTTP call.
Implement a circuit breaker — after N consecutive failures, stop calling the service and return a cached result or an error immediately.
Show the user a meaningful error if the service is unavailable, rather than a timeout.

Common tools include fetch (browser and Node.js), axios, Go’s net/http, and Python’s httpx.

Background Job and Task Queue

Your application accepts a request, writes it to a job queue, and returns a confirmation to the user immediately. A separate worker process picks up the job and does the work asynchronously.

When to Use

The operation takes more than a few seconds.
The user does not need the result immediately (they will be notified when it is done).
The operation might fail and needs to be retried automatically.
The operation should not be lost if the worker crashes mid-way.

How it works

User submits a request.
Your application creates a job record in the queue (database table, Redis list, or managed queue service).
Your application returns HTTP 202 Accepted with a job ID or confirmation number.
A worker process picks up the job, executes it, and updates the job status.
Your application notifies the user (email, webhook, polling endpoint) when the job completes or fails.

Key requirements

Retries with exponential backoff. If a job fails, retry it automatically after a delay. Each retry waits longer than the last (1s, 2s, 4s, 8s…). This prevents hammering a struggling downstream service.
Maximum retry count. After a fixed number of retries, stop and move the job to a dead-letter queue for investigation.
Idempotency. If a worker crashes after partially completing a job and the job is retried, running it again must produce the same result as running it once. See the Idempotency section below.
Job status visibility. The user and your support team should be able to check the status of a job: pending, in progress, completed, failed.

Common tools include BullMQ (Node.js), Sidekiq (Ruby), Celery (Python), AWS SQS + Lambda (managed), and Faktory (language-agnostic).

Message Queue and Event Bus

Multiple services need to react to the same event, or you need guaranteed delivery across service boundaries with backpressure handling.

When to Use

More than one downstream service needs to react to an event (document uploaded → antivirus scan + OCR + audit log).
Producers generate events faster than consumers can process them and you need the queue to absorb the difference (backpressure).
Consumers may be offline temporarily and messages must not be lost.
You want loose coupling — the producer does not need to know which services are consuming its events.

How it differs from a background job queue

A background job queue is typically owned and managed by one application. A message bus connects multiple applications. An event on the bus can trigger actions in multiple independent services simultaneously.

Government use cases

A document processing pipeline: citizen uploads a document, an event fires, three independent services respond — one scans for malware, one runs OCR to extract text, one logs the upload to the audit stream.
An inter-agency data sync: a state agency’s eligibility system publishes a change event when a citizen’s status changes; a benefits portal consumes the event and updates the citizen’s record.
An audit event stream: every write operation in a sensitive system publishes to an append-only audit log via a message bus.

Key requirements

At-least-once delivery vs. exactly-once delivery. Most queue systems guarantee at-least-once delivery — a message may be delivered more than once if a consumer crashes before acknowledging it. Design consumers to be idempotent.
Consumer group management. Configure which services consume which topics and track their position in the queue (offset management in Kafka, visibility timeout in SQS).
Dead-letter queues. Messages that fail to process after N attempts must be moved to a dead-letter queue and trigger an alert.

Common tools include AWS SQS, RabbitMQ, Azure Service Bus, Apache Kafka for high-throughput streaming use cases, and Google Cloud Pub/Sub.

Webhook

An external service notifies your application by making an HTTP POST request to a URL you provide, whenever a relevant event occurs.

When to Use

An external system needs to push state changes to your application.
You do not want to poll the external system repeatedly.
Examples: payment completed, document signed, identity verified, background check completed.

How it works

You register a URL with the external service (e.g., https://your-app.gov/webhooks/stripe).
When an event occurs (payment completes), the vendor sends an HTTP POST to your URL with event data in the body.
Your application receives the request, verifies it, processes it, and returns HTTP 200.
If your application does not return 200, most vendors retry the delivery.

Security requirements

Every inbound webhook must be verified before acting on it. An unverified webhook is a vector for attackers to inject fake events (e.g., a fake “payment completed” event to grant access without paying).

Verification methods vary by vendor:

HMAC signature: The vendor signs the request body with a shared secret. Your application computes the expected signature and compares it. If they match, the request is legitimate.
Shared secret token: A secret value in the request header that only the vendor and you know.
IP allowlisting: Only accept requests from the vendor’s published IP ranges (use this in addition to signature verification, not instead of it).

Always reject requests that fail signature verification immediately, before any processing.

Idempotency requirement: vendors retry webhook delivery if your endpoint does not return 200 quickly. Your handler may receive the same event multiple times. Process each event idempotently and use the event’s unique ID to detect and skip duplicates.

Common implementation: any HTTP server that can receive POST requests. Use the vendor’s official library for signature verification where one exists.

Batch File Transfer

Data is exchanged as files on a schedule — typically daily, weekly, or monthly. Common formats include CSV, JSON lines, XML, and fixed-width text. Common transfer mechanisms include SFTP, S3, and Azure Blob Storage.

When to Use

Data volumes are large and real-time processing is not required.
The partner system only supports file-based exchange (common with legacy state and federal systems).
The data represents a complete snapshot rather than incremental events (e.g., a monthly extract of all active Medicaid enrollees).
The receiving system needs to process all records before any downstream work begins.

Government context: many federal and state agency systems were built in the 1980s and 1990s. They do not have REST APIs. They exchange data through SFTP drops on a nightly schedule. This is not going away soon. Build integrations that work with the partner as they are, not as you wish they were.

Key requirements

Validate on receipt. Check file format, row count, expected columns, and data types before processing. A malformed file should fail loudly, immediately, with an alert.
Idempotent processing. If the same file is processed twice (retried after a failure), the result should be the same as processing it once.
Alerting on missing files. If a scheduled file does not arrive, your system must detect the absence and alert. A missing file that goes unnoticed is a data gap.
Audit trail. Log when each file was received, how many records it contained, and the outcome of processing. This log is your evidence that the data transfer happened.

Common tools include AWS S3 + Lambda (triggered on file arrival), Azure Data Factory, cron plus an SFTP client, and AWS Glue for large-scale ETL.

Decision Reference

Use this table to match your situation to the right pattern.

Situation	Recommended Pattern	Reason
User needs result to continue (login, validation)	Synchronous API call	Immediate result required
Operation takes > 3 seconds	Background job	Avoid blocking the user
Same event needs multiple consumers	Message queue / event bus	Loose coupling, guaranteed delivery to all consumers
External vendor pushes events to you	Webhook	Vendor-initiated; you receive and act
Large data volume, non-real-time	Batch file transfer	High volume, schedule-friendly
Legacy partner only supports files	Batch file transfer	Meet the partner where they are
Operation may need retry	Background job or queue	Built-in retry and dead-letter queue

Figure 1 (placeholder) — A decision tree diagram for choosing a data integration pattern. Start with ‘Does the user need the result immediately?’ → Yes → ‘Will it complete in under 3 seconds?’ → Yes → Synchronous API / No → Background Job with progress feedback. Back at the first node: No → ‘Do multiple services need to react?’ → Yes → Event Bus/Message Queue / No → ‘Is the partner pushing data to you?’ → Yes → Webhook / No → ‘Is this bulk, non-real-time data?’ → Yes → Batch / No → Background Job.

Idempotency

Idempotency means that running the same operation multiple times produces the same result as running it once.

Why this matters for async systems: async systems retry failed operations. If your processing logic is not idempotent, retries cause duplicates such as a payment charged twice, a benefit credited twice, or a notification sent three times.

How to implement idempotency

Assign a unique ID to every operation. When a user submits a form, generate a unique submission ID. Include it in every downstream message and job.
Check for duplicates before processing. Before performing a write operation, check whether a record with that ID already exists. If it does, skip the write and return success.
Use database constraints. A unique constraint on the idempotency key field in your database provides a hard guarantee — the second write will fail at the database level rather than silently duplicating data.
Use transactional outbox pattern for critical operations. Write the job record to the database in the same transaction as the primary record. This prevents the case where the primary write succeeds but the job is never queued.

Example: a worker receives a job to send an eligibility determination email. Before sending, it checks whether a record exists in the email_sent_log table with the same application_id and email_type. If the record exists, the job was already processed, so it skips and acknowledges. If not, it sends the email and inserts the log record.

Retries and Dead-Letter Queues

Every async operation must have an explicit retry policy. Failing silently — where a job fails and nothing happens — is not acceptable in a government system.

Retry policy components

Maximum attempts. How many times will you retry before giving up? Three to five is a common default.
Backoff strategy. How long do you wait between retries? Exponential backoff (1s, 2s, 4s, 8s) reduces load on a struggling downstream service.
Jitter. Add a small random delay to each retry to prevent a thundering herd of retries from all hitting the downstream service at the same moment.

Dead-letter queues: when a job exhausts all retries, move it to a dead-letter queue (DLQ). A DLQ is a separate queue or database table that holds failed jobs for investigation. Configure an alert to fire whenever a message lands in the DLQ. That is your signal that something needs human attention.

Every job that lands in the DLQ must be investigated. Common causes:

The downstream service changed its API contract.
The message is malformed (bug in the producer).
A required resource (database, external API) was unavailable for an extended period.
The job payload contains data the worker cannot process (unexpected null, wrong data type).

Backpressure

Backpressure is what happens when a queue fills up faster than workers can drain it.

Government example: a document processing system handles 500 uploads per day during normal operation. Open enrollment starts, and submissions jump to 5,000 per day for two weeks. Workers cannot keep up. The queue grows. Processing time for each document increases from minutes to hours. Users receive eligibility determinations late.

Backpressure handling strategies

Scale workers horizontally. Add more worker processes during high-volume periods. This is easier with managed queue services (SQS + Lambda autoscales automatically) than with self-managed worker pools.
Set queue depth alarms. Alert when the queue depth exceeds a threshold (e.g., more than 1,000 messages unprocessed). This is an early warning before user impact is felt.
Prioritize. If some jobs are more time-sensitive than others, use separate queues with different worker allocations. A time-sensitive eligibility determination goes to a high-priority queue; a nightly audit export goes to a low-priority queue.
Rate-limit producers. If you control both the producer and the consumer, slow the producer when the queue is deep. This prevents unbounded queue growth.
Communicate with users. If processing is delayed, tell users. A benefits applicant who knows their submission is in a queue and will be processed within 48 hours is less anxious than one who submitted and heard nothing.

Tool Comparison

Pattern	Common Tools	Managed Options (FedRAMP-eligible)
Synchronous API	`fetch`, `axios`, Go `net/http`, Python `httpx`	N/A — runs in your application
Background job	BullMQ (Node.js), Sidekiq (Ruby), Celery (Python), Resque	AWS SQS + Lambda, GCP Cloud Tasks
Message queue	RabbitMQ, Apache Kafka, AWS SQS, Azure Service Bus	AWS SQS (FedRAMP High), Azure Service Bus (FedRAMP High)
Webhook receiver	Any HTTP server	N/A — runs in your application
Batch / ETL	cron + SFTP, AWS Glue, Azure Data Factory	AWS Glue (FedRAMP Moderate), AWS Step Functions

When evaluating tools for a FedRAMP-authorized environment, verify the specific service’s authorization level at marketplace.fedramp.gov. Not all services within an authorized provider are themselves authorized.

Worked Example: Async Document Upload and Eligibility Check

A government benefits portal allows citizens to upload supporting documents (proof of income, residency documents) as part of an application. Eligibility is checked after each document is received.

Here is the full async pipeline, step by step.

Step 1: Receive the upload (synchronous)

The user selects a file and submits it. Your application:

Validates the file type and size (synchronous, immediate).
Generates a unique document_id.
Writes a record to the documents table with status = 'received'.
Returns HTTP 200 with the document_id and a message: “Your document has been received. Processing typically takes 2–5 minutes.”

The user is not waiting for eligibility. They have a confirmation and can move on.

Step 2: Store the file (background, immediate)

A background job triggered by the upload:

Moves the file from the temporary upload area to permanent object storage (S3 or equivalent).
Generates a content hash (SHA-256) of the file and stores it. This is tamper evidence.
Updates the document record: status = 'stored', storage_path = '...'.

Step 3: Queue the eligibility check

After storage succeeds, publish a message to the eligibility check queue:

{
  "job_id": "elig-check-8f4a2c",
  "document_id": "doc-1234",
  "application_id": "app-5678",
  "document_type": "proof_of_income",
  "submitted_at": "2026-05-28T14:23:11Z"
}

Step 4: Worker processes the eligibility check

A worker picks up the message:

Idempotency check: Has job_id = "elig-check-8f4a2c" been processed before? If yes, acknowledge and exit.
Retrieve the document from storage.
Call the state eligibility API (synchronous, with a 5-second timeout and circuit breaker).
On success: update the application record with the eligibility result. Update document status to 'processed'. Send a notification email to the applicant.
Acknowledge the queue message.

Step 5: Handle failure

If the state eligibility API is unavailable:

The worker does not acknowledge the message.
The queue returns the message to the worker pool after the visibility timeout expires.
The job is retried with exponential backoff (attempt 1: 30s, attempt 2: 60s, attempt 3: 120s).
After 3 failures, the message moves to the dead-letter queue.
An alert fires to the on-call engineer: “Eligibility check DLQ: 1 message. Job ID: elig-check-8f4a2c.”

Step 6: Manual recovery from dead-letter queue

The on-call engineer:

Checks the DLQ alert. Reviews the job payload and the error logs.
If the external API is now available: re-queues the job to the main queue. It processes on the next worker pickup.
If the external API is still down: flags the application for manual caseworker review. Updates the applicant: “Your document is being reviewed. We will contact you within 3 business days.”

Figure 2 (placeholder) — An async document processing pipeline for a government benefits application. Show: User uploads document (sync HTTP response: ‘Upload received’) → Document lands in S3 → SQS message queued → Worker picks up message → Runs eligibility check (calls external API) → On success: updates application record, sends notification email. Show the retry path: on worker failure → retry up to 3 times with exponential backoff → on max retries → moves to dead-letter queue → alert fires for manual review.

Government-Specific Considerations

Async Job Logs as Audit Evidence

Every job that runs in an async system generates log entries. These logs tell you what happened, when, and what the result was. In a government system, they are also evidence of process — they show that a document was received, processed, and the result was recorded. Retain job logs according to your agency’s record retention schedule. Do not discard them as mere operational noise.

PII in Message Queues

A message queue that carries document payloads or applicant information is as sensitive as your production database. Apply the same controls:

Encrypt messages at rest (most managed queue services support this).
Encrypt messages in transit (TLS).
Restrict access to the queue to only the services that need it.
Do not log message bodies to general-purpose log aggregators unless PII is redacted first.

Government Partners Requiring Batch File Transfer

Federal and state legacy systems often cannot receive API calls or publish webhooks. They exchange data through SFTP file drops on a nightly or weekly schedule. Do not try to modernize the partner’s side of the integration. Meet them where they are, document the integration in your inventory, and build appropriate monitoring around the file transfer.

Communicating Async Processing Latency

Citizens using a government portal expect to know what is happening. If an eligibility determination takes 10 minutes because it goes through an async pipeline, tell the user: “Your submission is being processed. You will receive an email confirmation within 15 minutes.” An application that accepts work silently and provides no feedback erodes trust.

Testing Async Systems

You cannot write a simple end-to-end test that submits a form and checks the result in the same HTTP response when the result comes from a background job. Use integration tests that listen to the queue, process a test job, and assert on the database state. Use test-mode configurations that bypass real external APIs.

Common Mistakes

Async When Sync Would Suffice

Not every operation needs a queue. A form field validation that takes 50 milliseconds does not need a background job. Reach for async patterns when the operation genuinely needs them, not because they feel more modern.

Missing Idempotency

This is the most common cause of duplicate records, duplicate emails, and double-charged payments in async systems. Every job must check before it acts.

Missing Dead-Letter Queue

Without a DLQ, jobs that fail after max retries are silently dropped. This means data loss and invisible failures. Always configure a DLQ and always alert on it.

Overpowered Infrastructure for Simple Jobs

Kafka and RabbitMQ have operational overhead. If a single application needs to run slow jobs in the background and only one consumer exists, a simple job queue (BullMQ, Sidekiq) is the right tool. Reserve the event bus for multi-consumer fan-out scenarios.

Incorrect Message Visibility Timeouts

In SQS and similar systems, a message becomes invisible to other workers while one worker is processing it. If the worker takes longer than the visibility timeout, the message becomes visible again and is processed by a second worker simultaneously. Set visibility timeouts to at least 2x your expected maximum job duration.

Unmonitored Queue Depth

A queue that is slowly filling up is a silent early warning of a problem — a downstream service is slowing down, a worker crashed, or volume is higher than expected. Monitor queue depth and alert on it before the queue grows so large that users are impacted.

Next Steps

Integration Inventory — Document every system your async jobs integrate with, including their SLAs and fallback behavior.
CI/CD from Zero — Automate the deployment of your worker processes alongside your web application.

External References

AWS SQS — Dead-Letter Queues
BullMQ Documentation
OWASP — Web Services Security Cheat Sheet
Stripe — Webhook Best Practices
Cloud Events Specification — a standard event envelope format for interoperability between systems