FRD: Jobs/Queue Provider

Document Summary

Field	Details
Feature Name	Jobs/Queue Provider
Status	Draft
Owner	David Holmes
Contributors	Engineering
Target Release	v2.0.0
Related Links	ADR-051 (Provider Pattern), ADR-065 (Webhook Handling), ADR-027 (Default Tech Stack), ADR-014 (Open Source First)
Last Updated	2026-06-02
Open Source Libraries	bullmq, ioredis
Documentation	BullMQ Docs · Queues · Workers · ioredis Docs

This FRD ships @dmwd-io/jobs as an optional package that re-exports bullmq and ioredis as the platform standard for background job queues. Per ADR-014 (Open Source First), the value is standards enforcement and drift prevention — not a custom abstraction on top of the library. Platform conventions (env var names, queue naming) are documented alongside the re-export.

---

Introduction

Overview

ADR-065 mandates that webhook handlers acknowledge fast and process work asynchronously on a job queue. Each app currently wires its own BullMQ or database-backed queue with ad-hoc retry logic and no shared conventions. Per ADR-014 (Open Source First), we designate bullmq and ioredis as the community libraries for background job processing and re-export them through @dmwd-io/jobs. The platform contribution is documenting REDIS_URL as the standard env var and {domain}.{action} as the queue naming convention — no custom QueueProvider interface is built on top.

Goals

• Designate bullmq / ioredis as the platform standard for background job queues per ADR-014.
• Ship @dmwd-io/jobs as a thin re-export package so all apps import from a single pinned source.
• Document REDIS_URL as the canonical env var for the Redis/Valkey connection string.
• Document {domain}.{action} as the platform queue naming convention per ADR-065.
• Define dead-letter and retry conventions that apps should adopt (using BullMQ’s native configuration).
• Serve as the async processing backbone referenced by ADR-065 webhook handlers.

Non-Goals

• Building a custom provider interface or adapter layer — the library’s own API is the interface (per ADR-014).
• SQS or database-polling adapters (non-default; use BullMQ per ADR-027).
• Implementing a job dashboard or monitoring UI.
• FIFO ordering guarantees (jobs are at-least-once, not exactly-once ordered).
• Priority queues — all jobs in a queue share the same priority.
• Workflow orchestration or job chaining (DAG execution).

---

Scope

In Scope

Area	Description
@dmwd-io/jobs package	Re-exports bullmq and ioredis; pins to platform-approved versions
Env var conventions	REDIS_URL as the standard connection string env var
Queue naming conventions	{domain}.{action} convention documented and enforced by example
Retry conventions	Recommended BullMQ defaultJobOptions (3 attempts, exponential backoff) documented
Dead-letter conventions	Platform recommendation for BullMQ failedJobsHistoryLength and DLQ patterns
Scheduled jobs	Documentation of BullMQ’s native delay and repeat options

Out of Scope

Area	Reason
Custom QueueProvider interface or adapters	ADR-014: library API is the interface
SQS or database-polling adapters	Non-default alternatives; BullMQ is the ADR-027-designated adapter
Job dashboard / admin UI	Operational tooling, not a library contract
Exactly-once delivery	At-least-once is sufficient; idempotency is the handler’s responsibility
Job priority levels	Future extension; initial scope is FIFO-ish within a queue
Workflow / DAG orchestration	Complex job chains are a separate concern

---

Users and Pain Points

User Groups

User	Description	Needs
App developers	Engineers building async processing pipelines	A single pinned import for BullMQ / ioredis with documented platform conventions
QA engineers	Testers validating async workflows	BullMQ’s built-in sandbox worker or a lightweight in-memory approach for deterministic tests
Design system maintainers	Library contributors	A clear re-export surface that webhook handlers can depend on per ADR-065

Pain Points

User	Pain Point	Impact
App developers	Each app wires its own BullMQ/SQS setup with different retry logic	Inconsistent failure handling; some apps retry forever, others drop silently
App developers	No shared env var convention — each app names the Redis connection differently	Config drift across services; ops burden at deploy time
App developers	No dead-letter convention — failed jobs disappear	No visibility into permanent failures; no recovery path

---

Definitions

Term	Definition
Job	A unit of async work with a type, payload, and retry policy
Queue	A named channel for jobs of a specific type
Retry policy	Configuration for how many times and at what intervals a failed job retries
Dead-letter queue (DLQ)	A holding area for jobs that exhausted all retry attempts
Scheduled job	A job that should not be processed until a specific time
At-least-once delivery	Each job is delivered to a processor at least once; duplicates are possible
Backoff	Increasing delay between retry attempts (exponential with jitter)

---

Current State

Existing Behavior

There is no shared queue abstraction. ADR-065 references async job processing but does not implement it. Apps that need background processing wire BullMQ or database polling directly.

Current Limitations

• No shared TypeScript types for jobs, retry policies, or dead-letter entries.
• No shared env var convention — each app names its Redis connection string differently.
• No dead-letter conventions — failed jobs are logged and lost.
• No scheduled job support — apps use setTimeout or cron for deferred work.
• Webhook handlers (per ADR-065) have no standard queue to hand off work to.

Existing Workarounds

• Apps create ad-hoc BullMQ queues with hard-coded retry counts.
• Some apps use database rows as a poor-man’s queue with polling.
• Failed jobs are logged to console with no recovery mechanism.

---

Proposed Solution

Summary

Create a packages/jobs/ directory containing @dmwd-io/jobs. The package re-exports bullmq and ioredis as peer dependencies so all platform apps import from a single pinned source. No custom interface is built — the library’s API is the interface. Platform conventions (env var names, queue naming, recommended retry defaults) are documented in the package README and Storybook docs page.

import { Queue, Worker, QueueEvents } from '@dmwd-io/jobs';

const queue = new Queue('email.send', {
  connection: { url: process.env.REDIS_URL },
});

The env var REDIS_URL is the canonical connection string used by all services. Queue names follow {domain}.{action} convention per ADR-065. No custom adapter or factory is required.

Key Conventions

• REDIS_URL — platform env var for the Redis/Valkey connection string.
• Queue names follow {domain}.{action} (e.g. email.send, report.generate).
• Recommended default retry options: { attempts: 3, backoff: { type: 'exponential', delay: 1000 } }.
• Dead-letter: use BullMQ’s failedJobsHistoryLength and a dedicated {queue}.dlq queue for manual replay.

User Experience

End users are not directly affected. The queue package powers async processing behind API routes and webhook handlers.

Developer Experience

Developers import directly from @dmwd-io/jobs using BullMQ’s native API. Platform conventions (env var name, queue naming, retry defaults) are documented in one place. No adapter or factory wrapper is needed.

---

Requirements

ID	Requirement	Priority	Notes
FR-001	@dmwd-io/jobs package re-exports bullmq and ioredis	Must	Per ADR-014: thin re-export, no custom interface
FR-002	Package documents REDIS_URL as the platform env var	Must	-
FR-003	Package documents {domain}.{action} queue naming convention	Must	-
FR-004	Package documents recommended BullMQ retry defaults	Should	3 attempts, exponential backoff, 1s base
FR-005	Package documents dead-letter conventions using BullMQ native options	Should	-
FR-006	Storybook docs page covers import, env var, and naming conventions	Must	-

Priority Definitions

Priority	Meaning
Must	Required for this feature to ship.
Should	Important, but can be deferred if needed.
Could	Nice to have. Not required for initial release.

---

Functional Requirements

ID	Requirement	User Benefit	Priority
FUNC-001	import { Queue, Worker } from '@dmwd-io/jobs' resolves to the pinned bullmq version	All apps use the same BullMQ version without independent version management	Must
FUNC-002	import { Redis } from '@dmwd-io/jobs' resolves to the pinned ioredis version	Redis client version is consistent across services	Must
FUNC-003	Failed jobs retry using BullMQ’s native attempts / backoff options	Transient failures recover automatically via library-native retry	Must
FUNC-004	Jobs that exhaust retries are captured using BullMQ’s failedJobsHistoryLength	Permanent failures are inspectable without custom DLQ code	Must
FUNC-005	Scheduled/delayed jobs use BullMQ’s native delay and repeat options	Deferred work without external cron or custom adapter	Should
FUNC-006	Package README and Storybook page document REDIS_URL usage with a connection example	Developers can connect in under 5 minutes	Must

---

Non-Functional Requirements

ID	Requirement	Category	Priority
NFR-001	@dmwd-io/jobs adds no runtime logic beyond re-exports	Maintainability	Must
NFR-002	bullmq and ioredis are listed as peer dependencies, not bundled	Bundle size	Must
NFR-003	Package ships TypeScript types (pass-through from bullmq / ioredis)	Developer experience	Must
NFR-004	Package version is pinned in platform package.json across all services	Consistency	Must
NFR-005	pnpm typecheck passes with zero errors	Quality gate	Must

---

API / Interface Requirements

Public API

The public API is BullMQ’s native API, re-exported from @dmwd-io/jobs. Key exports:

Name	Source	Description
Queue	bullmq	Create and manage a named job queue
Worker	bullmq	Process jobs from a named queue
QueueEvents	bullmq	Subscribe to queue lifecycle events
Job	bullmq	Job instance type
Redis	ioredis	Redis client (use with connection option)

Example Usage

import { Queue, Worker } from '@dmwd-io/jobs';

// Enqueue work
const queue = new Queue('email.send', {
  connection: { url: process.env.REDIS_URL },
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 1000 },
  },
});

await queue.add('send-welcome', {
  to: 'user@example.com',
  subject: 'Welcome',
  body: 'Hello!',
});

// Process work
const worker = new Worker(
  'email.send',
  async (job) => {
    await sendEmail(job.data.to, job.data.subject, job.data.body);
  },
  { connection: { url: process.env.REDIS_URL } },
);

API Notes

• Queue names follow {domain}.{action} convention per ADR-065.
• All services read REDIS_URL from the environment — no hard-coded connection strings.
• No custom factory or adapter is needed; instantiate Queue and Worker directly.

---

Accessibility Requirements

ID	Requirement	Notes
A11Y-001	Package is a backend data layer with no UI — accessibility requirements do not apply directly	No UI components

Checklist

• Keyboard support is defined. (N/A — no UI)
• Focus behavior is defined. (N/A — no UI)
• Screen reader behavior is defined. (N/A — no UI)
• Color contrast requirements are met. (N/A — no UI)
• Reduced motion behavior is considered. (N/A — no UI)
• Semantic HTML expectations are documented. (N/A — no UI)
• ARIA usage is defined only where needed. (N/A — no UI)

---

Content and Documentation Requirements

ID	Requirement	Location	Priority
DOC-001	Storybook docs page explaining @dmwd-io/jobs, env var conventions, and ADR-065 integration	Storybook	Must
DOC-002	Package README with import example, REDIS_URL setup, and queue naming convention	packages/jobs/README.md	Must
DOC-003	Example: webhook handler enqueuing work per ADR-065	Storybook	Must
DOC-004	Example: retry and dead-letter configuration using BullMQ native options	Storybook	Should
DOC-005	Example: scheduled/delayed job using BullMQ delay option	Storybook	Should

---

Dependencies

Dependency	Type	Owner	Status	Notes
ADR-065 webhook handling	Architecture	Engineering	Ready	Requires async job handoff from webhook handlers
ADR-027 default tech stack	Architecture	Engineering	Ready	Designates BullMQ (TS) and River (Go) as default job queue implementations
ADR-014 open source first	Architecture	Engineering	Ready	Mandates thin re-export over custom interface
Webhook handling package	Library	Engineering	Not Started	Will use @dmwd-io/jobs for job dispatch

---

Risks and Tradeoffs

Risk / Tradeoff	Impact	Mitigation
No custom test adapter — BullMQ tests require Redis or a sandbox worker	Tests are harder to run without infrastructure	Document BullMQ sandbox worker and testcontainers patterns; add to platform test utilities
At-least-once delivery means handlers must be idempotent	Developers may forget idempotency	Document idempotency requirement prominently in Storybook docs
No priority queue support	High-priority work may wait behind bulk jobs	Recommend separate queue names for different priority levels
Thin re-export means upstream BullMQ breaking changes surface directly	Major BullMQ version bumps require coordinated updates	Pin to a minor range; update in one place (packages/jobs) across all consumers

---

Open Questions

ID	Question	Owner	Status	Resolution
Q-001	Should @dmwd-io/jobs also re-export a lightweight in-memory test helper (BullMQ’s own sandboxedWorker or a small wrapper)?	David Holmes	Open	—
Q-002	Should retry policy support a retryIf predicate for conditional retries based on error type?	David Holmes	Open	—
Q-003	Should the platform document pause / resume operational patterns using BullMQ’s native queue methods?	David Holmes	Open	—
Q-004	Should the package also document cron-style recurring jobs using BullMQ’s repeat option?	David Holmes	Open	—

---

Acceptance Criteria

ID	Criteria	Related Requirement
AC-001	import { Queue, Worker } from '@dmwd-io/jobs' resolves correctly in a TypeScript project	FR-001
AC-002	import { Redis } from '@dmwd-io/jobs' resolves correctly	FR-001
AC-003	Package README documents REDIS_URL with a working connection example	FR-002
AC-004	Storybook docs page covers import, env var, queue naming, and retry conventions	FR-006
AC-005	pnpm typecheck passes with zero errors	NFR-005
AC-006	bullmq and ioredis are listed as peer dependencies in packages/jobs/package.json	NFR-002
AC-007	No custom provider interface, adapter, or factory is implemented in the package	ADR-014

---

LLM Handoff Instructions

Expected LLM Behavior

• Create packages/jobs/ if it does not exist.
• Add packages/jobs/package.json with bullmq and ioredis as peer dependencies (not bundled).
• Create packages/jobs/src/index.ts that re-exports everything from bullmq and ioredis:

  export * from 'bullmq';
  export * from 'ioredis';

• Document REDIS_URL as the canonical env var in the package README.
• Document {domain}.{action} queue naming convention per ADR-065.
• Document recommended BullMQ defaultJobOptions: { attempts: 3, backoff: { type: 'exponential', delay: 1000 } }.
• Run pnpm typecheck and confirm it passes.

LLM Should Not

• Build a custom QueueProvider interface, test adapter, mock adapter, or factory function.
• Import BullMQ into any file other than the re-export index.ts.
• Implement SQS, database-polling, or other non-ADR-027 queue adapters.
• Add runtime logic beyond re-exports.
• Implement a job dashboard or monitoring UI.
• Implement exactly-once delivery guarantees.

---

Decision Log

Date	Decision	Reason	Owner
2026-05-26	Provide both a test adapter (synchronous) and a mock adapter (capture-only)	Different testing scenarios need different behaviors — some tests need processing, others just need enqueue assertions	David Holmes
2026-05-26	At-least-once delivery, not exactly-once	Exactly-once requires distributed transactions; at-least-once with idempotent handlers is simpler and sufficient	David Holmes
2026-05-26	Default retry policy: 3 attempts, 1s base, 30s max, 2x multiplier	Conservative defaults that handle transient failures without long waits	David Holmes
2026-05-26	BullMQ designated as the TypeScript queue adapter	ADR-027 §2 explicitly names BullMQ (TS) and River (Go) as the default job queue implementations	David Holmes
2026-06-02	Reframed per ADR-014 (Open Source First): adopt bullmq / ioredis as thin re-export rather than building a custom provider interface. Library API is the interface.	ADR-014 mandates designating a community library over building a custom abstraction when the library already solves the problem	David Holmes

---

Document History

Date	Author	Change
2026-05-26	David Holmes	Initial draft
2026-06-02	David Holmes	Reframed per ADR-014: ship as thin re-export of bullmq / ioredis, drop custom interface.