LLMNarrative answers a question brands have started asking: when someone asks an AI about you, what does it say? Answering that properly means running the same prompts across many models — OpenAI, Anthropic, DeepSeek, Perplexity, xAI — and comparing the narratives. Which turns the backend into a fan-out orchestration problem with slow, flaky, rate-limited upstreams.
The unit of work is (prompt, model)
The first design decision was granularity. An evaluation run for a company might be dozens of prompts across five models. I enqueue one RQ job per (prompt, model) pair rather than one job per run. This buys three things:
- A failure is retried at the pair level. One model timing out doesn’t re-burn tokens on the four that succeeded.
- Progress is naturally incremental — the UI can show results as pairs complete instead of waiting for the whole run.
- Slow providers don’t block fast ones; the pairs just drain through the worker pool at whatever pace each upstream allows.
RQ on Redis is an unglamorous choice and that’s the point. I’ve run Redis+RQ in production at the day job for years; I know exactly how it fails, which is the property I want in the component holding everything together.
Providers behind one door
All model calls route through OpenRouter with a provider/model string. The worker doesn’t know or care whose SDK is behind a model — adding a new one is a config entry, not a code change. Prompt construction is Jinja2 templates with per-company context injected at render time, so the prompt library is decoupled from both the company and the provider. When a new model shows up, evaluating a client against it is genuinely zero backend work.
Each worker job renders its template, makes the call, parses out the sentiment breakdown, and persists the result along with job lifecycle state. Retries with backoff are handled at the job level; a pair that keeps failing lands in a failed state a human can see, rather than silently vanishing.
The polling problem
Long-running fan-out jobs plus a web UI means polling (I didn’t want to operate websockets for this). The naive version polls every two seconds forever and DDoSes your own API the moment a few users run evaluations concurrently. The fix is exponential backoff on the client: poll quickly at first while cheap pairs finish, then stretch the interval as the run settles into the slow tail. Combined with pair-level results, the UI feels live early — which is when users are actually watching — and quiet later, when they’ve switched tabs anyway.
What this design got right
Nothing here is novel, and that’s what I’d defend about it. Queue per smallest retryable unit, one gateway in front of N providers, templates decoupled from context, backoff on the client. Every piece is replaceable and inspectable. When a provider changes behavior — and one always does — the blast radius is one job type and one config line, not an evening of my life.