How we built RAG on 397B params for $0.005 per conversation

Our average WhatsApp AI conversation on MsgHub costs us $0.005 — about ₹0.42 — across reasoning, retrieval, and classification combined. A GPT-4 Turbo baseline of the same conversation would run $0.04–0.10. We’re 8–20× cheaper, and the quality gap is small enough that customers can’t tell.

This post is the engineering breakdown. The screenshot above is real OpenRouter usage from a live conversation: four Qwen3.5 397B reasoning calls plus three embedding lookups, totalling $0.00785. Mid-complexity conversation; our mean across all conversations sits at $0.005.

Why this number matters

Indian SMB unit economics are tight. A WhatsApp Business conversation already costs your business ₹0.14 (Utility), ₹0.78 (Marketing) or ₹0.13 (Authentication) — Meta’s prices, not ours. If the AI sitting on top costs more than the WhatsApp conversation itself, the math breaks. Most MsgHub customers run on ₹2,000–10,000 monthly platform budgets. There’s no room for $0.05-per-conversation AI.

So we set a hard ceiling: AI cost must be less than 30% of the WhatsApp conversation cost it accompanies. For a ₹0.14 Utility conversation, that’s ₹0.04 ($0.0005) — which we miss; we run higher on reasoning-heavy threads. For a ₹0.78 Marketing conversation, it’s ₹0.23 ($0.0028) — comfortably under. Blended, we land at $0.005 mean and customers don’t see a per-AI line item; we eat it under the flat platform fee.

To make that economically possible, the model stack had to look very different from what an SF-based AI startup would default to.

Our stack

That’s the entire AI bill of materials. No Cohere reranker, no Pinecone, no Anthropic. Each line is the cheapest option that wasn’t a quality regression — checked with A/B tests against a GPT-4 Turbo + Pinecone baseline.

The math, shown openly

A typical MsgHub conversation has 3 turns. Each turn does:

1. Embed the customer query → ~6 tokens → $0.00000012 (OpenAI)
2. Retrieve the top-K chunks from pgvector — local, free
3. Classify intent with Llama Maverick — short prompt, ~200 tokens → ~$0.00002
4. Reason and respond with Qwen3.5 397B — ~1500 input tokens (RAG context + system prompt + history), ~300 output tokens → ~$0.0021

Per turn ≈ $0.0021. Three turns ≈ $0.0063. Shorter conversations (1–2 turns) drag the mean down to $0.005.

The screenshot you saw at the top of this post is from one mid-complexity 3-turn conversation:

Four model calls because the reasoning path includes tool calls (search KB, lookup contact, escalate to human) — each tool call is a separate Qwen round-trip. Output speed averaged 45 tokens/second across the four calls, which is fast enough for WhatsApp chat (customers tolerate 1–3 second response latencies on messaging better than on voice).

Why Qwen3.5 397B A17B specifically

The “A17B” suffix is the headline. It’s a Mixture-of-Experts model with 397B total parameters but only ~17B active per token. That structure is the entire reason it’s affordable:

• Quality comes from the 397B parameter capacity — comparable to GPT-4 class models on most reasoning benchmarks (MMLU, MATH, HumanEval), and notably strong on Indian-language tasks where the training data overlap is real.
• Inference cost scales with active params, not total. Running a query through 17B active params costs roughly what running a dense 17B model would — but you get reasoning quality closer to a dense 200B+ model.

We pay AtlasCloud (the OpenRouter provider) ~$0.0008 per 1K input tokens, ~$0.0033 per 1K output tokens. GPT-4 Turbo on the same routes would be $0.010 in / $0.030 out — about 12× the input cost and 9× the output cost for our average call mix.

On quality: we ran 200 production-realistic prompts (Hindi/English mixed customer queries, product catalogue lookups, order status questions) through GPT-4 Turbo, Claude 3.5 Sonnet, Mistral Large, and Qwen3.5 397B. Blind-graded by three of us:

• GPT-4 Turbo: 187/200 acceptable
• Claude 3.5 Sonnet: 184/200
• Qwen3.5 397B: 178/200
• Mistral Large: 168/200

A 9-point gap from GPT-4 to Qwen, for ~1/12th the cost. That’s the trade we made. We accept slightly more “could be sharper” answers in exchange for being able to actually ship the product.

Why Llama 4 Maverick for classification

The first thing a customer query needs is a routing decision: is this a product question (→ knowledge base lookup), an order status query (→ database lookup with tool call), a complaint (→ human handoff), a delivery question (→ shipping policy retrieval), or off-topic (→ polite deflection)?

You don’t need 397B params to answer that. A small fast classifier does it for two orders of magnitude less. Llama 4 Maverick at ~$0.0001 / 1K tokens for input, ~$0.0005 / 1K tokens for output, generally returns in under 200ms.

We tried doing classification inside the Qwen reasoning prompt to save a hop. Two problems:

1. Qwen sometimes “decided” mid-reasoning to flip categories, leading to weird mixed outputs.
2. We couldn’t cheaply track classification accuracy as a separate metric for tuning.

Splitting classification out also lets us swap it independently. We’re trialing gpt-oss-120b (just released) against Maverick on the same 5,000-example evaluation set this week.

Why text-embedding-3-small (and not Voyage, Cohere, or open-source)

Customer queries on WhatsApp are short. Median query length in our logs is 6 tokens (e.g. “is the 5L disinfectant in stock?”, “track my order”). At 6 tokens per embed, OpenAI’s text-embedding-3-small costs us ~$0.0000001 per query. We do roughly 30,000 embeds per day across all tenants. That’s $0.003/day on embeddings. Total. The price of one chai.

We tested Voyage voyage-3-lite — better recall by ~3 points on our eval — but at $0.02/1M tokens for the cheaper tier, and we’d be paying for retrieval quality that doesn’t move our customer-visible metric (answer correctness). When 3-point retrieval improvement doesn’t change the reasoning model’s answer, you don’t pay for it.

Cohere embed-v3 was solid but priced for higher volumes than ours. Open-source (sentence-transformers, bge-large) would have needed a GPU somewhere — we evaluated the all-in cost including infra and decided against. OpenAI’s free-at-our-scale pricing wins on operational simplicity alone.

The RAG architecture (in 12 lines)

Customer message arrives (WhatsApp webhook)
        ↓
[Llama Maverick]  ← classify intent  ($0.00002)
        ↓
  Intent === 'product_question' or 'support'?
        ↓
[text-embedding-3-small]  ← embed query  ($0.0000001)
        ↓
[pgvector top-K]  ← retrieve relevant KB chunks  ($0)
        ↓
[Qwen3.5 397B]  ← reason over chunks + history  ($0.002)
        ↓
  Tool calls?  ← yes  → another Qwen round-trip  ($0.002)
        ↓                no
[Response sent to customer via WhatsApp Cloud API]

That’s it. No agentic graph of seven specialised models. No fine-tuning. No proprietary algorithms. Just well-chosen off-the-shelf parts wired together with strict cost discipline at each hop.

What we tried that didn’t work

GPT-4 Turbo standalone. Beautiful quality. Wrecked the unit economics at $0.04–0.06/conversation. We could have charged premium pricing to cover it, but our SMB segment doesn’t have that price elasticity. Killed after a 30-day pilot.

Claude 3.5 Sonnet. Comparable quality to GPT-4. Comparable cost. Same problem, different vendor.

Mistral Large. ~70% the price of GPT-4. Quality dropped notably on Hindi and Gujarati customer queries — Mistral’s training corpus is more European-language-weighted. Customers in Rajkot/Surat noticed.

Self-hosted Llama 3.1 70B on Hetzner. We spent two weeks on this. A single A100 instance costs ~₹40,000/month before bandwidth. Even at high utilisation, the per-conversation cost was higher than Qwen on OpenRouter once you factor in idle hours, warm-up latency, and an engineer’s time to babysit GPU memory. Hard pass.

Pinecone for vectors. Worked great. Costs $70/month for the starter tier we’d outgrow within a quarter. pgvector on our existing Postgres does the same job for $0 incremental. We left Pinecone behind two months in.

Single-model architecture (no classification step). Tried letting Qwen do its own routing inside one mega-prompt. Classification accuracy dropped 8 points and average response latency increased 600ms because the model spent reasoning tokens on routing logic. Splitting it back out was a 30-minute change that recovered both.

Trade-offs we openly accept

• Qwen3.5 occasionally inserts Chinese characters in mixed-Hindi responses (~0.4% of outputs). We catch this with an output validator (regex on suspect ranges) and silently retry. Acceptable.
• 47 tokens/second output is slower than GPT-4 Turbo’s ~60+. Fine for WhatsApp. Would be tight for synchronous voice (we’re not in voice yet).
• Indian-language outputs use more tokens per character due to BPE encoding quirks. A 100-word Hindi response can be 350+ tokens versus ~140 for English. We factor this into per-tenant cost projections.
• AtlasCloud occasionally has 3–5 second cold-start latencies on Qwen3.5. ~2% of calls. We fall back to a second OpenRouter provider when this hits.
• Llama Maverick over-escalates to humans on ambiguous intents — better than under-escalating, but tuning is ongoing.

What we’d change next

1. Semantic caching for repeated questions (“track my order”, “what’s your address”, “shipping policy”) — could shave another 25–30% off reasoning cost. On the roadmap for Q3.
2. Trial gpt-oss-120b for classification against Llama Maverick on our 5,000-prompt eval set. Initial signals are competitive.
3. Hindi/Gujarati-optimised embeddings when an Indian-language-native option ships at competitive pricing. Currently English embeddings handle Hindi-in-Roman-script just fine, but Devanagari script queries take a quality hit we’d like to recover.
4. Move from OpenRouter to direct AtlasCloud once volume justifies the contract — saves ~5% routing markup.

The general takeaway

The “expensive AI” reputation that LLMs have is mostly an artifact of defaulting to the SF stack (GPT-4 + Pinecone + Cohere reranker + LangChain) without questioning whether each piece is needed at your particular scale.

If you’ve sized your model choice to your actual quality threshold (not “the best benchmark score we can find”), and your vector store to your actual query volume, and your embeddings to your actual query length — the bill drops by 10–20×. Often without your customers being able to tell the difference.

Boring engineering wins. Cost-aware engineering wins more.

Footnotes

• Pricing accurate as of OpenRouter and OpenAI rates in May 2026. Models and prices change; the approach outlasts the specific numbers.
• We have no affiliation with AtlasCloud or OpenRouter beyond being paying customers.
• Cost figures are our averages; your mileage will vary by conversation complexity, language mix, and KB depth.