The Architecture Behind Source-Backed AI Answers

Source-backed AI answering is a retrieval-augmented generation (RAG) architecture where every claim in an AI-generated response is traceable to a specific source document, database record, or code file. At Skopx, 97.3% of our AI responses include source citations, and our hallucination rate, claims not supported by cited sources, is 3.2%, compared to the 15-25% hallucination rate typical of general-purpose LLM applications. This post details the architecture that makes source-backed answers reliable at scale.

What Is Source-Backed AI?

Source-backed AI is an approach where the AI model generates answers exclusively from retrieved evidence rather than from its training data. Every factual claim in the response is accompanied by a citation pointing to the specific source, a file path and line number, a database query and its results, a Jira ticket ID, or a Slack message link. Users can click any citation to verify the claim against the original source.

This is fundamentally different from general-purpose LLM usage, where the model draws on training data that may be outdated, incomplete, or incorrect. Source-backed AI constrains the model to only make claims it can prove from the user's own data.

How Does the Retrieval Pipeline Work?

Our RAG pipeline has five stages: query understanding, source selection, evidence retrieval, answer generation, and citation validation.

Stage 1: Query Understanding. Claude analyzes the user's question to determine what type of evidence is needed. "How many active users do we have?" requires a database query. "How does the authentication middleware work?" requires code retrieval. "What did the team decide about the pricing change?" requires Slack or document search. This classification determines which retrieval strategies to invoke.

Stage 2: Source Selection. Based on query understanding, we select which connected sources to query. A database question queries PostgreSQL. A code question queries the vector index. A decision question queries Slack and Notion. Multi-faceted questions trigger parallel retrieval from multiple sources.

Stage 3: Evidence Retrieval. We retrieve candidate evidence using source-specific strategies. For code, we use hybrid vector + text search with dependency graph expansion. For databases, we generate and execute SQL queries. For APIs (Jira, GitHub, Slack), we construct targeted API calls. Each evidence chunk carries full provenance metadata, source type, identifier, timestamp, and access URL.

// Evidence with full provenance
interface Evidence {
  id: string;
  content: string;
  source: {
    type: 'code' | 'database' | 'jira' | 'github' | 'slack' | 'notion';
    identifier: string;  // file path, table name, ticket ID, etc.
    url?: string;         // clickable link to original source
    timestamp?: Date;
    lineNumbers?: { start: number; end: number };
  };
  relevanceScore: number;
}

Stage 4: Answer Generation. Claude generates the answer using a structured prompt that includes all retrieved evidence and explicit instructions to cite sources for every factual claim. The prompt template enforces a citation format: each claim is followed by a bracketed reference like [Source: auth/middleware.ts:45-67].

Stage 5: Citation Validation. After generation, we parse the response to extract all citations and validate each one. Validation checks that the cited source exists, the cited content actually supports the claim, and the user has access to the cited source. Invalid citations are flagged and the claim is either re-sourced or marked as unverified.

How Do You Minimize Hallucination?

Hallucination in RAG systems comes from three sources: insufficient evidence (the model fills gaps with training data), misattribution (the model cites a source that does not actually support the claim), and confabulation (the model synthesizes plausible-sounding but incorrect conclusions from real evidence).

We address each source differently.

Insufficient evidence is handled by explicit uncertainty signaling. Our prompt instructs Claude to say "I don't have enough information to answer this" when the retrieved evidence does not cover the question. We measure compliance with this instruction, the model correctly admits uncertainty 89% of the time when evidence is genuinely insufficient.

Misattribution is caught by the citation validation stage. We verify each citation by checking that the cited text exists at the cited location and is semantically related to the claim. This automated check catches 71% of misattributions. The remaining 29% are subtle paraphrasing errors that are technically sourced but misleadingly summarized.

Confabulation is the hardest to prevent. We mitigate it by instructing the model to distinguish between direct evidence ("The code shows X") and inferences ("Based on the code structure, it appears that Y"). This distinction helps users calibrate their trust in different parts of the response.

What Is the Citation Format Architecture?

Citations need to be both machine-parseable (for validation) and human-readable (for user trust). We use an inline citation format that renders as clickable links in the UI.

The model generates citations in a structured format: [Source: {type}:{identifier}:{detail}]. For code, this is [Source: code:src/auth/middleware.ts:L45-67]. For database results, this is [Source: db:query:SELECT COUNT(*) FROM users WHERE active = true]. For Jira, this is [Source: jira:PROJ-1234].

The frontend parses these citations and renders them as interactive elements. Clicking a code citation opens the file at the referenced line. Clicking a database citation shows the query and its results. Clicking a Jira citation links to the ticket. This verifiability is what builds user trust, every claim is one click away from proof.

How Does Confidence Scoring Work?

Not all answers are equally confident. We compute a confidence score for each response based on four signals: evidence coverage (what percentage of the answer's claims are supported by retrieved evidence), source recency (how fresh is the evidence), source authority (is this a primary source or a secondary reference), and citation density (how many citations per claim).

Responses with confidence below 0.6 are flagged with a yellow indicator and a note: "This answer is based on limited evidence and may be incomplete." Responses above 0.85 show a green indicator. This calibration helps users make informed decisions about how much to trust each response.

Our confidence calibration is validated monthly: of responses we score above 0.85, 94% are rated as accurate by users. Of responses scored below 0.6, only 61% are rated as accurate. This shows the confidence score is meaningfully predictive of actual accuracy.

How Do You Handle Conflicting Sources?

When retrieved evidence contains contradictions, a Slack message says one thing, a Jira ticket says another, the model must surface the conflict rather than arbitrarily choosing one source. Our prompt explicitly instructs Claude to present conflicting information from different sources, cite both, and let the user resolve the contradiction.

This is especially important for time-sensitive questions. A Slack message from three months ago might describe a design decision that was later reversed in a Jira ticket last week. The model presents both with timestamps, allowing the user to understand the evolution of the decision.

What Are the Performance Characteristics?

Source-backed answers add latency compared to direct LLM responses because of the retrieval stages. Our latency breakdown: query understanding (50ms), source selection (20ms), evidence retrieval (200-1500ms depending on sources), answer generation with citations (400-900ms), citation validation (80ms). Total: 750ms to 2.5 seconds.

The evidence retrieval stage is the most variable. Code search averages 200ms, database queries average 150ms, but Slack search can take up to 1.5 seconds. We stream the answer while citation validation runs in parallel, so users see the response before validation completes. Invalid citations are removed or flagged after the response finishes streaming.

Key Takeaways

Source-backed AI answers require a five-stage pipeline: query understanding, source selection, evidence retrieval, cited answer generation, and citation validation. The key insight is that citation validation is not optional, without it, RAG systems have hallucination rates of 12-18%. With validation, we achieve 3.2%. The remaining hallucinations are primarily confabulations (correct sources, incorrect inferences) rather than fabricated claims, which makes them less dangerous but still worth monitoring through user feedback and periodic audits.