AI Analysis Quality Metrics

Last updated: 2026-11-05

Overview

Inqura runs an automated check on every AI analysis. The check measures how well the analysis was produced: whether its claims trace to your evidence record, whether it addressed the question, and how relevant the retrieved evidence was. It helps you decide which analyses to review closely and where evidence may be missing.

The check looks at:

1. Faithfulness - Are the AI's claims grounded in the evidence?
2. Coverage - Does the analysis address all aspects of the question?
3. Retrieval Quality - How relevant was the evidence found?
4. Schema Validation - Is the output in the expected structure?

These combine into one result, shown on the analysis as the Check badge.

The check is not a judgment of the evidence. How strongly the evidence supports a finding is the finding's own rating (Established, Probable, Possible or Insufficient), shown separately on each finding and conclusion. The two are computed from different inputs and can disagree: a well-produced analysis can conclude that the evidence is insufficient, and an analysis of strong evidence can still fail the check if some of its claims were not traced.

---

The Check Badge

Each analysis shows one of three check results:

Badge	Color	Meaning
Check: passed	Green	The analysis's claims were traced to your evidence record and the question was addressed.
Check: review needed	Amber	Some claims were not traced to evidence, parts of the question were not covered, or validation failed. Review before relying.
Check: not yet run	Gray	The faithfulness and coverage checks have not finished on this analysis yet, or could not complete.

Faithfulness and coverage run in the background after an analysis is generated, so a new analysis shows Check: not yet run for a short time before its result appears. The analysis content does not change when the check finishes.

How the Result Is Decided

The check passes when all of these hold:

• the output passed schema validation;
• faithfulness is at least 85% and coverage is at least 80%;
• if retrieval was weak (average similarity below 0.15), faithfulness and coverage are both at least 95% with no coverage gaps;
• the relevancy check did not rate the analysis as only tangentially related to the question, or as not addressing it.

Otherwise the check reads review needed. A failed schema validation is always review needed.

---

Faithfulness Score

What It Measures

Faithfulness measures whether the factual claims made in the AI's analysis are actually supported by your evidence record. This is critical for ensuring the AI isn't "hallucinating" or making unsupported assertions.

How It Works

1. The system identifies factual claims in the AI's output
2. Each claim is checked against your investigation's full evidence record (evidence text plus attachment extracted text; very large investigations are capped by size)
3. Claims are marked as "supported" if evidence exists, "unsupported" if not

Score Interpretation

Score	Interpretation	Action
95-100%	Excellent - Nearly all claims verified	High confidence in factual accuracy
85-94%	Good - Most claims verified	Review any flagged unsupported claims
70-84%	Moderate - Notable unsupported claims	Carefully verify conclusions before use
<70%	Poor - Many unverified claims	Consider regenerating or manual review

Example Output

Faithfulness: 91%
11/12 claims verified

Unsupported claims:
- "The vendor has a history of similar billing issues"

What to Do When Faithfulness Is Low

1. Review unsupported claims - The system lists specific claims that couldn't be verified
2. Check if evidence exists - The claim might be true but evidence wasn't uploaded
3. Add missing evidence - Upload documents that support the claims
4. Regenerate analysis - After adding evidence, generate a new analysis

---

Coverage Score

What It Measures

Coverage measures whether the AI's analysis addresses all the key aspects of the question or topic being analyzed. A high coverage score means the analysis is comprehensive; a low score indicates gaps.

How It Works

1. The question/topic is broken down into constituent elements
2. The AI's response is checked to see which elements are addressed
3. Elements not addressed are flagged as "gaps"

Score Interpretation

Score	Interpretation	Action
95-100%	Comprehensive - All aspects addressed	Analysis is thorough
85-94%	Good - Minor gaps only	Review identified gaps
70-84%	Partial - Notable gaps	Consider follow-up analysis on gaps
<70%	Incomplete - Major gaps	Regenerate with more specific direction

Example Output

Coverage: 80%
4/5 elements addressed

Gaps:
- No analysis of timeline discrepancies

Understanding Coverage Elements

The system identifies elements based on the question structure:

Example Question: "Did the vendor submit invoices for services that were not performed, and what is the financial impact?"

Elements identified:

1. Whether invoices were submitted
2. Whether services were performed
3. Connection between invoices and services
4. Financial impact assessment

What to Do When Coverage Is Low

1. Review the gaps list - Understand what wasn't addressed
2. Check evidence availability - Gaps may exist because evidence is missing
3. Provide direction - Regenerate with specific instructions to address gaps
4. Split the question - Complex questions may need multiple analyses

---

Retrieval Quality

What It Measures

Retrieval quality measures how effectively the system found relevant evidence for the analysis. It uses semantic search to find evidence chunks that relate to the question being analyzed.

How It Works

Inqura uses a three-stage hybrid retrieval pipeline:

1. Keyword search (PostgreSQL tsvector) finds evidence by exact terms
2. Semantic search (Titan Embeddings V2 via pgvector) finds evidence by meaning
3. Reranking (Cohere Rerank 3.5) re-scores merged results for query-time relevance
4. Results are combined via team-draft interleaving before reranking
5. Statistics are calculated on the final retrieved set

Retrieval Levels

Level	Avg Similarity	Meaning
Strong	> 0.35	Highly relevant evidence found
Moderate	0.15 - 0.35	Reasonably relevant evidence
Weak	< 0.15	Limited relevance in available evidence

Note: Thresholds calibrated for Amazon Titan Text Embeddings V2. Titan V2 cosine similarities are naturally lower than other embedding models — relevant evidence averages ~0.24 similarity. JE Vectors recalibrates these thresholds whenever the embedding model changes.

Detailed Statistics

The system provides detailed retrieval statistics:

Metric	Description
Chunks retrieved	Total evidence chunks found above threshold
Chunks used	Chunks actually included in AI context (may be limited by token budget)
High relevance	Chunks with similarity > 0.40
Medium relevance	Chunks with similarity 0.20 - 0.40
Low relevance	Chunks with similarity < 0.20
Similarity range	Min and max similarity scores
Avg similarity	Mean similarity across all retrieved chunks

Example Output

Retrieval: Moderate
Avg similarity: 0.24

Chunks retrieved: 12
Chunks used: 10
High relevance: 3
Medium relevance: 7
Similarity range: 0.09 - 0.41

What Affects Retrieval Quality

Higher scores when:

• Evidence directly discusses the question topic
• Evidence uses similar terminology to the question
• Sufficient evidence has been uploaded and processed

Lower scores when:

• Evidence is tangentially related
• Technical jargon differs between question and evidence
• Limited evidence available for the topic

What to Do When Retrieval Is Weak

1. Add more relevant evidence - Upload documents that directly address the question
2. Rephrase the question - Use terminology that matches your evidence
3. Ensure embeddings are processed - Check that evidence has been indexed
4. Consider the evidence scope - Some questions may not have strong documentary evidence

---

Schema Validation

What It Measures

Schema validation checks whether the AI's output conforms to the expected structured format. This ensures the analysis can be properly parsed and displayed.

Validation Status

Status	Meaning
Passed	Output matches expected schema
Failed	Output structure is invalid or incomplete

Why Validation Matters

• Ensures consistent analysis format
• Enables structured data extraction
• Supports report generation
• Maintains data integrity

What to Do When Validation Fails

1. Review the raw output - Check for formatting issues
2. Regenerate the analysis - AI occasionally produces malformed output
3. Report persistent issues - Contact support if validation consistently fails

---

Using Quality Metrics in Practice

For Investigators

Before relying on an analysis:

1. Check the Check badge
2. Review any warnings (unsupported claims, coverage gaps)
3. Consider retrieval quality - weak retrieval may indicate missing evidence

When the check reads review needed:

1. Don't discard the analysis entirely - it may still provide useful insights
2. Treat flagged items as hypotheses requiring verification
3. Use gaps as a guide for additional evidence collection

For Reports

Quality metrics can be included in reports to provide transparency:

Analysis Quality Assessment
---------------------------
Automated check: passed
- Faithfulness: 95% (19/20 claims verified)
- Coverage: 100% (all elements addressed)
- Evidence Retrieval: Moderate (avg similarity 0.76)

Note: One claim regarding vendor history could not be
verified against available documentation.

For Quality Assurance

Quality metrics enable systematic review:

• Track metrics across analyses to identify patterns
• Flag analyses below thresholds for supervisor review
• Use metrics to prioritize evidence collection efforts
• Monitor for degradation in AI performance over time

---

Technical Details

Embedding Model

• Model: Amazon Titan Embeddings V2
• Dimensions: 1024
• Similarity metric: Cosine similarity

Similarity Thresholds

Analysis Type	Default Threshold
Question Analysis	0.20
Topic Analysis	0.20
Overall Summary	0.50
Gap Analysis	0.50

Context Limits

• Maximum evidence context: 8,000 tokens
• Chunks are included by similarity rank until limit reached
• Excluded chunks are logged for audit purposes

Database Fields

Quality metrics are stored in the analysis table:

Field	Type	Description
quality_confidence	enum	Stored check result (see below)
faithfulness_score	float	0-1 score for claim verification
faithfulness_details	jsonb	Claim-by-claim breakdown
coverage_score	float	0-1 score for question coverage
coverage_details	jsonb	Element-by-element breakdown
retrieval_stats	jsonb	Detailed retrieval statistics
validation_passed	boolean	Schema validation result

quality_confidence stores one of four values for historical reasons: established and probable display as Check: passed, possible and insufficient as Check: review needed, and an empty value, or possible before faithfulness and coverage have run, as Check: not yet run. The stored values are not evidence ratings.

---

Frequently Asked Questions

Why is my faithfulness score low even though the evidence supports the claims?

The faithfulness check verifies claims against your full evidence record, not only the passages retrieved for generation — so a claim supported anywhere in your evidence should verify. If the score still looks wrong, try:

• Confirming the supporting evidence finished processing (text extracted from uploads)
• Checking whether the claim goes beyond what the evidence states or clearly implies
• Adding evidence that directly addresses the claims

Why does coverage show gaps when I think everything was addressed?

Coverage is evaluated against a structured breakdown of the question. The AI may have addressed topics implicitly rather than explicitly. Review the specific gaps listed - sometimes rewording the question or providing direction helps.

What's a "good" retrieval similarity score?

• > 0.35: Strong - evidence highly relevant
• 0.15-0.35: Moderate - evidence reasonably relevant
• < 0.15: Weak - limited direct relevance

These bands are calibrated for the embedding model Inqura uses (Amazon Titan Text Embeddings V2); relevant evidence typically scores in the 0.15-0.40 range, so a 0.30 is a solid match, not a weak one. Scores vary based on evidence type and question complexity. Technical investigations may naturally have lower scores than straightforward document reviews.

Should I regenerate every analysis with low quality scores?

Not necessarily. Consider:

• Is the analysis still useful despite limitations?
• Can you add evidence to address gaps?
• Would human review suffice for flagged items?

Regeneration helps when: new evidence is available, the question can be rephrased, or you can provide specific direction to address gaps.

How do quality metrics affect the final report?

Quality metrics are informational - they help you understand AI reliability but don't prevent report generation. You can:

• Include quality summaries in reports for transparency
• Use metrics to prioritize which analyses need human review
• Note limitations based on quality findings