Why I care about this
I’m excited by AI that can help people think, but not by systems that make certainty feel effortless. When an answer depends on changing, incomplete, or conflicting information, the useful question is not only “what is the answer?” It is “what evidence supports it, what pushes against it, and how much confidence does that earn?”
This work explores that question as a technical system: externalize factual knowledge, keep the reasoning trace inspectable, and let contradiction reduce confidence rather than disappear behind fluent text.
Research state
Ongoing
Core concern
Overconfidence
Output
Auditable proof
The logic
The runtime represents a response as a temporary claim graph. Each evidence attachment receives a truth score from support, contradiction, and freshness; it is then accepted, contested, rejected, or dropped when its lifetime ends.
T(a) = s(a) − w₍c₎c(a) + w₍f₎f(a)
An attachment’s score combines support, weighted contradiction, and freshness.
p = min(1, w₍c₎(r + λq))
Contradiction pressure grows with rejected evidence r and contested evidence q.
g = clip(gₘᵢₙ, gₘₐₓ, wₐa + wₛt − wₚp)
The gate weighs accepted evidence a, normalized truth score t, and contradiction pressure p.
c′ = c × g
Final confidence c′ is the original confidence c after evidence earns—or loses—the right to support it.
What has been built
The implementation is a working retrieval-and-verification runtime, not only a diagram or idea.
01
Evidence first
A query is expanded into supporting and contradiction probes. Retrieved material becomes temporary evidence, not permanent factual memory.
02
Claims, not just context
The system compiles evidence into a claim graph, then asks what supports each claim, what challenges it, and what is still unresolved.
03
Confidence must earn its place
A confidence gate discounts an answer when rejected or contested evidence accumulates. The output includes the trace behind that decision.
04
Make the result falsifiable
Every run records its configuration, evidence, claim graph, critiques, confidence diagnostics, and evaluation artifact.
Current internal evidence
On the current 1,500-task internal proxy benchmark across two corpora and four replications, the strict CNER variant improved calibration over the no-BT-TAM ablation while preserving score.
| Variant | Score | ECE ↓ | Brier ↓ |
|---|---|---|---|
| No BT-TAM | 91.61 | 0.359 | 0.130 |
| CNER strict | 91.70 | 0.332 | 0.111 |
This is internal evidence, not a general performance claim. The benchmark’s external-proxy task construction, simplified baselines, and absence of independent replication remain material limits.
What gives me energy
The work makes a hard question concrete: confidence should respond to evidence quality, disagreement, and what the system cannot settle. That feels more useful than asking a model to sound certain more often.
What has to happen next
Freeze one external benchmark, compare against canonical RAG baselines, test on real retrieval corpora, and invite independent replication. Only then can the calibration result become a broader research claim.
Let’s make reliable AI less mysterious
I’d love to work with researchers and engineers in RAG evaluation, calibration, AI safety, and decision support who care about making evidence and uncertainty visible.
Start a research conversation