Track C
Assure
How do we know each output is good enough, at runtime, before it ships?
Orchestration manages where a request runs. Quality orchestration manages whether the result is good enough, as a control loop. Quality is treated as a managed service level.
Why it matters
A cheap answer that ships unchecked is not a service an auditor can accept.
Directions
1. Quality contracts: acceptable quality, error cost, abstention rule.
Quality contracts: acceptable quality, error cost, abstention rule.
2. Runtime estimation: verifiers, self-consistency, cross-provider agreement, graph consistency.
Runtime estimation: verifiers, self-consistency, cross-provider agreement, graph consistency.
3. Cascades: cheap model first, escalate below contract, calibrated methods.
Cascades: cheap model first, escalate below contract, calibrated methods.
4. Judge reliability, with human-adjudicated samples when an LLM is used as a judge.
Judge reliability, with human-adjudicated samples when an LLM is used as a judge.
5. Drift monitoring and a closed learning loop.
Drift monitoring and a closed learning loop.
End-to-end pipeline (conceptual)
Flow: Extract, Ground, Route, Assure, Learn. Learn feeds back into Route and Assure. Each step links to a program page.
Feedback: Learn → Route and Assure.
View as table
| Step | Order | Opens |
|---|---|---|
| Extract | 1 | /research/ground/ |
| Ground | 2 | /research/ground/ |
| Route | 3 | /research/route/ |
| Assure | 4 | /research/assure/ |
| Learn | 5 | /research/assure/ |
| Learn to Route / Assure | Feedback | /research/assure/ |
Quality cascade (conceptual)
Decision tree: cheap model, then a quality estimate, then pass, escalate to a stronger model, or abstain. Conceptual. No measured rates.
View as table
| Node | Next |
|---|---|
| Cheap model | Quality estimate |
| Quality estimate | Pass, Escalate, or Abstain |
Planned metrics
Values show Pending until measurements exist.
Cost per correct answer
Pending
Spend required for an answer that meets the quality contract
Abstention-adjusted accuracy
Pending
Accuracy after accounting for allowed abstention
Calibration error
Pending
Gap between estimated quality and observed quality
Escalation rate
Pending
Share of requests sent to a stronger model or reviewer
Time to detect drift
Pending
Delay between a quality shift and a raised alert
Connects to
The lab designs and studies five research objects. They are planned artifacts on the agenda, not products with published measurements.
Work with HOPN Lab
Enterprise buyers, academic collaborators, students and funders in Europe can write to us.