See what your evaluation set is hiding.
Thousands of model responses and human scores form one living map. AnnotRift finds the contested ratings, ambiguous rules, and unreliable data inside it — and helps your team resolve them.
Walk one contested rating from the full map to a resolved decision. Click any stage — everything is visible, nothing waits on scrolling.
Anomaly region
Two scoring standards collide on the same kind of answer. Red marks where reviewers and the rubric disagree; lime marks where the model output itself looks anomalous.
Inside a Disagreement Pocket of 38 records, three reviewers gave the same model response three different scores.
"Summarize the refund policy for orders over $500."
"Orders above $500 qualify for expedited refunds within 5 business days, minus a restocking fee."
Rubric overlay — where the disagreement actually lives.
AI review suggestion · awaiting confirmation
Recommend Borderline → Reject: response asserts a restocking fee not supported by the policy source.
The map recomputes. This one judgment reclassifies the pocket and updates the Gold Dataset and report.
Import your evaluation set. Get back what's wrong with it.
AnnotRift ingests model responses, human ratings, labels, the rubric, and review notes — then works the data through a fixed reliability pass. Every AI suggestion shows its rules, historical cases, and evidence; a human confirms the final call.
Check rules
Ratings and labels tested against the rubric.
Surface rifts
Find reviewer-to-reviewer disagreement.
Spot confusion
Isolate easily-confused labels and criteria.
Flag bad data
Duplicate, contaminated, thin samples.
Queue experts
Route hard cases for human review.
Ship gold
Human-verified Gold Dataset + report.
Where reviewers apply the rubric differently.
Every reviewer is a track. When their scores for the same response cross, that's not noise — it's a rift worth resolving. AnnotRift tells you whether it's an error, an ambiguous rule, missing context, or a fair judgment call.
- Crossing lines mark contested ratings
- Classified: error · ambiguity · missing context · subjective
- Send recurring rifts straight to Rubric Studio
Reviewer tracks · summarization batch
Turn the rubric everyone argues about into rules you can test.
AnnotRift breaks your scoring guide into Definition, Positive Evidence, Negative Evidence, Exceptions, Priority, and Examples. Edit a rule and the historical disagreement recomputes instantly — so you fix the rubric, not just the symptom.
- Ambiguity Lens highlights vague, overlapping rules
- Test a rule change against past disputes
- Every version records owner and affected range
"Accurate" = every claim is supported by the provided source.
"Minor omissions are acceptable" — no threshold for "minor".
Any unsupported policy or fee → Reject.
Find what makes your AI evaluations unreliable.
Point AnnotRift at your evaluation set and get a traceable quality report in minutes — every finding backed by data and rules, every decision confirmed by a human.