Quick answer
For your first Calibration task, do not change many settings: run defaults once, then adjust one item at a time. Calibration fixes the problem of two managers scoring the same ticket differently.
Why it matters
Change one variable at a time so you know which setting matters.
Use 5–10 samples, score independently before discussion.
Step by step
- Define QA goals and observable behaviors before building a giant scorecard.
- Stratify samples by shift, channel and risk so they represent real problems.
- Calibrate scoring boundaries before ranking or performance decisions.
- Attribute issues to knowledge, process, system or people before choosing fixes.
Common mistakes
- Scoring without calibration makes managers incomparable.
- Sampling only easy tickets biases the result.
- Using QA as punishment makes agents hide context.
- Looking only at CSAT while ignoring resolution and rework.
Checklist
- Scorecard has behavioral anchors.
- Sampling has stratification rules.
- Calibration records disputed samples.
- Issues have owners and review dates.
Key facts
- Use 5–10 samples, score independently before discussion.
- Resolve disputes against scorecard anchors, not persuasion.
- Update scorecard version and example library after calibration.
FAQ
How do I start with Calibration?
For your first Calibration task, do not change many settings: run defaults once, then adjust one item at a time. Calibration fixes the problem of two managers scoring the same ticket differently.
How long does it take?
Usually 1-3 minutes; for intermittent issues, run 30 seconds and retest elsewhere.
Is data uploaded?
Runs locally and does not upload video/audio; extensions may still read the page.
Source: MaestroQA calibration