How to Rank Candidates Fairly: A 6-Step Method With a Comparison Matrix
A six-step method to rank candidates on the same evidence, the main ranking methods compared, a candidate comparison matrix template, a worked example and the fairness checks to run before you shortlist.
By Beatview Team · Published · Updated · 7 min read
Key takeaways
- Rank candidates on the same job-related criteria, measured from the same evidence sources, combined by a rule fixed before any scores are seen.
- A must-have gate followed by a weighted sum, reviewed in bands rather than exact positions, suits most roles.
- Only compare candidates who have completed the same stages; an unfinished interview is not a low score.
- Treat near-equal totals as ties and decide on the competency the role depends on most.
- Before shortlisting, check that scores were submitted independently, interviewers are calibrated and every decision has a written reason.
To rank candidates fairly, score every candidate on the same job-related criteria using the same sources of evidence, combine the scores with a rule you fix before you see any results, compare only candidates who have completed the same stages, and read the evidence behind the top of the list before you decide. A ranking is a sorting aid: it makes comparison consistent and the gaps visible. It is not the decision. This guide gives you a six-step method, a candidate comparison matrix you can copy, a worked example, and the checks to run before a shortlist leaves your desk.
What candidate ranking is — and what it is not
Candidate ranking means putting applicants in order using the same evidence for everyone. That evidence usually comes from three places: the application (resume or CV and screening answers), a structured interview, and — for some roles — a work sample or assessment. A fair ranking is not the absence of judgment; it is judgment applied the same way to every candidate, with the reasoning written down.
What ranking cannot do: an ordered list does not make a decision fair on its own, and no single number predicts job performance. Two candidates with the same total can be very different hires. The ranking's job is to tell you where to look.
How to rank candidates in 6 steps
Choose four to six competencies from the outcomes the role must deliver. Remove anything you cannot tie to the work, such as "culture fit" or pedigree.
Decide where each criterion is measured: the resume, a specific interview question, a work sample. Every candidate must be measured on the same sources.
Write the weights (for example 40/30/30) and how you will combine them before you look at any scores, so the rule cannot drift toward a favourite.
Each assessor rates on a 1–5 scale with written anchors and an evidence note, before any group discussion. The interview scorecard template has the format.
Calculate the composite score, but only rank candidates who have completed the same components against each other.
Open the evidence for the top candidates, settle ties on the competency the role depends on, and write one sentence on why each shortlisted candidate goes forward.
Candidate ranking methods compared
| Method | How it works | Best for | Watch out for |
|---|---|---|---|
| Simple average | Mean of the criterion scores | Small searches; when criteria matter equally | Gives trivial criteria the same weight as critical ones |
| Weighted sum | Sum of (score × weight), weights total 100% | Most roles with clear priorities | Weights must be set before scoring and kept identical for everyone |
| Must-have gate, then weighted sum | Anyone below the bar on a must-have is excluded; the rest are ranked | Roles with genuine non-negotiables (licence, right to work) | Only use gates for objective, job-necessary requirements |
| Score bands | Group candidates into bands (for example 4.0+, 3.5–3.9) and treat each band as equal | High-volume hiring where exact ranks are false precision | Define bands in advance; decide within a band on evidence |
| Pairwise comparison | Compare candidates two at a time on each criterion | Small final shortlists (under ten) | Time-consuming; randomize the order of pairs |
For most teams, a must-have gate followed by a weighted sum — reviewed in bands rather than exact positions — is the best balance of rigour and effort.
Candidate comparison matrix (template)
A comparison matrix puts candidates in columns and criteria in rows, so the shape of each candidate's evidence is visible at a glance. Copy this layout:
| Criterion (weight) | Evidence source | Candidate A | Candidate B | Candidate C |
|---|---|---|---|---|
| Must-have: right to work | Application question | Yes | Yes | Yes |
| Role knowledge (35%) | Resume + interview Q1 | |||
| Client communication (25%) | Interview Q2–Q3 | |||
| Problem solving (25%) | Interview Q4 or work sample | |||
| Motivation for the role (15%) | Interview Q5 | |||
| Weighted score | ||||
| One-line reason |
Worked example: ranking four candidates
The candidates below are synthetic, created to show the arithmetic — not customer data or benchmarks. The role is a client-facing coordinator where client communication decides success.
| Candidate | Resume score | Interview score | Overall | How to read it |
|---|---|---|---|---|
| A | 82 | 74 | 78 | Strong stated background, thinner interview evidence on the main competency |
| B | 68 | 88 | 78 | Same overall as A from the opposite direction: modest resume, specific interview answers |
| C | 91 | Not completed | Resume only | Cannot be placed against A or B yet — chase the interview |
| D | 60 | 62 | 61 | Consistently below the others on the evidence available |
A and B tie at 78, which is exactly why a total is a sorting aid rather than a decision. Open both interviews and read the answers on client communication. If the role lives or dies on that competency, B goes forward. C is not "below" anyone — C is at a different point in the process.
Handling missing evidence and ties
| Situation | What to do |
|---|---|
| Resume screened, interview not yet sent | Compare only with candidates at the same stage |
| Interview sent but not completed | Follow up or close the invitation; an unfinished interview is not a low score |
| Resume could not be read | Review the file by hand instead of ranking on a missing score |
| Two candidates within a few points | Treat them as tied and decide on the competency the role depends on |
| A strong candidate just under your cut line | Check the evidence; if you move the line, move it for everyone and record why |
Fairness checks before you finalize a shortlist
- Every candidate was measured on the same evidence. No one got an extra question, a longer time limit or a friendlier interviewer.
- Scores were submitted before discussion. Otherwise the ranking records the loudest opinion.
- Interviewers are calibrated. If one interviewer's average is a full point below everyone else's, discuss anchors with them before trusting their ratings.
- Selection rates look reasonable across groups. In the US, the EEOC's Uniform Guidelines use a "four-fifths" rule of thumb: a selection rate for any group below 80% of the highest group's rate is generally treated as evidence of adverse impact worth investigating. Where you have lawful access to demographic data, check it at each stage.
- Every shortlisted and rejected candidate has a written reason tied to a named criterion.
For the broader practice, see our guide to reducing bias in hiring.
Ranking candidates after resume screening vs after interviews
Early in the process you rank on thinner evidence, so the goal is different: decide who is worth interviewing, not who to hire. Rank in broad bands and keep the cut line generous. The detailed method for that stage is in how to shortlist candidates. After structured interviews you have comparable evidence on the competencies that matter, and the weighted comparison above becomes meaningful.
How Beatview ranks candidates
Beatview produces two kinds of evidence. The resume score compares the parsed application with the role and shows the skills and experience behind it. The interview score comes from a structured video interview in which every invited candidate answers the same timed questions, scored on relevance, clarity, depth of knowledge and overall, with the answer beside each score. The overall match score is the plain average of the resume and interview scores for the components a candidate has completed — there are no configurable weights or hidden adjustments. If you want weighting, do it on your own comparison matrix and keep it separate from the product score. An optional Work Style assessment adds its own six dimensions and flags as separate evidence. Recruiters read the evidence and decide who goes forward; the event log records how each candidate reached their position. See resume screening and AI interviews.
Frequently asked questions
How do you rank candidates after an interview?
Rate each candidate on the same competencies with anchored 1–5 scales, combine the ratings with weights agreed before interviewing, then read the evidence behind the top candidates and settle close calls on the competency that matters most for the role.
What is a candidate comparison matrix?
A table with candidates in columns and job criteria in rows, showing each candidate’s rating, the evidence source and a weighted total. It makes the shape of each candidate’s strengths visible, not just the total.
Should I use weighted scores to compare candidates?
Yes, when some criteria matter more than others. Set the weights before scoring, keep them identical for every candidate and treat the weighted total as a sorting aid rather than the decision.
What is the four-fifths rule in hiring?
A US rule of thumb from the Uniform Guidelines on Employee Selection Procedures: if a group’s selection rate is less than 80% of the rate for the highest-selected group, that is generally regarded as evidence of adverse impact that should be investigated.
Does a higher rank mean a better hire?
No. It means more of the evidence you asked for lined up. Read the evidence before recommending anyone, and let a person make the final decision.
Sources
- U.S. Code of Federal Regulations, 29 CFR § 1607.4(D): Uniform Guidelines on Employee Selection Procedures — adverse impact and the "four-fifths rule".
- Sackett, P. R., et al. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology.