What's this episode about?
Does your hiring test actually work? A test can give you the same answer every time and still not predict the job. This episode explains reliability and validity in plain language using the NBA Draft Combine and Kawhi Leonard's 2011 draft, covers which hiring methods research says predict performance best, and explains why the Supreme Court of Canada's Meiorin decision means a test that screens people out has to be tied to the real job.
Key takeaways
Reliability means a test gives the same answer twice. Validity means it predicts the job. A test can be very reliable and still not valid.
In a study of six years of NBA Draft Combine results, size and length measures predicted NBA performance best, and the flashier athletic drills didn't stand out.
The Myers & Briggs Foundation says the MBTI is “not designed to be used for hiring purposes.”
Structured interviews ranked at the top of Sackett and colleagues' 2022 review of hiring methods, with job knowledge tests and work samples also near the top.
Under Canadian human rights law, a standard that screens people out has to be tied to the real job (Meiorin, 1999).
Full transcript
Welcome back to Hiring Without Regret. I'm Ashley Wesley.
Last episode was about which personality traits to look for. Today's question comes right after that: how do you know your test is telling you the truth?
Here's the call. A hiring test that gives you the same answer every time isn't automatically a test that works. Consistent and correct are two different things... and a lot of hiring tools only check the first one.
Two words today. Reliability and validity. Stay with me, this is the fun version.
Reliability means the test gives you the same answer twice. Step on the bathroom scale, step on it again, same number. Reliable.
Validity means the test predicts the thing you actually care about. Your bathroom scale is very reliable. It still can't tell you if someone can sell.
So let's go to the NBA Draft Combine. Every spring, the top prospects get measured, timed and poked like a used car on the lot. Height. Wingspan. Hands. Sprints, agility drills, the vertical leap.
In 2011, a quiet kid from San Diego State showed up. Kawhi Leonard. Bleacher Report's notes from that Combine said he had the biggest hands in the draft... and that he looked tense and uncomfortable in the drills. Scouts had questions about his jump shot.
Fourteen teams passed. He went fifteenth. He now has two Finals MVPs, including the one that brought a parade to Toronto in 2019.
So was the Combine wrong? Not exactly. And this is where it gets useful.
In 2018, a research team led by Masaru Teramoto lined up six years of Combine results against how those players actually performed in the NBA. The measures that predicted best were size and length... height, reach, wingspan, hands. Upper-body strength came next. The highlight-reel stuff, the vertical, the sprint, the agility drill, didn't stand out as predictors.
Think about that. The tape measure turned out to be the useful test. The drills that look great on TV told teams less. And nothing on that sheet measured work ethic, the thing scouts kept mentioning about Kawhi.
Now you're the GM. Except your Combine is a free online personality quiz somebody's cousin swears by.
Even the foundation behind the Myers-Briggs says, and I'm quoting, it's "not designed to be used for hiring purposes." It doesn't measure ability. I respect the honesty.
So what does predict? In 2022, Paul Sackett and his colleagues reran the big numbers on hiring methods. Structured interviews came out on top... same questions, same scoring, tied to the job. Job knowledge tests and work samples also ranked near the top. None of them is a crystal ball. But they beat gut feel, and they definitely beat a quiz built for a team-building retreat.
One more reason this matters, and it isn't about performance. It's the law.
In 1999, the Supreme Court of Canada decided the case of Tawney Meiorin, a forest firefighter in British Columbia. She'd done the job well for three years. Then the province brought in a fitness test: run two and a half kilometres in eleven minutes. After four tries, she missed by about forty-nine seconds, and she was let go.
The Court found the government hadn't shown that standard was reasonably necessary to do the job. She won. And employers across Canada got a clear test: if a standard screens people out, it had better be tied to the real work.
The sales version? "Must be outgoing." "Must look the part." If you can't connect it to the job, it isn't a requirement. It's a liability.
So, Monday. Three moves.
One. List every step in your hiring process. Next to each one, write the job behaviour it's supposed to predict. If you can't, cut it.
Two. Check reliability the cheap way. Have two people score the same candidate separately, then compare. If they're miles apart, your interview isn't measuring the candidate. It's measuring the interviewer.
Three. Lead with a structured interview and a short work sample built from the real job. For a sales role, that might be a live call with a real objection from your own customers.
Next week: skills. Does the ten thousand hour rule actually hold up, or did we all just love a good book?
That's the segment. Measure what matters... and leave the vertical leap to the scouts.
Quick note: this is general information, not legal advice. Talk to an employment lawyer before you change how you test or hire.
Frequently asked questions
What is the difference between reliability and validity in hiring?
Reliability is consistency: the test gives the same result when you repeat it, or when two people score the same candidate. Validity is whether the result predicts how someone will actually perform in the job. You need both, and a test can be very reliable while predicting very little.
Can I use a free personality quiz to hire salespeople?
Not one that wasn't built for hiring. The Myers & Briggs Foundation says the MBTI is not designed for hiring and doesn't measure ability. If you use an assessment, use one validated for selection, treat it as one input, and tie it to the job.
Which hiring methods predict job performance best?
In Sackett and colleagues' 2022 review, structured interviews came out on top, with job knowledge tests and work samples also ranking near the top. None of them is perfect, so pairing a structured interview with a work sample built from the real job is a sensible place to start.
Work with us
Not sure your interview is measuring the candidate rather than the interviewer? We help Ontario owners pick the right sales hire. Our Final Candidate Review validates the match before you make the offer, using documented, job-related evidence, and it’s covered by our handshake guarantee.
References
Bleacher Report. (2011, May). 2011 NBA draft combine day 2: Athletic tests, measurement notes and analysis. https://bleacherreport.com/articles/707832-2011-nba-draft-combine-day-2-athletic-tests-measurement-notes-and-analysis
British Columbia (Public Service Employee Relations Commission) v. BCGSEU, 1999 SCC 48, [1999] 3 S.C.R. 3. https://www.canlii.org/en/ca/scc/doc/1999/1999canlii652/1999canlii652.html
Kawhi Leonard. (n.d.). In Wikipedia. https://en.wikipedia.org/wiki/Kawhi_Leonard
Myers & Briggs Foundation. (n.d.). Ethical use of the MBTI. https://www.myersbriggs.org/unique-features-of-myers-briggs/ethical-use-of-the-mbti/
Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. https://doi.org/10.1037/apl0000994
Teramoto, M., Cross, C. L., Rieger, R. H., Maak, T. G., & Willick, S. E. (2018). Predictive validity of National Basketball Association Draft Combine on future performance. Journal of Strength and Conditioning Research, 32(2), 396–408. https://doi.org/10.1519/JSC.0000000000001798
Disclaimer: This episode and page provide general HR and business information, not legal advice. Employment law turns on the specific facts and changes over time. Before acting on anything here, confirm it with an Ontario employment lawyer.
