Raising the Standard: Quality Assurance in Human and AI Screening

Artificial Intelligence (AI) is transforming recruitment, but with innovation comes concern: can machines really be fair? At our recent breakfast event, we were able to tackle this head on when talking about our new approach to AI video interview (VI) scoring. During the session, we were able to reassure attendees who had concerns about how we will be introducing AI into our processes. To us, assuring the quality of screening, whether human or machine, is a non-negotiable, and we are putting our robust quality assurance (QA) process into place for our new AI scoring as well. 

Our vision is simple: that every candidate should get equal opportunity to demonstrate their potential. To support this, screening should be consistent, fair, and free from bias; whether the screener is human or machine. In a climate increasingly shaped by high volumes and rapid decision-making, we believe that every assessment should be held to the same high standard. 

That’s why we’ve created principles to support a robust Quality Assurance (QA) process, one that doesn’t just catch errors but actively supports better decision-making. It’s a system we’ve refined with our human screeners, and now we’re using those principles to support our roll out of AI screening too.  

Our human process

I’ve worked in environments without a robust quality assurance process and I’ve seen good, experienced screeners make the wrong decision whilst screening as a result. For instance, a previous client was concerned about one of their top screeners and wanted my advice as their trusted partner about what was happening. The screener had been underscoring a group of candidates during a high-volume campaign and after scrutinising their scores and rationale, it was clear that they were not being consistent in their approach. After taking a deep dive into why, I found the screening guidelines took a high cognitive load which meant they required significant mental effort to apply consistently. With something personal going on outside of work, the screener was struggling to stay aligned to the guidelines. 

This highlighted the importance of having a rigorous and well-defined quality assurance process. Ensuring that good people continue to get it right, providing a consistent and fair screen we see as an important part of a candidate’s recruitment journey. 

At Amberjack, we have a QA process that we are proud of. It’s independent, fair, and scalable; not just catching mistakes once they have happened but preventing them from happening in the first place. Supporting our screeners to be successful and improving consistency and alignment along the way. 

Our current QA process is grounded in some key principles:

• Resource the right team: we work with a focused group of assessors, only as many as needed to meet the client’s requirements. This approach supports consistency across the screening cohort, encourages shared learning, and helps maintain high standards throughout the campaign.
• Set up for success: we don’t drop screeners in at the deep end. Every assessor receives foundational training aligned to BPS best practice, covering what video interviews are and how to assess them effectively. This ensures a consistent, fair approach from the very start.
• Utilise our expertise: before any video interview goes live, both the questions and scoring guidance are reviewed by our assessment and QA teams. This dual-layer review helps us catch ambiguity early, clarify expectations, and ensure screeners have the tools they need to assess fairly and consistently.
• Collaborate with the client: before training our screeners, we hold a dedicated call with the client to gather insights, context, and campaign-specific nuances. This ensures our assessors are aligned with the client’s expectations and that every aspect of successful delivery is covered.
• Calibrate for consistency: before screening begins, we run a calibration session to ensure all assessors understand the client, the campaign, and the scoring criteria. We live score real candidate responses together, aligning on expectations and approach. Client stakeholders are encouraged to join, helping build a shared understanding from the start.
• Maintain the quality: before moving to volume screening, all assessors must meet a minimum alignment threshold with our QA team. But alignment doesn’t stop there; ongoing QA ensures screeners continue to assess consistently and fairly throughout the campaign.
• Feedback to drive fairness: whether at the move to volume stage, or during ongoing QA, screeners receive personalised feedback on their screening. If concerns arise, they repeat the QA process until alignment is achieved; ensuring every candidate is assessed fairly and consistently.

We apply these same principles to our telephone interview screening, ensuring all candidates get a consistent experience. The principles could also be used to assure assessment exercise, interview and long answer question scoring – it’s all about ensuing those doing the scoring are doing so in a consistent and fair way. 

We are so proud of our QA function that we have packaged up the service for clients who do their own screening, so they can also benefit from this value add and ensure their screening teams are aligned and consistent.

Our new AI screening

You might ask why we are bringing in AI scoring at all – following a bumper year of growth we screened approximately 50% more video interviews in the 2024/25 campaign year than the previous year. We want to ensure we continue to offer our clients the same high standards of screening that we have previously, and AI scoring, implemented correctly and with a human wrap-around, is going to support us to do this. 

Now, we’re bringing AI into the screening mix, applying the same rigorous QA process we apply to humans. AI promises consistency, speed, and scalability. But without QA, it’s just a black box and that’s why we’re treating our AI like a human screener. Here’s our AI principles which shape how we’re doing it, alongside the relevant principles from our human scoring approach:

• Train for context: our AI isn’t generic. It’s trained specifically on the client’s campaign and individual video interview so it understands their specific scoring criteria and context. 
• Utilising the data: our AI scoring is based on the transcript of the candidate’s VI response, rather than the live recording – our AI will tell us how confident it is that it transcribed the response properly, as well as how confident it is that it scored the transcription effectively. If the AI provides a lower confidence rating for either of those things, that candidate will be flagged for human review, ensuring no candidate is disadvantaged by ambiguity.
• Check for consistency: we apply the same QA checklist to AI decisions as we do to human screeners, ensuring every assessment meets our high standards for fairness, consistency, and transparency. But QA isn’t a one-time exercise. Every new campaign, stream, or level receives a fresh review before launch, and ongoing QA is carried out throughout delivery to maintain alignment and uphold quality.
• Monitor for fairness: we know AI can reflect bias in its scoring, which is why our QA process includes regular checks and ongoing monitoring to prevent unfair outcomes. We also handle all candidate data with strict confidentiality and in full compliance with GDPR.
• Respect the candidate: candidates need to opt in for the AI scoring; candidates are given the choice to opt into AI scoring. Those who prefer a different approach are assessed by our trained human screeners, in line with our usual rigorous QA principles. This ensures every candidate is treated fairly, with transparency and respect for their preferences.

Why it matters

Research shows that candidates tend to prefer human screening (Lavanchy et al., 2022, Nobel et al., 2021). But is that preference rooted in trust, or a lack of it? Headline-grabbers like Netflix documentaries on algorithmic bias andAmazon’s CV screening controversy have understandably raised concerns about fairness and transparency in machine-led hiring.

That’s why we believe it’s not enough to simply say “we use AI.” We must show how we use it and more importantly, how we assure its quality. When we share the steps we’re taking; our campaign-specific training, confidence rating checks, bias monitoring, and opt-in approach, we’re confident that trust can be built. Not just in the technology, but in the integrity of the process behind it.

We’re not just rolling out AI scoring. We’re rolling out AI scoring with care, transparency, and the same rigorous standards we apply to our human screeners. Because whether it’s a person or a machine, quality assurance is what makes screening successful, and what makes candidates feel seen, heard, and fairly assessed.