Fairness, bias and discrimination
In brief: fairness and ADM
Fairness means organisations should handle personal information in ways people reasonably expect. They should not use it in ways that have unjustified adverse effects on people.
Organisations are responsible for the algorithms and tools they use to process personal information, even if they didn’t build the software or create the algorithm. When obtaining and using a tool, organisations could consider:
- conducting trials to ensure that results limit bias; and
- requesting information from developers on:
- the demographic groups a model was trained on
- whether any underlying bias has been detected or may emerge; and
- any algorithmic fairness testing that it has conducted.
Further reading
- Fairness in AI
- Draft guidance on Automated decision-making, including profiling
- AI in Recruitment Outcomes Report
Other resources
Equality and Human Rights Commission Guidance on:
Our findings about fairness
Our findings about meaningful human involvement, transparency and safeguards (covered above) all point to a lack of fairness. This is because employers may not be processing information as people reasonably expect. Candidates may, therefore, be subject to a power imbalance where they have limited access to the rights and safeguards designed to protect them from potentially negative consequences. Our evidence also suggests that employers aren’t fully:
- considering the specific circumstances and context of the processing; or
- putting appropriate technical and organisational measures in place to ensure that processing is fair or transparent.
Case study
An employer conducts digital chat-based assessments of candidates in the initial phase of recruitment. The tool scores candidates based on their responses to set questions, producing a result based on predefined score requirements. It gives each candidate a role suitability score and personality profile based on a traffic light system of ‘yes / green’ ‘no / red’ and ‘maybe / amber’. A green score does not guarantee an interview. Likewise, amber or red does not necessarily mean a candidate won’t get an interview. However, there’s no evidence that the employer reviews all the tool’s decisions periodically to check for accuracy, fairness or bias, nor does it track how often a human reviewer offers a job to a person who has been given a red score.
This indicates that people are likely to be having their information processed unfairly. Where employers have failed to ensure meaningful human involvement has been applied to each decision and they have also failed to implement the safeguards under the ADM provisions, they have also breached these requirements of UK GDPR. This further suggests that their processing is not fair.
However, as evidence of good practice, many employers reported that they:
- asked developers about their own bias testing as part of procurement;
- conducted trials to check that results limited bias; and
- engaged in ongoing monitoring of their tools and outputs, such as dashboards that provided ongoing bias monitoring and monthly bias reviews.
We didn’t see evidence of hiring managers using unapproved tools that hadn’t been risk-assessed. This indicates a growth in positive practice in line with the recommendations made in our 2024 AI in recruitment report 33.
Some employers had taken steps to configure their tools. For example, hiring managers could only see relevant information about candidates or required outputs, with unneeded functionality disabled. This reduced the risk of hiring managers:
- introducing their bias into the recruitment process; or
- using unapproved functionality that hadn’t been risk-assessed.
We also found that employers typically provided reasonable adjustments for people with disabilities who weren’t able to access automated recruitment methods. This often involved a phone interview.
We asked employers about their use of special category data. Most of the recruitment tools being used had an optional demographic information question at the end of the process. Answering this question would involve providing special category data. Employers didn’t use this information to make recruitment decisions. Instead, it was collected it to help monitor for bias and discrimination in the system.
We didn’t fully explore how far employers were using personal information to further train tools or their underlying systems by defining a ‘good’ candidate. We also didn’t fully explore how accurate the tools were. However, we covered both of these areas in our AI in recruitment report.
Discussion
We recognise that many automated recruitment tool providers market their tools as fairer and less biased than human recruiters. They often point to the well-documented existence of bias and discrimination in the traditional hiring process. Producing a fairer hiring process is one of the key ambitions of this growing industry 34.
In our public perception research, participants agreed that ADM could create fairer decisions because it could make more consistent choices with less bias. However, participants were also concerned that ADM could introduce or perpetuate bias, though they accepted that human decision-making could do the same. Academic research often highlights the competing, diverging and subjective perceptions of fairness among both job applicants and HR practitioners 35.
Example
A company uses a tool to make automated decisions about who to hire for a specific role. The tool was trained using historical data based on the personal information of previous successful hires for the role. It bases its decisions on these exact criteria. All previous hires were men in their mid- to late 40s. All other gender and age groups could face an unfair disadvantage because of the training data provided to the tool.
Our public perceptions research also highlighted how attitudes towards the use of automation varied, depending on the type and impact. For example, people generally accepted using automation to filter CVs. However, they viewed the use of more profiling-based automation, such as online behavioural assessments, more negatively. This points, again, to the importance of meeting people’s reasonable expectations and addressing power imbalances.
In our roundtable discussions:
- civil society stakeholders highlighted the need for scientific validity and fairness in ADM systems, stressing the importance of:
- validating systems’ accuracy; and
- justifying the relevance of the data points used in decision-making;
- all stakeholders highlighted the importance of monitoring and regular feedback to ensure ADM does not lead to discrimination;
- stakeholders agreed that employers need to be able to understand how their systems work; and
- civil society and trade union stakeholders argued that employers should not focus on whether specific people have been treated fairly. Instead, monitoring and reporting based on aggregated data should be standard practice.
Expectations
We expect to see fairer processes introduced in line with the expectations set out above about meaningful human involvement, transparency and safeguards. We expect employers to review their practices and ensure that they are treating candidates fairly and that their processes are in line with what candidates reasonably expect.
For processing to be fair and compliant with the ADM provisions, candidates must be able to contest a decision in a timely manner.
We expect to see continued evidence that employers are asking developers about their own bias testing and conducting their own trials to check that results limit bias. Employers should also transparently provide information on the accuracy and performance of the tools they are using.