The Fairness Innovation Challenge: key findings · first published 3 December 2025
In 2023, the Department for Science, Innovation and Technology (DSIT) ran the Fairness Innovation Challenge, giving over £465,000 in grants to four research teams to tackle bias and discrimination in Artificial Intelligence (AI) systems. The challenge concluded in March 2025 and this report publishes the key findings. The four projects covered recruitment CV screening, financial sentiment models, university learning analytics, and hospital cardiac-arrest prediction — all finding measurable bias in existing AI tools. For small businesses, the most directly useful output is 'Fairground', a free open-source toolkit to test automated CV-screening software for bias, and the Alan Turing Institute's FAID (Fair AI Development) tool for fairness monitoring in AI development pipelines. The report does not change procurement rules or create new legal obligations, but it signals that regulators (the Equality and Human Rights Commission (EHRC) and the Information Commissioner's Office (ICO)) are actively scrutinising AI fairness and that 'data protection by design' is the expected standard from the design stage onwards.
WHO THIS APPLIES TO
THE ENKII VIEW
For SMEs that use or build AI tools — especially in hiring, finance or ed-tech — this report is an early warning: regulators are watching, bias is provably present in widely-used models, and the ICO is clear that compliance starts at design, not at the privacy notice. The free Fairground and FAID tools lower the cost of self-assessment, making it feasible for a small firm to check its AI hiring or customer-facing tools without expensive consultants. SMEs that adopt these tools now and document their bias-testing process will be better placed as regulatory scrutiny of AI tightens.
1. Download and run the free 'Fairground' open-source toolkit (available on PyPI) to test your automated CV screening tool for racial and gender bias before your next recruitment round.
SMEs using AI-powered CV screening or applicant tracking systems — The Coefficient Systems project proved that widely-used LLMs exhibit measurable racial and gender bias in CV ranking, and the EHRC has signalled active regulatory scrutiny of AI discrimination under the Equality Act 2010. Fairground is explicitly designed for this self-assessment.
2. Adopt the ICO's 'data protection by design' principle from day one: document your design decisions, justify use of personal or special-category data, and build in role-based access controls and explainability features before launch.
SMEs building or integrating AI tools into their products or services — The ICO was explicit that 'data protection compliance starts at the design stage, not at the privacy notice' — retrofitting compliance after deployment is harder and costlier.
3. Use the Alan Turing Institute's FAID tool and published design patterns (available on GitHub at alan-turing-institute/fairness-monitoring) to build fairness traceability, auditability and transparency reports into your AI development pipeline.
SMEs developing AI systems that process demographic or sensitive data — The FAID tool provides free templates and functions for exactly this purpose, and proactive documentation of fairness decisions reduces regulatory risk and demonstrates responsible AI practice to enterprise customers and public-sector buyers.
4. Ask suppliers directly whether their AI tools have been audited for bias, request transparency on how the model works, and check whether the tool's fairness claims are backed by independent testing — not just marketing.
SMEs procuring third-party AI tools (e.g. hiring software, financial analytics, customer-service chatbots) — The market review found many commercial tools are 'black boxes' that 'prevent algorithmic scrutiny,' and all four challenge projects found bias in AI systems already in use. Buyer due diligence is now part of your equalities and data-protection obligations.
5. Before applying reweighting or other techniques that actively favour disadvantaged groups, take legal advice on whether the approach could constitute positive discrimination under UK equalities law.
SMEs applying bias-mitigation techniques to AI systems — The report explicitly flags that approaches 'like reweighing — which actively seek to favour disadvantaged groups — may be interpreted as positive discrimination,' creating potential legal exposure.
6. Read the publicly available 'Fairness Tales' blog series (from Coefficient/Bays Consulting) and the Open University's Framework for Learning Analytics in Higher Education to understand sector-specific bias risks and mitigation approaches relevant to your context.
SMEs in healthcare, financial services, recruitment or education using or building AI — These are free, practice-focused resources produced directly from the funded projects and are intended to support knowledge dissemination beyond the research teams.
Every rule below quotes the official document verbatim.
The Fairness Innovation Challenge provided over £465,000 in grant funding across four projects to address bias and discrimination in AI systems. (Four funded projects (Open University, Alan Turing Institute, King's College London, Coefficient Systems Ltd), concluded March 2025)
“a grant challenge that provided over £465,000 funding to drive the development of novel solutions to address bias and discrimination in AI systems”
All four projects detected some level of bias in the AI systems they evaluated, demonstrating the pervasiveness of the issue across different sectors. (Recruitment, financial services, higher education and healthcare AI systems tested in the challenge)
“Notably, all projects detected some level of bias in the AI systems they evaluated, demonstrating the pervasiveness of this issue across different sectors.”
LLMs tested for CV screening exhibited racial and gender bias; bias varied by role, with one model favouring different ethnicities for different job types. (AI-powered CV screening tools tested by Coefficient Systems Ltd across 4 job roles)
“The team found that the LLMs tested exhibited racial and gender bias when sorting CVs. This bias varied depending on the role being selected for; for example, one model favoured black candidates over other ethnicities for the HR Business Partner while disproportionately favouring Asian candidates for the Senior Software Engineer role.”
Coefficient Systems Ltd developed 'Fairground', a free open-source toolkit to enable developers to test automated CV software and applicant tracking systems for bias and discrimination. (Any organisation using or building AI-based CV screening or applicant tracking systems)
“the team developed 'Fairground,' an open-source technical toolkit to enable developers to test their automated CV software and applicant tracking systems for bias and discrimination.”
The Alan Turing Institute developed the FAID (Fair AI Development) tool to support fairness-related metadata management and transparency throughout machine-learning development pipelines. (AI developers and organisations deploying AI systems, particularly in financial services)
“the team developed their FAID (Fair AI Development) tool. The FAID tool is intended to support better fairness-related metadata management throughout the ML pipeline to improve information sharing among stakeholders involved in the development process.”
The ICO states that data protection compliance must start at the design stage, not at the privacy notice. (All organisations designing or deploying AI systems that process personal data)
“Our message is that data protection compliance starts at the design stage, not at the privacy notice.”
Processing special category data (e.g. health, ethnicity) requires an additional condition under data protection law, but the ICO states this should not deter innovators if the data is essential to the purpose. (Organisations using AI that processes sensitive personal data such as health, ethnicity or disability characteristics)
“The processing of special category data requires an additional processing condition under data protection law – but that shouldn't be interpreted by innovators as a reason not to process this data if it is essential to the purpose of processing.”
Purely technical approaches to mitigating unfair bias may risk breaching UK equalities law, for example reweighting approaches that actively favour disadvantaged groups may constitute positive discrimination. (Any organisation applying bias-mitigation techniques to AI systems used in the UK)
“some mitigation techniques may introduce additional risks; for example, approaches like reweighing - which actively seek to favour disadvantaged groups - may be interpreted as positive discrimination.”
Coefficient Systems Ltd's market review found that while many AI CV-screening tools claim to reduce human cognitive bias, they risk amplifying systemic bias if trained on historically prejudiced data, and many operate as 'black boxes' resistant to scrutiny. (Organisations procuring or using commercial AI-powered Applicant Tracking Systems (ATS))
“while many tools claim to reduce human cognitive bias, they risk amplifying systemic biases if trained on historically prejudiced data. A significant challenge identified was the 'black box' nature of many commercial systems, which prevents algorithmic scrutiny and makes it difficult to explain or challenge automated decisions.”
The EHRC states that transparency, explainability and accessibility of AI systems are key requirements to ensure human scrutiny and fairness under equality and human rights law. (All organisations deploying AI systems that may affect individuals' rights under the Equality Act 2010 or Human Rights Act)
“Accessibility, explainability and transparency of information and ways of working are also key to human involvement and scrutiny of AI systems, to ensure they are working fairly.”
Synthetic data was successfully used by two projects to detect AI bias without needing to collect real-world demographic data, demonstrating a practical alternative for organisations facing data access barriers. (Organisations seeking to test AI systems for bias where real-world demographic data is unavailable or restricted)
“Both Coefficient Systems Ltd and the Alan Turing Institute used synthetic data as part of the solutions they developed in the Fairness Innovation Challenge.”
This briefing is enkii's interpretation of the official document — the official text always governs.
Source document © Crown copyright, reused under the Open Government Licence v3.0 via the GOV.UK Content API. enkii tracks every Procurement Policy Note and briefs changes the day they land — see all briefings.