REPHRAIN Centre, University of Bristol

Evaluating AI Tools with Ethics, Fairness, and Purpose in Mind

Data use

Evaluation

AI tools

As child safety technologies advance, many are being developed for deployment in end-to-end encrypted (E2EE) environments to detect and prevent CSAM. But how do we assess whether these tools are actually doing what they’re supposed to—and doing it fairly, ethically, and effectively?

That’s the question researchers at REPHRAIN Centre (National Research Centre on Privacy, Harm Reduction and Adversarial Influence Online), University of Bristol set out to answer. Rather than focusing only on standard performance metrics like accuracy or precision, they proposed a broader, context-specific evaluation framework grounded in values such as trustworthiness, fairness, and appropriateness for use in sensitive environments.

“This is not an easy thing to do, but we wanted to set up a framework for evaluating such tools that takes into account additional criteria to performance and the typical metrics that are used in the context of AI tools such as accuracy, precision and recall. We wanted to look at other aspects that are more focused on human-centric values, and values that are already part of the AI community in terms of trustworthy and ethical AI”

Claudia Peersman, University of Bristol

Their research surfaced several technical and ethical concerns:

Repurposing risk:

While the tools may be developed for a specific purpose, such as online child protection, the way data is collected and structured creates a risk that it could later be repurposed; for example, used to analyse private communications or contexts it wasn’t originally intended for, such as limiting free speech

✓
Fairness limitations

Existing CSAM detection tools frequently underperform when applied to victims from underrepresented backgrounds. A key cause was the lack of diversity in training datasets (particularly across ethnicity, gender, and age) leading to tools that systematically miss certain populations. An equitable performance across different demographic groups is essential to ensure that no children are left behind.

✓
Benchmarking gaps:

The team highlighted the absence of diverse benchmark datasets, which limits the ability to meaningfully compare tools or evaluate their performance across different demographics and use cases.

✓

One of the team’s key findings was that evaluation frameworks must be tailored to the tool’s specific purpose and use context (e.g., private vs. non-private communications, digital forensics investigations). Standardizing evaluation without regard to context could miss important risks or falsely validate tools that aren’t fit for purpose.

As AI tools become embedded in online child protection, we need to design evaluation processes that go beyond performance.

PREVIOUS
Internet Watch Foundation – Every Child in the Frame

NEXT
Meet the Data Champions Stories of Impact

REPHRAIN Centre, University of Bristol

We'd love to stay in touch!

Learn to report abuse

REPHRAIN Centre, University of Bristol

We'd love to stay in touch!

Learn to report abuse

Our purpose in detail