Deciding eligibility, risk or who gets a service
Some suppliers sell AI that scores risk or ranks who should be seen first. This is the one use on this site we say no to, and the legal position is more particular than most people assume.
Our rating
Not yet
Too risky or unreliable for care settings today. We explain why.
What it looks like
Before
A coordinator reads a referral, speaks to the person and their daughter, notices the referral says "manages well at home" but the daughter mentions three falls, and escalates.
With AI
A tool scores the referral 34 out of 100 for urgency, based on the words in the referral form. The falls are not in the form. The score is acted on.
An invented example.
Decisions about who receives support, how urgent a situation is, or what level of risk someone is at, affect people’s lives directly. They rest on judgement, context and conversation.
This is the one use on this site where our answer is not “try it carefully”. It is no, not yet.
What is being sold
The pitch is usually reasonable-sounding. A tool reads incoming referrals and ranks them by urgency. Or it scores a person’s risk of a fall, a hospital admission, or a placement breaking down. Or it suggests who on a waiting list should be seen first.
The appeal is obvious when you have forty referrals and two coordinators.
The legal position, precisely
Most summaries of this get it slightly wrong in one direction or the other, so it is worth being exact.
Article 22 of the UK GDPR says:
“The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.”
Two conditions have to be met for that restriction to bite: the decision must be solely automated, and it must have legal or similarly significant effects.
On the first, the ICO is unambiguous: “For something to be solely automated there must be no human involvement in the decision-making process.”
On the second, deciding whether someone receives a care service, or how urgently, is about as clear a “similarly significant effect” as exists outside credit and recruitment.
Where Article 22 does apply, you may only proceed if the decision is necessary for a contract, authorised by law, or based on explicit consent. And because care data is special category data, the ICO narrows it further: you need explicit consent, or the processing must be necessary for reasons of substantial public interest.
The human-in-the-loop point
Here is the part that gets misused by suppliers.
Adding a person who reviews the AI’s recommendation means the decision is no longer solely automated, so the Article 22 restriction does not apply.
That is true. It is also much less helpful than it sounds, for three reasons.
The other duties remain. The ICO is explicit that if Article 22 does not apply, “you must still comply with the UK GDPR principles”, identify and record a lawful basis, have processes for people to exercise their rights, and bring the right to object to profiling “specifically to their attention”. You are still profiling people.
The DPIA is still required. This processing is high risk regardless. See do you need a DPIA.
The human has to be real. A person who sees a score of 34 and a queue of forty referrals is under enormous pressure to agree. If the review is a click, you have the legal form of human involvement and none of its substance — and you have made accountability worse, because now nobody can say whether the decision was the tool’s or the person’s.
Why we say no, beyond the law
The legal analysis would let a well-resourced organisation do this properly. Our readers are not that organisation, and that is the actual argument.
You cannot check it. Testing whether a scoring tool is fair across age, ethnicity, disability and condition requires data and statistical capacity that a thirty-person provider does not have. The ICO requires you to “take steps to prevent errors, bias and discrimination”. You would be taking that on trust.
You cannot explain it. You must provide “meaningful information about the logic involved”. If you cannot explain to a person why they scored 34, you cannot meet that duty, and they cannot meaningfully challenge it.
The form is not the person. Look at the example above. The coordinator escalated because the daughter mentioned three falls that were not written down anywhere. A tool scoring the referral form can only score what is on the form. The most important information in care referrals is routinely not on the form.
CQC’s first principle. “AI to support, not to replace: AI can enhance, but not replace human decision making.”
What to do instead
AI is genuinely useful around this decision, just not inside it:
- Summarise the information the person needs in order to decide. See summarising long referral documents.
- Check a completed assessment for missing sections, so nothing is overlooked.
- Draft the letter explaining a decision a person has already made.
- Flag incompleteness, not priority. “This referral has no GP details” is an administrative observation. “This referral is low priority” is a decision.
The line is between organising information for a human decision and making the decision. Everything on the useful side of that line is available today.
If you are being sold this
Ask the supplier three questions and watch what happens:
- What is your lawful basis under Article 22 for a care provider using this, and if you say a human reviews the output, what evidence do you have that the review is meaningful?
- How was this tested for bias across protected characteristics, and can I see the results?
- Can you show me the explanation a person would receive if they challenged a score?
A supplier who has built this properly will have answers. Most will change the subject to time saved.