Judgement is not a calculation

Much of the conversation about artificial intelligence assumes that a good decision is the output of a good enough calculation. Give the system the right objective, the right data and enough capacity, and the right answer follows. For some problems that is true. For the problems people most want help with, it leaves out the part that matters.

Where calculation runs out

Judgement is what we call the capacity to decide well when the rules do not settle the case. A doctor weighing a treatment against a patient’s stated wishes, an editor deciding whether a claim is strong enough to print, a manager choosing between two people who both deserve the role: each has information, and each still has to decide what the information means here, for this case, with these stakes.

Alignment, in its technical sense, asks how to make a system pursue the goals we intend rather than the ones we accidentally specified. It is a serious problem. But it presupposes that we can state our goals clearly enough to be pursued. Judgement is needed precisely where we cannot: where values conflict, where the situation is new, where the honest answer is that it depends.

Agency without judgement

Systems are now being given agency in a practical sense: the ability to take actions, chain them, and act on the results without a person approving each step. Agency without judgement is not malicious; it is literal. It does what it was set up to do, in situations its designers did not foresee, and it does so quickly.

The question for the coming decade is therefore not only whether machines will become intelligent but where we decide judgement must stay with people, how we keep it there, and how we notice when it has quietly moved. That is a design question, a legal question and, above all, a question about what we are prepared to delegate.

AI Agency, Alignment and the Future examines mind, machines and the question of judgement.

Further reading: AI Agency, Alignment and the Future: Mind, Machines, and the Question of Judgement

Drafted with AI assistance for the Y Academy Press editors, in line with the Press's Generative AI policy.