Who answers when the machine did the work?

When something goes wrong in an organisation, the first question is usually who did it. For most of history the answer pointed at a person: the clerk who approved the payment, the engineer who signed off the design. Automated systems break that link. The action happened, the consequences are real, and nobody in particular did it.

The responsibility gap

This is sometimes called a responsibility gap, and it is not a philosophical curiosity. A customer wrongly refused a loan, a candidate screened out by a model, a shipment misrouted by an agent acting on stale data: each needs someone who can explain the decision, correct it and answer for it. If the honest reply is that the system decided, the organisation has given up something it cannot afford to give up.

The gap does not close by itself. Vendors point to configuration, operators point to the vendor, and managers point to the process. Each is partly right, which is exactly why accountability has to be designed rather than assumed.

Designing accountability in

Some principles follow. Every automated action should have a named owner, a person whose role includes answering for it. The reasons for a decision should be recorded in a form that a person can read, not only reconstructed after a complaint. Automation should be easy to pause and its decisions easy to reverse. And the people affected should be able to reach a human who has the authority to change the outcome.

None of this slows useful automation down much. What it prevents is the slow drift in which work moves to machines and responsibility moves nowhere. An organisation that cannot say who answers for its systems does not really control them.

Accountable Agency asks who answers when machines do the work.

Further reading: Accountable Agency: Who Answers When Machines Do the Work

Drafted with AI assistance for the Y Academy Press editors, in line with the Press's Generative AI policy.