The Future of Design Comes First
What changes when responsible AI stops being a principle and becomes a signature? A world built to rehearse the accountability market forming now.
- Horizon
- 2036, a decade ahead
- One rule
- No high-agency system ships without a named person who can say no
- Recognised
- Winner, DesignUp Futures Challenge 2026, judged by futur2
The premise
Every bridge has an engineer’s name on it. No algorithm does.
In 2036 that changes. High-impact systems need a chartered designer’s clearance before they reach people, and the record is public. In the fiction, Mira Rao refuses clearance to SHIFT/9, a scheduling AI that works perfectly and still quietly punishes the workers who turn down overtime. The full world is on the project site; this page is about the gap it exposes today.
SHIFT/9Relay workforce system · H4
- Accuracy testsPassed
- Deliveries planned and staffedPassed
- A worker refuses overtimeRefused
- Later shifts for that workerQuietly worse
Clearance refused
The argument
Every profession was unlicensed until it hurt enough people.
Designers ship public-facing systems with no licence, no duty and no register. That is not because design is fundamentally different from medicine or engineering. It is because the regulators have not caught up yet, and they are starting to.
- 1910Medicine
Through most of the 19th century anyone in the US could practise. After the Flexner Report, roughly half of medical schools closed or merged.
- 1937Engineering
A gas explosion at a school in New London, Texas killed around 300 people in March. Texas licensed engineers by emergency legislation in May.
- 2023Interfaces get named
The EU’s Digital Services Act bans interfaces that deceive or manipulate. India’s consumer regulator lists thirteen dark patterns.
- 2025Interfaces get priced
Amazon pays $2.5bn over Prime sign-up and cancellation design. The harm was a screen, and the fine was for the screen.
Each profession was licensed once the harm became impossible to ignore and traceable to a decision. Chartering is what a profession looks like after the explosion. The fiction asks what it looks like if we build it before one.
Signals and drivers
It is already arriving, in pieces.
Courts, lawmakers and insurers are each building a part of clearance today, without calling it that.
- 2023Robo-firing goes to court
The Amsterdam Court of Appeal found Uber and Ola had deactivated drivers by algorithm and ordered the decisions explained.
- 2024Human review becomes a right
The EU’s Platform Work Directive puts account suspensions before a trained human who can overrule the system, by December 2026.
- 2025Algorithms enter Indian labour law
Karnataka’s gig worker law requires transparency on automated monitoring and decisions, though appeal rights stay weak.
- 2025Insurers price AI failure
Lloyd’s insurers now cover AI that underperforms, and turn down systems too prone to failing: underwriting as clearance.
Autonomy is outpacing accountability. Gartner expects agentic AI to make at least 15% of day-to-day work decisions by 2028, while the EU has pushed its high-risk AI duties back to December 2027. And the market is already pricing the gap: the UK counted 524 AI assurance firms and over £1bn of value in 2024, projects £18.8bn by 2035, and notes there is still no accredited certification scheme and too few qualified people.
Implications for today
The governance gap isn’t ethics. It’s a signature.
SHIFT/9 is invented. Its failure is on record. Most organisations already have AI principles, review boards and risk registers, but courts and insurers are starting to ask something those don’t answer: who decided, and could they have said no?
- Saying no
In 2020 a Bologna court ruled against Deliveroo’s rider ranking. It marked riders down for cancelling without asking why, so illness and strikes counted the same.
Accuracy tests won’t catch this. The harm sits in what the system does to the person who declines. - The human in the loop
In 2023 the EU’s top court held that a credit score can itself be the automated decision when a lender draws strongly on it, even though the lender makes the final call.
A person at the end of the process may not make it a human decision. How much they rely on the output is what gets examined. - The vendor
Air Canada was held to what its chatbot told a customer. In Mobley v. Workday, a US court let claims proceed that treat an AI hiring vendor as its customers’ agent. That case is still undecided.
Buying the system doesn’t hand over the blame, and selling it may not keep you clear of it. - Insurance
Since January 2026 Verisk has offered US insurers optional generative AI exclusions for general liability, while Lloyd’s insurers sell cover for AI that underperforms.
AI risk is being carved out of old policies and priced in new ones. The next question may come on a renewal form: what do you run, and who answers for it?
On the calendar
Some of the dates are already set.
None of these requires a chartered designer. Each asks, in its own way, for what a principles document can’t show: who reviewed a decision, on what basis, and what was recorded.
- Dec 2026EU platform work
Member states are due to transpose the directive, including human review of account suspensions.
- Jan 2027California hiring and work
Mid-size and large employers using automated tools for significant decisions face notice, opt-out and risk-assessment rules, unless a person meaningfully reviews the decision.
- Dec 2027EU high-risk AI
The AI Act’s high-risk duties land, after being pushed back.
- OpenWho carries vendor risk
No date yet. The Workday case has further rulings ahead, and they could shape how far vendors answer for what their tools decide.
Rival futures
Clearance arrives in one of four shapes.
The fiction rehearses the fourth because it is the only one where design holds the pen. The other three arrive by default.
- TheatreA checkbox, signed by the most junior person
Watch for: sign-offs on record, refusals never.
- CapturedIncumbents write the rules; the gate becomes a moat
Watch for: certification priced for the largest vendors only.
- UnderwrittenNo regulator; insurers decide what ships
Watch for: human sign-off as a condition of cover.
- CharteredA named professional with a duty, an appeal and a register
Watch for: a regulator recognising an AI governance credential.
Worth checking now
Questions that matter whether or not clearance ever becomes law.
None of these needs a new law or a new team. They show whether the answers already exist. The first starts where the fiction does: risk follows what a system can do to people, not how intelligent its model is.
- H1 ToolRoutine practitioner sign-off
People retain full intent and execution authority.
- H2 AssistantContext-specific examination
It may propose and recommend decisions, but cannot act.
- H3 DelegateChartered team + regular review
It may act on a person’s behalf.
- H4 Autonomous systemOperator licence + senior team + audit
It shapes conditions and meaningfully alters human agency.
- 01What can it do to people?
A system that advises is a different risk from one that sets someone’s shifts, whatever model sits underneath.
- 02When was a launch last stopped?
If nobody can name one, the review board may be a formality.
- 03What happens after someone says no?
Decline or opt out in your own product, then follow that account for a few weeks. Deliveroo’s ranking would have failed this.
- 04Who pays when it’s wrong?
Read the vendor contract and the insurance renewal side by side. The gap between them is your exposure.
For strategy and futures teams
Accountability is becoming a market.
Professions form where liability lands. AI governance already has its own certification (IAPP’s AIGP, 2024) and management standard (ISO/IEC 42001, 2023). Bring the charter and the refused review into a leadership session and ask three questions of every live system: would it pass, who would sign, and what happens to the person who says no. The silence after the second question is the finding.
The provocation
“Who signed this?”
When people can see who cleared a system, trust stops being an anonymous promise. Design doesn’t take over engineering or policy; it gains authority at the one boundary where technology changes the terms of human agency.
Recognition
DesignUp Futures Challenge Winner, again.
A year after Future of Currency is Alive, the project won the futur2 and DesignUp Futures Challenge for a second time. I presented it on the DesignUp main stage and once more received the award from Thomas Kueber and Andreas Wegner of futur2 studio. The full world holds the charter, the agency classes, the clearance and Mira’s review.
