Back to blog

What software shouldn't try to do

A lot of us are uneasy about where technology and AI are heading. That unease is worth listening to.

A lot of us feel uneasy about technology right now, and especially about AI.

It shows up in different ways. Some of us worry about jobs. Some worry that children won't learn to think for themselves, or about who sees our information. Some just have a quiet feeling that more of life is being decided by systems we can't see into.

We don't think that unease is foolish. A lot of it is reasonable. As people who make software, we think it's worth listening to rather than talking over.

For us, that starts with being clear about what software shouldn't try to do.

Decide for us

Software is good at sorting options, showing what's there and asking a useful question. Choosing is different. What matters, what to let go of, what today is for: those choices belong to people. A tool that quietly makes them for us takes something away, even when it means well.

Tell us who we are

It's easy for software to turn a few patterns into a conclusion, like a score, a category or a colour. But a pattern isn't a person. Most of us are better at making sense of our own lives than any summary of them.

Do the work that matters

Some work is the point: writing a first paragraph, planning a week, practising something hard. If software does it for us, the task gets done and the learning doesn't. Helping someone get started is different from doing it for them.

Stand in for people

Tutors, parents, practitioners and friends are the people who know us. Technology can help between conversations and make the next step clearer. It can't notice things the way a person does, and it shouldn't pretend to.

Pretend to be more than it is

Every tool has limits. We'd rather say plainly what something is for, and what it isn't, than let people find out the hard way.

Where AI fits

AI can be genuinely useful. It can help break a big task into small ones, ask a good question at the right moment, or take some routine work off a busy person's plate.

But we think of it as a helper, not a replacement. Wherever it's used, it should support the people involved rather than act in their place. A person should stay involved wherever it matters, able to see what's happening, change it, or say no.

That isn't because AI is bad. It's because people are the point. A tutor's judgement, a parent's knowledge of their child, a practitioner's experience, our own sense of what our day needs: these aren't things to automate away. They're what we want technology to support.

Holding ourselves to it

These lines aren't always easy to hold. There's always a reason to add one more automatic step, one more score, one more nudge. When we're unsure, we ask a simple question: is this helping people, or doing something to them?

Want to hear how we’re getting on?

Get updates