The recent publicity over the Post Office scandal, where more than 900 people were wrongly prosecuted based on information provided by a faulty computer system, has underscored the importance of humans having the confidence to question technology. The fallibility of technology is obviously a salient issue of the scandal, but equally alarming is the way in which people were told, and had no choice other than to accept, that they were the ones at fault.
However, that they would accept this is not unusual. People often doubt their own understanding, skills or ‘know-how’, frequently defaulting to the answers a technological system provides. That they trust the answers it gives more than they trust their own mind is not surprising when our prevalent cultural narrative often suggests that technology knows you better than you know yourself.
Music streaming services suggest songs you will like; health and fitness trackers tell you things about your sleep and activity patterns that seem significant; you are told 'People like you also bought…' and Alexa and Siri can anticipate your needs based on your routines or past interactions. Couple this with our natural human tendency towards confirmation bias, where we keep track of the hits (where tech gets it right) rather than the misses (where it gets it wrong), and we have a perfect storm of unquestioning. It makes it even harder to feel anything other than 'technology knows best'.
Yet computer systems are fallible, they always have been. This is perhaps a reason why the scandal was not picked up in Computer Weekly – the idea that a computer system had problems was not news to that audience of frequent, competent and expert users.
However, it seems that many ‘normal’, non-techy people are beginning to remember this once more. It is this confidence to question the computer, the AI, or the algorithm that we need to continue to develop within our young people as we move forward.