Up the propaganda output, Comrades!
The condition doesn’t appear in the standard medical literature — because it doesn’t exist. It’s the invention of a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. “I wanted to see if I can create a medical condition that did not exist in the database,” she says.
The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.
Both amusing and worrying and so on. The particular lesson from it is that another propaganda front has now opened up. AI believes what other people say - it regurgitates what other people say. So, in order to sway the answer from an AI it is merely necessary to introduce your pet falsehood into what the AIs train upon and it becomes the accepted wisdom - look, AI says it!
We’ve been through all of this before of course. We’ve retailed how deliberate misinformation over private children’s homes profits have been spread around the UK media. And of course we’ve had near two decades of reports on how RCP 8.5 is going to lead to climate change drowning us all in our beds. You know, the RCP 8.5 that never was going to happen and even the official stance is now that it isn’t going to happen. But those years of propaganda about how bad it will be have still coloured the more general conversation. Some sections of some of the more campaigning newspapers specialise in the promotion of this sort of pollution of the information well.
Now that pollution is moving a step further back. Pollute the information well of the AI, not the newspaper.
Anyone who thinks that people will not deliberately so pollute the AI infomation well doesn’t know anything about human beings. There really are people out there who desire to, are delighted to, manipulate people and society. As and when a new tool to do so arrives they will do so - they gain that power they desire over society by that pollution of the information society uses for decision making.
We then get to Brandolini’s Law: “The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it."
They - and one of the joys of this observation is that everyone gets to define their own “they” who will be so nefarious - will do this and it will take significant effort to refute them.
Which does lead to the interesting question. What the heck do we all do now?
Tim Worstall