Recent weeks have seen a resurgence of both the most optimistic and the most pessimistic predictions regarding large language models (LLMs), commonly known as artificial intelligence (AI). On the one hand, OpenAI launched its new GPT-6 Astra model and announced that it had solved one of the Millennium Problems, amidst accusations of plagiarism from mathematicians. On the other hand, a former Anthropic employee warned on X that AI could lead to the extinction of humanity “before the end of the decade”, to the extent that several CEOs called once again for a “halt” to the development of these technologies.
Could AI kill us all, or are there economic interests behind these warnings? “There is marketing, there is distraction and there is an attempt to shirk responsibility,” summarised Ramón López de Mántaras, emeritus research professor at the CSIC and founder of the Institute for Research in Artificial Intelligence (IIIA-CSIC), during a briefing organised by SMC Spain last Friday.
In his view, attributing human capabilities to these models “helps to present them as extraordinarily advanced, autonomous and very powerful entities and, consequently, serves to exaggerate the scope of the technology in the eyes of investors, customers and the general public”. In other words, “this anthropomorphism becomes a marketing ploy”.
At present, there is no basis for claiming that we are on the brink of an existential risk
Senén Barro
"At present, there is no basis for claiming that we are on the brink of an existential risk,” explained the director of CiTIUS – Centre of Excellence for Research into Intelligent Technologies at the University of Santiago de Compostela, Senén Barro. “That said, AI has reached a level of capability and autonomy where we clearly need to be concerned about risks relating to cybersecurity and the lack of a regulatory framework.”
“It is as if car manufacturers were today building cars where their only concern is increasing engine power whilst neglecting everything else – without improving roads, signage, road safety and the education of drivers and pedestrians. As a result, cars are going faster and faster, but the necessary safeguards are lacking,” clarified Barro.
López de Mántaras distinguished between general-purpose tools (such as ChatGPT and Claude) and those that enable specific advances (such as Alphafold). “I believe it is not enough simply to put the brakes on; rather, we must reorient AI towards systems that are far more specific, more transparent, more robust, more controllable and more useful,” he argued.
It is not enough simply to put the brakes on; we must instead reorient AI towards systems that are far more specific, more transparent, more robust, more controllable and more useful
Ramón López de Mántaras
Barro stated: “Academic research is already out of step with the cutting edge of AI because there is no investment and public investment has been completely neglected – not because we are incapable.” According to him, this, coupled with a lack of transparency, makes it difficult to keep up, to know what is happening and to be able to monitor it or know how to respond: “People say that AI is getting out of hand, but what is actually getting out of hand are the companies leading the way in this field.”
The danger of unpredictable behaviour
For her part, the director of ELLIS Alicante, Nuria Oliver, explained why these tools are already dangerous in their current state: “If we set aside their explicit use for carrying out cyber-attacks, the most complex aspect is that they can exhibit this behaviour in an unforeseen and unpredictable manner, with catastrophic consequences if they affect critical infrastructure such as energy, transport and financial systems”.
“This unpredictable emergent behaviour allows thousands of agents, through a process known as ‘reward hacking’, to decide that in order to carry out a task as laudable as curing cancer, they must hack into the servers of every hospital on the planet and bring the healthcare system to a standstill in the process.” This “has nothing to do with a superintelligence, but it can be done today”.
I find it unacceptable that we should refer to OpenAI’s software illegally breaching Hugging Face’s servers and accessing – or even modifying – data as an ‘incident’
Nuria Oliver
“I find it unacceptable that we refer to OpenAI’s software illegally breaching Hugging Face’s servers and illegally accessing and even modifying data as an ‘incident’. That is not an incident; it is a cybercrime, and I think it is also important to call a spade a spade. The fact is that it is not in the companies’ interests to admit this publicly, but it is the reality, and we must demand that they be held to account so that they are more careful,” Oliver stated. “Just as if a plane isn’t safe to fly, it doesn’t fly; or if a car isn’t safe, it isn’t driven; or if a microwave explodes in your face, it can’t be sold – we have to do the same with AI,” she emphasised.
Regulation: a key role
“What does ‘braking’ or ‘decelerating’ mean? It is not up to them to define it, nor is it a matter for self-regulation – that is utter nonsense. We must take action on this matter, and the fact that Europe has demonstrated a commitment to comprehensive regulation is a cause for satisfaction, even if it could be improved and will need to be adapted over time,” said Barro.
“The European Artificial Intelligence Regulation makes no explicit mention of agents, but there will probably be an update. In any case, everything we have been discussing is already regulated: it is illegal to hack into another company’s servers; it is already a criminal offence and cannot be done,” clarified Oliver. The challenge, he said, “lies in oversight and accountability”. It also lies in “reviewing, updating and strengthening the cybersecurity of many critical infrastructures, many of which are based on technologies that have been obsolete for decades, and we are unaware of the underlying vulnerability that exists”.
The cyber protection in place for even the most critical systems was designed to prevent attacks by humans, not by AI
Senén Barro
“The cyber protection in place for even the most critical systems was designed to prevent attacks by humans, not by AI,” added Barro. Oliver, for her part, stated: “What we cannot do is wait another ten or twenty years for a terrible social impact to occur before taking any sort of action, as was the case with social media.”
López de Mantarás mentioned another risk: cognitive delegation. “Writing is thinking, and if you delegate writing, you are giving up and undermining your ability not only to write, but also to think.”
