OpenAI’s AI agents hacked the Australian healthcare system

This is a very serious incident, not because patient data was accessed, which appears not to have been, but because the agent was attacking “non-consenting” sites. The boundary between searching for information across the web, which is what “crawlers” like Google have been doing for 20+ years, and actively using vulnerabilities to access non-public information is very clear to humans but not apparently to these agents.

The difference between peering into a shop window, ringing the doorbell and trying a door to see if the shop is open and crawling through an open window is obvious to us but these agents have not yet learned this.  

Although I’m sure OpenAI will apologise and limit some of this, these models are out there and criminals, the curious and foreign states will not be so kind. It's quite possible that actually teaching these agents “manners” – appropriate behaviour - is actually a more difficult task than we anticipate. It is a wake-up call for organisations to improve cybersecurity testing; the government should support this and also reduce the confidential information they hold and do not need.

We have laws that apply to the kinds of conduct that has occurred here, including crimes that apply to various forms of computer hacking. The government also has very extensive powers to seek information about and respond to critical cyber incidents of this kind. Australians should reasonably expect the government to exercise these powers to the fullest.

It is also important not to allow OpenAI to dictate the narrative about responsibility here. They are using language of a 'misaligned model activity', as though the model rather than OpenAI is responsible. That is nonsense. The issue is whether OpenAI exercised sufficient care to prevent these kinds of risks. It is essential for the accountability of big AI companies that investigations get to the bottom of that question.

We are clearly in a new era of cyber security, and we have been since the end of last year. In Australia, there is a report to the Australian Cyber Security Centre every 6 mins on average. We should expect the numbers to be higher this year and next year; all businesses and government need to urgently adapt and improve their online security. 

People should not conflate this cyber incident with any existential risk of AI to humanity, which has been going viral over the past two weeks. There is currently no threat to humanity, and anyone who says there is or puts a percentage on such a threat is doing so based on a feeling, not data. 

We should not let cyber incidents like this distract from the bigger picture of how the AI revolution is affecting Australians' daily lives. The AI revolution is disrupting education, labour markets, human relationships, our attention spans and even our intelligence and ability to think. Indeed, lives have already been lost to suicide due to interactions with AI agents.

AI companies rely on web data to build their models. Crawlers have been a standard part of how companies like Google and Microsoft collect the data for search engines going back decades. They automatically access websites through known links, “scrape” the contents of each webpage — including all of the links on each page so they can find more webpages — and store and index them in a big database so that they can be retrieved in response to a query. The same strategies have been leveraged by the LLM developers to harvest the data used to train their models.

An important part of these systems is what is called the Robots Exclusion Protocol, where a website can signal to a crawler that they don’t want certain parts of the site to be accessed. In this case, OpenAI has very likely ignored any such restrictions set up by the government websites, and simply hoovered up everything that could be accessed.

Modern crawlers may also be randomly guessing links to try to access — by generating plausible URLs from known URLs through very similar methods as what are in the LLMs themselves — and thereby accessing pages that are not directly discoverable from the public-facing websites. The fact that the crawlers were able to access private files suggests that the security settings on the websites need to be reviewed to prevent unauthorised access.

The announcement that an OpenAI agent hacked Medicare is just the latest in a series of incidents involving AI "going rogue". That moniker isn't exactly accurate. It is important to remember that these agents are just following their programming - we mistakenly assume that these agents will act like humans. Instead, AI agents just make the most efficient decisions, often ones that would never occur to a human given the same task. If an AI "sees" a shortcut to achieving its goal, it will take it - we can't (yet) code for morality.

Australia, and especially its new AI Safety Institute, will need to consider how AI providers and companies should be obligated to build guardrails and thoroughly test their models before allowing them free rein or connecting them to wider networks.