Karin Verspoor
AI companies rely on web data to build their models. Crawlers have been a standard part of how companies like Google and Microsoft collect the data for search engines going back decades. They automatically access websites through known links, “scrape” the contents of each webpage — including all of the links on each page so they can find more webpages — and store and index them in a big database so that they can be retrieved in response to a query. The same strategies have been leveraged by the LLM developers to harvest the data used to train their models.
An important part of these systems is what is called the Robots Exclusion Protocol, where a website can signal to a crawler that they don’t want certain parts of the site to be accessed. In this case, OpenAI has very likely ignored any such restrictions set up by the government websites, and simply hoovered up everything that could be accessed.
Modern crawlers may also be randomly guessing links to try to access — by generating plausible URLs from known URLs through very similar methods as what are in the LLMs themselves — and thereby accessing pages that are not directly discoverable from the public-facing websites. The fact that the crawlers were able to access private files suggests that the security settings on the websites need to be reviewed to prevent unauthorised access.