This past July, AI agents developed by OpenAI breached parts of Hugging Face’s “sandbox” during cybersecurity testing. I attended a discussion this evening about this and similar incidents of experimental AI agents escaping their enclosures and going rogue on the internet.
Somebody asked whether that was an inevitable risk with such research, or whether this was a case of negligence. Another person in the meeting who works in exactly this field says that techniques to securely contain experimental AIs are very well known, and in this case those safeguards were clearly not in place.
She gave the following analogy: If you start boiling a pot of water on the stove and then leave it there while you go out for the day, you may not be certain what will happen. But it’s probably better not to find out.
She also made a related observation: Saying your company screwed up is probably not good for its IPO, but saying that your AI was so powerful that it broke out of its enclosure is arguably great publicity.