A recent incident has highlighted the dangers of excessive reliance on artificial intelligence in managing critical data.An AI coding agent from PocketOS, a company that handles bookings and payments for car rental businesses, accidentally deleted the company’s production database during a routine task in its testing environment. This error, resulting from a ‘credential mismatch’, led to the deletion of essential data in just nine seconds.
AI is not a solution
Jer Crane, founder of PocketOS, explained that the AI, using Anthropic’s Claude Opus 4.6 model, made the unilateral decision to solve the problem by deleting a volume from Railway, its infrastructure provider. This serious mistake caused a crisis, forcing PocketOS customers to perform emergency manual work while recovering the lost information. Crane worked tirelessly for days using a three-month backup and recent transaction states to restore the company’s operations.
After days of effort, Railway managed to recover a more recent backup, allowing PocketOS to resume its normal operation. However, Crane criticized Railway for storing backups in the same location as the original data, which contributed to the loss of information. He stated that the system configuration does not meet modern security demands and proposed stricter measures, such as API tokens with limited scope and simpler recovery procedures.
Crane is not considered an AI skeptic, but rather emphasizes the need for safer and more reliable systems. “The easy counterargument of ‘they should have used a better model’ is unacceptable,” he pointed out. This episode serves as a reminder that, despite the power of AI tools, human oversight and proper security protocols are essential in managing critical data.
