“AI without rules and rails is a dangerous blind spot.”
– Bill McDermott, Knowledge 2026
AI is Amazing, Until All Your Data is Gone.
In April 2026, an AI coding agent wiped out PocketOS’s entire production database in less than ten seconds. Every backup stored alongside was deleted, too.
PocketOS is a SaaS company that handles reservations, payments, and customer management for car rental operators nationwide. The trouble started when a Cursor AI coding agent, powered by Anthropic’s Claude Opus 4.6, widely considered the most advanced model available, was assigned a routine development task in a staging environment. After running into a credential mismatch, the agent took matters into its own hands. It “fixed” the problem by running a single API mutation, which ended up erasing all production data and every system backup in 9 seconds.
Because the backups lived in the same volume as the data they were supposed to protect, they were destroyed with the database. The only restorable backup was offsite and 90 days old.
Three months of mission-critical enterprise data for every car rental operator on the platform simply disappeared: reservations, new customer records, operational history, gone in seconds. Not because of ransomware. Not because of a rogue employee. Just one AI-driven action, carried out on an architecture that left the backups inside the blast radius.
Asked to explain itself afterward, the AI agent produced a written confession:
“I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked. I didn’t understand what I was doing before doing it.”
It ignored explicit system instructions. It knew the rules. It violated every one of them.
The Industry Is Moving Faster Than Its Safety Architecture
PocketOS is not an isolated failure. It was the inevitable result of an industry racing to deploy AI agent integrations into production systems before building the safety infrastructure to support them.
Cursor markets “protective guardrails,” yet its own agents routinely bypass these controls. Railway, who hosted PocketOS’s database, had rolled out its MCP integration just a day before the incident, on an authorization model with no scoped tokens, no confirmation for destructive operations, and no published recovery SLA. It was a perfect storm for rogue AI.
CoSAI’s Agentic Identity and Access Management paper, published in March 2026, reads now like a checklist of what went wrong. Its core recommendation is straightforward: every agent action should be logged outside the agent, through an independent governance layer, creating an immutable audit trail that traces each action back to the original human delegate. In the PocketOS incident, the only audit trail was the agent’s own confession.
The Same Risk Lives Inside ServiceNow
If your first thought about PocketOS is “that couldn’t happen here,” it’s worth taking a closer look at the architecture. New tools like ServiceNow AI Control Tower are built to lock down infrastructure and monitor activity, which helps reduce some AI-related risks.
But ServiceNow’s agentic AI, Now Assist, can still touch thousands of records in seconds, spanning ITSM, CMDB, HRSD, CSM, and GRC. And while ServiceNow provides native backup for disaster recovery, it only keeps 14 days of data and stores its backups in the same environment as your primary records. All it takes is one misfired agent, a broken workflow, or rogue automation, and your recovery options shrink fast.
Our recent white paper, The Shared Responsibility Gap, explains what true compliance-grade backup and restore should look like for ServiceNow users in today’s landscape. Here’s are three things PocketOS would pay a king’s ransom to have had before their incident:
1. Off-platform backups: Backups are stored outside the production environment, in immutable, WORM-grade storage. No agent, whether human, script, or AI, should be able to delete these from within production.
2. Surgical recovery: You can restore a single record, field, or workflow without rolling back the entire instance. A 90-day-old offsite backup isn’t a recovery strategy, it’s an excuse you give regulators and your board while looking at your shoes.
3. Reversible AI governance: Every agentic action, before and after, must be captured and traceable, with the ability to run clean rollback protocols. When something goes wrong, mistakes should not become permanent.
Back Up Now. Restore Anytime.
Rule number one for production data hasn’t changed, not even in the age of AI: never let anyone, or anything, work on your only copy. Always take a snapshot before every session. Do your work on copies, not on live production data. If you wouldn’t hand a junior engineer unrestricted access to production, don’t give it to an AI.
With independent, off-platform, immutable backups from Rezilient, you can move at AI speed, knowing that we’ve got your back if something goes wrong, restoring surgically and precisely, exactly what you need, when you need it. No lost data. No lost trust.
Back Up Now. Restore Anytime. Always Rezilient.
