INCIDENT

The reset that dropped every database

2026-08-01 · 6 min read

Talos had a "reset server" action for decommissioning a Frappe host. Underneath, it ran an Ansible playbook. That playbook enumerated the MySQL databases on the host and dropped every one that was not a system database.

Not every Frappe database. Every database. If the host also carried an analytics schema, a staging copy, or anything else a colleague had put there eighteen months ago, reset took it too.

It was reachable from three places

The action was not limited to Frappe projects. It was exposed on database, deployment and Kubernetes project types as well — contexts where the assumption "everything here is Frappe" is not merely unsafe, it is obviously false.

The confirmation modal described a narrower scope than the operation had. It said, in effect, that Frappe data would be removed. That was the intent. It was not the behaviour.

The fix that did not work

The obvious repair is a better filter: identify the Frappe databases and drop only those. We started there and stopped, because it does not survive the question "what if the filter is wrong?" A filter you cannot inspect is the same trust you had before, with more code.

Plan, then apply

Reset is now two operations. The first produces a plan: every database, bench, volume and container it intends to destroy, each annotated with how it was identified — this database is owned by that bench, this bench was matched by its apps file. You read the list.

The second applies exactly what the plan named, and nothing that was not on it. Anything discovery could not positively attribute is excluded rather than swept up, which means the failure mode changed direction: an unrecognised database is now kept, not destroyed.

reset plan · legacy-02
✓ bench/erp-prod      frappe bench · matched by apps.txt   DESTROY
✓ db/erp_prod         owned by bench erp-prod              DESTROY
✕ db/analytics_ro     no owning bench — not attributable   KEPT
✓ backup verified     restored to scratch and checked      GATE

And a backup that has been restored

A mandatory backup now runs before the destructive step, and the restore is verified — restored into a throwaway instance and checked — before apply is allowed to proceed. A backup taken immediately before an irreversible operation is exactly the backup you cannot afford to have silently failed.

The lesson is not "add a filter". It is that a destructive operation should be able to tell you what it is about to do, in terms you can check, before it does it.

One caveat worth stating: this work is built and unit-tested, and the host-level test against a real MySQL box is still outstanding. It is listed as outstanding rather than quietly assumed done.