Database Backups

Michael E. Kirkpatrick ·

Instapaper, former owned and developed by Marco Arment, recently had a major outage due to database limitations. Coincidentally, Marco recently posted about Automatically test[ing] your database backups. Brian Donohue, Director of Engineering for Instapaper, wrote a good technical overview for web developers and backend engineers on Medium about the incident titled Instapaper Outage Cause & Recovery

The critical system that failed was our MySQL database, which we run as a hosted solution on Amazon’s Relational Database Service (RDS)…

Root cause

In short, the data failure was caused by a 2TB file size limit for RDS instances created before April 2014. On Wednesday, February 9 at 12:30PM PT our “bookmarks” table that stores articles saved by Instapaper users exceeded the 2TB file size limit. Subsequent attempts to insert new entries into the bookmarks table began to throw the following error:

OperationalError: (OperationalError) (1114, “The table ‘bookmarks’ is full”)

The reason this limitation exists is because MySQL RDS instances created before April 2014 used an ext3 filesystem which has a 2TB file size limit. Instances created after April 2014 are backed by an ext4 filesystem and subject to a 64TB file size limit.