Instapaper, former owned and developed by Marco Arment, recently had a major outage due to database limitations. Coincidentally, Marco recently posted about Automatically test[ing] your database backups. Brian Donohue, Director of Engineering for Instapaper, wrote a good technical overview for web developers and backend engineers on Medium about the incident titled Instapaper Outage Cause & Recovery
The critical system that failed was our MySQL database, which we run as a hosted solution on Amazons Relational Database Service (RDS)…
Root cause
In short, the data failure was caused by a 2TB file size limit for RDS instances created before April 2014. On Wednesday, February 9 at 12:30PM PT our bookmarks table that stores articles saved by Instapaper users exceeded the 2TB file size limit. Subsequent attempts to insert new entries into the bookmarks table began to throw the following error:
OperationalError: (OperationalError) (1114, “The table ‘bookmarks’ is full”)
The reason this limitation exists is because MySQL RDS instances created before April 2014 used an ext3 filesystem which has a 2TB file size limit. Instances created after April 2014 are backed by an ext4 filesystem and subject to a 64TB file size limit.