On September 24, 2020 at 11:00 PM Eastern, we were unsuccessful in an attempt to perform 1 hour of database maintenance which caused an additional ~1 hour of downtime affecting all customers. We’re very sorry this occurred and would like to share with you what happened, what we learned, and what we're doing to reschedule the maintenance and prevent this from happening in the future.

What happened

We stopped our database to perform the maintenance around 11:05 PM Eastern. At 11:30 PM Eastern, we noted that the remaining time to stop the database was unusually high: 2 to 8 hours. We logged a case with our hosting provider (AWS), made the decision to abort the maintenance, and started recovery procedures.

Aborting the planned maintenance required a point-in-time database recovery, which is slower than normal database restoration procedures. The additional hour of downtime was caused by waiting for the recovery to finish. Access to ProcedureFlow was restored around 1 AM Eastern.

What we learned

• Testing is important but doesn't always highlight potential gaps in a plan. Our prior testing did not identify any problems with stopping the database. We tested using recent backups of our Production database and also performed a full dry-run in a Staging environment.
• When stopping a database, a snapshot is created from the primary database and not the standby database. Normally, snapshots are very fast because they're incrementally created from the standby database (like nightly backups). By design, snapshots are not normally created from the primary database. Since there were no previous incremental snapshots of the primary database, a very slow snapshot was created for the full database.
• In hindsight, the steps to perform the maintenance did not require us to completely stop the database. Our hosting provider informed us that there were alternative steps we could have taken. In early August, we had confirmed the steps for our plan with a different representative. However, at the time, they did not identify this problem as a gap in our plan, nor did they suggest alternatives to the plan we presented.
• The point-in-time recovery procedure can be triggered even while the database is in the middle of stopping. We confirmed this during the phone call with our hosting provider. This is helpful for other emergency situations.

Prevention in the future

• We will confirm any major maintenance plans with multiple representatives at our hosting provider to ensure a consistent plan.
• We will avoid stopping the database until it's no longer needed. Using the new approach that we learned, we can pause traffic to the database, snapshot the standby database while it's running, and launch a new database from the snapshot with any changes applied.
• We will scrutinize maintenance plans from additional perspectives that might seem trivial, like stopping a database or server.
  

We know how much you rely on ProcedureFlow to help your business succeed. We will continue to analyze this event for opportunities to serve you better and continue to earn the trust you place in us.