From approximately 12:15 PM ET on April 22 to 6:00 AM ET on April 23, all customers were unable to create change requests due to a database issue. Editing existing change requests was mostly unaffected.
One of our database tables reached the maximum number of allowed entries (~2.1 billion), preventing change requests from being created. The maximum number was reached due to growth of ProcedureFlow usage. Although we have monitoring in place for database limits, we underestimated the rate of growth and were only monitoring these types of limits infrequently.
The issue required an urgent database migration to expand the ID capacity and restore normal functionality. The migration was completed successfully without any data loss, and system availability for viewing and editing existing data was maintained throughout.
The database migration and related mitigation steps took approximately 18 hours to complete due to the large volume of data involved. Under normal circumstances, we would perform this type of migration gradually and without any customer impact. However, because this issue affected all customers and prevented critical functionality, we prioritized speed and executed the migration as quickly as the system would allow, balancing urgency with data integrity and stability.
We've since completed the necessary upgrades to prevent this issue from recurring and have made improvements to our monitoring and deployment processes to ensure faster detection and recovery in the future. For example:
Introduce more frequent manual and automated monitoring for database table entry limits.
Monitor and plan to upgrade any "legacy" tables that have the ~2.1 billion entry limit. All newer database tables in ProcedureFlow do not have this limit.
We've completed the necessary upgrades to prevent this issue from happening again and have made improvements to our monitoring and operational processes to enable faster detection and recovery in the future. For example:
We've implemented more frequent manual and automated checks for database table entry limits.
We're monitoring and planning upgrades for any legacy tables that have the ~2.1 billion entry limit. All newer tables in ProcedureFlow do not have this restriction.
We sincerely apologize for the disruption caused by this issue and understand the impact it had on your ability to make changes in ProcedureFlow. We are committed to ensuring this doesn't happen again, and we are continuously working to make ProcedureFlow more reliable and resilient.
We value your trust and appreciate your patience as we continue to enhance the reliability of ProcedureFlow.