The purpose of this document is to provide an assessment of the intermittent issue with call processing on Thursday, June 12th.
Timeline of issue and root cause:
We received reports regarding call processing failures at approximately 5:15 PM PST.
Our engineering team was engaged, and we started troubleshooting. We determined that there was a storage issue with the F5 firewall LTM (Local Traffic Manager) at our primary data center (ORD). Our engineering team worked closely with the vendor to isolate the root cause. It was determined that a backup file was not getting compressed properly. Necessary updates were made, and we resumed normal behavior at approximately 5:47 PM PST.
Corrective Action:
We worked with the vendor and corrected the backup process. Additionally, we made sure that we get proper notifications moving forward to prevent this issue in the future.