The entire New York datacenter is experiencing a fire and is in emergency power off. An isolated fire in an UPS in an electrical room was detected and put out by fire suppression. The local fire department arrived on the scene, and per NEC guidelines and likely local laws and general best practices for firefighters, cut the power to the building. This caused the down -> up -> down cycle noted earlier today.
Addressing "is our data safe?" concerns:
Our understanding of the scope of the issue is that the fire was limited to an UPS in an electrical room, which was extinguished almost immediately by on-site fire suppressant, and did not extend to the data halls. This means that data should indeed be safe, however, our team has not been permitted on site yet, and power remains off to the building.
Current Status
Site management, the fire marshal, and electrical contractors are currently meeting to review the process of the cleaning effort to get approval from the fire marshal to re-energize the site.
Update:
We have been granted access to our pod. All servers are backup and running.
A reminder to please check your server and submit a ticket if you need assistance after checking it. We are even providing 1 week of credit all affected users.
Monday, July 10, 2023
