On Wednesday 10 March, we experienced a major incident at our Strasbourg, France (SBG) site.
A fire broke out during the night, affecting many international customers who have data hosted in our Strasbourg data centers. Our industrial and technical teams took action in an exemplary manner throughout the night, under particularly difficult circumstances.
Thankfully there were no casualties, either among our ranks or among the firefighters who worked to bring the fire under control. Our teams are also very grateful for the words of support sent by our customers throughout the day.
Ever since the local authorities reopened access to the site in the late morning, we have been putting all our efforts into restoring services and supporting our users. Below is some information and recommendations for our customers.
[LOCATION] How do I know if my servers are located in Strasbourg?
Your OVHcloud Control Panel will show you where your server is located.
You can also use a trace-route to find out where your traffic is being routed.
[LOCATION] Which data centers are affected in Strasbourg?
Of the four OVHcloud data centers in Strasbourg:
- SBG1 was affected by the fire: four of its rooms were severely affected (racks 61Bxx, 61Cxx, 62Bxx and 62Cxx); the other eight were spared. The SBG1 network was working. However since Friday, March 19 we have switched the network off as a precaution and determined this data center will not be reactivated.
- SBG2 was the most affected: it was destroyed in the blaze.
- The servers housed in SBG3 and SBG4 were not affected.
[FOLLOW-UP] Where can I follow updates?
You can follow the evolution of the incident and find information concerning future corrective actions using our Incident Status page.
[FOLLOW-UP] What is the timing to resume my services hosted in Strasbourg (SBG)?
The corrective action plan to restore the infrastructure and service will be published and updated daily here.
[FOLLOW-UP] How do I restart my server?
The recoverable servers are coming back online. They are booted into rescue mode.
We recommend that you keep your server in rescue mode to extract the most critical data before rebooting to the server’s operating system.
Once critical data has been saved/recovered, we recommend carrying out hardware tests (processor, memory, and hard disks SMART tests) to ensure that your server is working properly.
For more details about rescue mode, refer to our technical guide here.
After all the above actions have been taken, you can reboot your server from its operating system.
If your server fails to boot from its OS, we recommend connecting via IPMI to a KVM remote control interface.
For more details about IPMI, refer to our technical guide here.
[FOLLOW-UP] What should I do if my service cannot be restored or this time-frame is not soon enough?
We recommend deploying your infrastructures in another data center. Our Roubaix (RBX) and Gravelines (GRA) sites have a stock ready to be delivered to the majority of affected customers. We will further enhance availability in these data centers, deploying nearly 10,000 new servers in the coming weeks.
We also recommend where a US data center can be used to consider this location.
[FOLLOW-UP] What can I do if I have not received an email regarding the status of my services and the commercial measures taken?
Since the start of the issue on March 10, 2021 OVHcloud US has been sending personalized emails to all customers affected by the Strasbourg (SBG) incident. However, if you did not receive any of these emails and you have any services hosted in this data center under your US account, we suggest the following tasks:
- Check your spam folder in your email
- Check the email address entered in the Control Panel for the administrator of your service(s)
If these email addresses are not up-to-date, please change them in your Control Panel by editing your contact information.
If you have taken all measures above and still do not receive our email communications - please contact our Support teams standing by to assist you in your OVHcloud Control Panel via Support chat during business hours only or via support request 24/7.
For security reasons, messages are only sent to the addresses you have entered in your Control Panel.
[FOLLOW-UP] How do I contact support?
As we have recommended in our direct communications to OVHcloud customers with services in Strasbourg, we recommend activating the Disaster Recovery Plan, which is specific to your company’s internal organization.
For any other requests regarding replacement services or order delivery, please create a support request from your OVHcloud Control Panel or reply directly to our emails with Strasbourg information.
You can also follow the status of this incident here
[BACKUP] Are DNS zone operations disrupted?
They are not directly affected. However, they may run more slowly, given the situation. The rest of the network and infrastructure has taken up the slack and is facing a substantial load.
[BACKUP] How do I migrate my SBG IP to another data center?
You can migrate your failover IP linked to your server originally located in Strasbourg (SBG) to another server located in France only.
[BILLING] The replacement service I wish to order is not available. When will it be available and how quickly will it be delivered?
We are prioritizing orders from impacted customers and our production lines have been multiplied so that they can be processed as quickly as possible.
You can follow our increased production numbers using our Incident Status page.
[BACKUP] When will my infrastructure be available again?
Our teams are working hard to accurately assess the time-frame for the restoration of our data centers, based on the audits in progress. You can track this progress here as we post daily updates.
To stay informed of this information please monitor your emails for direct impacted customer communication and track on our Incident Status page.
[BILLING] How do I know if I'm eligible for a refund on the server I have ordered, and when will I be refunded?
If you were affected by the incident in Strasbourg (SBG) you should have received an email from OVHcloud US informing you of your eligibility to receive a free credit depending on the status of recoverability of your impacted services.
For your most critical services that require a quicker restart, we recommend that you order one or more alternative solutions in one of our other data centers, where we are increasing our capacity. In this case, we can help you to apply the free credit to any new services.
[BILLING] I received a credit, what can I use this for in my account?
Credit vouchers on OVHcloud US accounts can be applied towards the purchase of any net new service as well as towards the renewal of an existing service.
- To use the credits towards a new purchase - customers will need to contact our Sales Department. Sales can be reached during business hours via chat on the front of our OVHcloud US website or at the top by clicking 'Contact Sales' in the top header.
- To use the credits towards renewals of existing purchases - customers do not need to take any action. By default our renewal system will look for credit vouchers before looking for a valid credit card.
[SECURITY] What security risks, if any, should be monitored following the incident?
Scammers could take advantage of the incident that occurred in our Strasbourg data center by carrying out several malicious actions: spam, phishing, telephone solicitation, suspicious web links, identity theft, viruses, etc. We recommend remaining vigilant, and following all of the best practices for security listed in our guide: “Securing your OVHcloud account with two-factor authentication”.