The worst thing, in my opinion, is that there has been no communication from OVH directly to their customers. I am impacted and I have not received a single e-mail about the situation even now.
I am not directly impacted (not running anything that is advertised as physically there) but emails have been taking multiple hours to arrive for days. It would be nice to have a way of knowing whether I should change my MX servers or not.
I think it's fine for them to take some time to fix their infrastructure, but if they could let me know, I can easily work around it. Incoming email is kind of difficult to monitor and needs to work...
I'll want to know what is going on with a thing and find out "oh they mostly post that kind of news on twitter" or "oh this person uses instagram", or "that news goes out via email", or youtube ... or ... who knows where ...
Like guises come on, where am I supposed to get information when you're using like 8 different channels and not consistently...
I did get an email yesterday around 16h00 GMT.
It was pretty generic, the only new thing it contained (for me at least) was the FAQ regarding this incident.
SBG1 and SBG4: we are planning to bring these datacentres and the network room onlinenext week.
SBG3: we are planning to bring this datacentre online in 1-2 weeks.
I'm not an OVH customer, are these building names? It seems a bit confusing that a datacentre doesn't correspond to an availability zone or is this typical at large scale?
SBG is OVH Strasbourg, a complex of data centres that are adjacent to each other, but which were not built at the same time and are architecturally not the same. Yes, SBG1-4 are the different buildings. Some of them are made from stacked shipping containers, but SBG2 was an actual building.
SBG2 basically burned to the ground. The fire also spread into SBG1 if I remember correctly: parts of SBG1 were destroyed but some parts are intact. SBG3 and 4 are also intact. Under the circumstances the entire complex is offline at the moment though, which is not surprising given the scale of the fire and the firefighter response. The photos reminded me of the WTC attack, which shut down lower Manhattan for months.
SBG2 was built like a vertical tunnel, a quadrangle of server rooms with a huge air shaft in the middle, intended to convection-cool the servers through the natural motion of the rising air through it as the air was warmed by the servers. In this case it may have resulted in something like a firestorm, where the fire maintained its own airflow through that column.
I'd be interested to know if there is any hope and/or intention in this situation to recover possibly-operable hard drives from the charred servers for data recovery purposes.
I couldn't believe the complaints about the audio in that mp4. I could understand him perfectly well except a few times his accent got in the way. Better audio wouldn't have mattered much.
The video was badly prepared. He was talking about datacentre names as if the audience are his own employees and are fully aware of the different buildings.
He could at least have some graphics.
It certainly wasn't well prepared. But it did everything there is to be done in such a situation. An quick apology, details about the situation, and what is being done. I didn't have any problem understanding what was being said. I don't know the details about the buildings but I don't think it is necessary to understand the general idea. Datacenter caught fire and was completely destroyed, some damage in the surrounding buildings, old buildings are worse than new buildings.
A nice presentation would have been great, but if there is a choice, I'd rather have the CEO work on the problem than polish his presentation.
Edit: He also named some people. I don't know who they are, I can only guess they are employees, and while I don't really care personally, I think it is tasteful to name them since they did the right thing by intervening quickly and not being reckless.
OVH deserve points for being one of the only hosting providers I know that make their entire network weathermap available to the public, showing all their global interconnects and then letting you channel it down to a per-DC level: http://weathermap.ovh.net/#europe
Interesting. Are there popular tools to build such diagrams or is this a bespoke creation? ie are they using scripts+graphviz directly or is there some middlewear doing things?
And probably you as a customer are a main target of such video, but the fire was such a high-profile case (I think) that this should be more understandable for general audience.
edit: not sure why I got downvoted (and I don't really care that much), I just think that communication from CEO of big company after such event should be clear for everyone, not only for those directly affected.
But ... the video isn't for a general audience. He doesn't have any reason to apologize to a general audience. His customers are the only people that matter in this situation and this video is for them.
Honestly, nobody lost their lives, the impact is pretty much limited to OVH (who need to find a new building), their customers (who need to find new servers and restore backups), and the city of strasbourg (who run the emergency services and presumably will have some involvement in the cleanup). While we're all interested in a "Oh, are we prepared if that happened to our provider?" sense, I don't see why OVH is obliged to provide communications to us, rather than to their customers.
Had the same shit with them few months ago and also have troubles with load balancing.
I am on the step of choosing a new provider.
You still with OVH?
Maybe you could recommend something you really confident in.
I am choosing between Constellix (https://constellix.com/) and Azure (https://azure.microsoft.com/) now...
Pros, cons?
I'm glad they compartment their data centers. Having one data center for everything would be a big fail. I say this because you would be surprised at how many cloud services centralize everything in one spot.
Most cloud providers have availability regions and zones, with some services specifically having multi-region/multi-zone options (usually with extra cost). Is there any provider that hides this?
That's why you have multiple availability zones. Those are the physically separate locations that you have visibility into and can spread your infrastructure across. Having multiple data centers in a single availability without visibility into where your systems are is fairly useless. And if you do have visibility then you now simply have more availability zones so why bother renaming things.
