Details on Last Week's Ning Platform Instability

[Sridatta Viswanath]

We have always prided ourselves on our uptime record. In March, we had 99.999% uptime. However, Ning Networks experienced slowness last week and brief downtime on Thursday evening. I wanted to take a few moments to walk you through what happened last week and what we’re working on to improve on situations like this going forward.

Starting Tuesday morning we saw sporadic instability in our servers, resulting in slowness that many of you experienced. Our operations and engineering teams investigated a variety of potential causes for the instability including a back-end release that was pushed out on Monday. We rolled the release back as part of our investigation and continued to look for the root cause.

On Thursday morning we believe we identified and addressed the primary cause of the instability: we had started testing a prototype to deliver real-time information to Ning Networks on Monday. We stopped the test and continued to monitor the platform. At around 8pm that day a cluster of cache servers that had become slightly unstable during the investigation started failing. At this point, we began an unplanned maintenance to restart them and it took about 70 minutes to complete the restarts and go live.

Whenever there’s an issue on the Ning Platform, we pursue the following protocol: First, our on-call engineers begin investigating what might be causing the issue. Then, for larger issues like this week, we escalate it to all-hands on deck. We also strive to get information out to you as fast as possible through several channels including the Ning Status Blog, Creators Ning Network and Ning Status Twitter account. Longer-term, we are working on an NC landing page with an announcement bar for important messages and a separate, lightweight “Report an Issue” link, which will allow NCs to quickly and simply report an issue they may be experiencing.

Following this incident, I wanted to highlight two specific areas we have identified for improvement:
1 It took longer than we’d like to identify and fix the problem: To address this, we are continuing our work on a series of projects that will simplify the platform and improve our ability to identify problems quickly when they happen. I will share more details with you early next week.
2 -We were not as effective as we could have been in communicating both the state of the platform and what we were working on: To address this, we are reviewing our internal and external communication processes to ensure that we can give you the most accurate and timely information on what is happening.

Keeping your Ning Network online and speedy is our top priority. Thanks again for all of your patience. As always we appreciate your feedback, so please let us know if you experience any problems on your Ning Network or have ideas for how we can improve the service.

[Marilynn]

Hi Eric - thanks, I don't need more info, was just pointing out that it seemed to effect some of us more than the post said.
Thanks for explaining regarding the different networks. Hopefully the one I'm on won't be hit too hard in the future!

[Shevie]

Network behavior is quite sporadic. Sometimes it works great, sometimes it seems dead in the water. A couple of hours ago it was fine, 5 minutes ago I got the above error message, and just now it's fine again.

[Shevie]

BTW, I'm in the same network as WillyN(Bon).

[Shevie]

I just tried to access a group in our network and received the following message:

Our apologies

We're sorry, this social network is under maintenance. Please check out the status blog for the latest updates, or let us know if you get this message again.

[Ronald Buijs]

Thanks for the info, greatly appreciated. It would be great as during down town we can talk to our members in our own language. Is it possible to import the we are down sentence in the language editor?


Thanks

Ronald

[Addy]

thanks for the update sridatta.....appreciate it

cheers

[Marilynn]

Hi and thanks for the info - but I think it was more serious than what you were saying? My site was down three times and yet I would look at other NING sites and they weren't. Is there a reason some sites stayed on and others did not?
Great to hear it is sorted now.

PS I'm not angry, just curious :)

[celina[D]]

Although I was not effected by these problems I would like to say this is the type of communication I'm looking for. Thank you very much, this gives me a better feeling about Ning

[Amit S]

Happy to know the developments ! Crisis usually gives way to new beginnings ;-)

[Jen]

Sridatta,
Thank you for this update.

On your #2 Comments:
"2 -We were not as effective as we could have been in communicating both the state of the platform and what we were working on: To address this, we are reviewing our internal and external communication processes to ensure that we can give you the most accurate and timely information on what is happening."

Hold on to your seat belt. Because you will likely receive some unpleasant retorts. But, I for one, would like to say: It is great to hear the Ning Team be forthcoming and honorable about the situation. I am extremely happy to hear that you all recognize that this was an issue - - not-so-good communication, and you are assessing the situation and plan on improving this for the future.

For me, this blog post proves you are committed to this promise.

Thanks!
Jen