Incident on August 5, 2010

[Mike Tria]

For several hours, Network Creators and Members were unable access many
of their Ning Networks. Users were seeing 404 errors across a large
swath of all Ning Networks. I will explain what happened, how we are
fixing it, and how we are going to avoid the issue in the future.

What happened

Early this week, we were seeing issues when deploying new software to our
production environment. As you know, we have been furiously deploying
new features and enhancements to Ning since we launched Ning Mini, Plus,
and Pro. Specifically, we were noticing that some of our servers were
not functioning properly with new code. To resolve this issue, we made
some operational changes in how we maintain these servers. Part of that
change was deleting old software deployments. Unfortunately, one of the
side effects of this was an errant process that started deleting some of
your Ning Networks. Note however, that this script only impacted the
PHP code for some of your Ning Networks. It did not delete any content.

Our incident team was immediately engaged and online. Once we found the
root cause, we stopped the errant process and begin restoring your Ning
Networks.

How we fixed it

We keep backups of all your Ning Networks both hourly and daily. We ran a
script that restored the deleted code. This script was able to work very
fast and restored many Ning Networks in a few hours. As Ning Networks
were restored from the backup, they were immediately available online.

How we will prevent this issue from occurring in the future

We are avoiding this in three ways:

1. Strengthen our production review process. All production changes go through a review process at Ning. We make
   dozens of changes in our production environment per week without
   incident. We are further strengthening this review process with deeper
   reviews of all one-time scripts and long-running scheduled jobs/scripts
   in our production environment.
2. Perform an audit of all our existing jobs. This is something we do regularly but will be doing immediately in light of the incident.
3. Put safeguards in our system against this. Even if another errant script runs in our environment, we are putting
   safeguards in our system to prevent this kind of deletion or alteration.

[solo]

good job

[Patrick Chappelle]

Surprised there's no mention of the problems nc's seem to be having logging into the Help Center.

The 404 incident didn't affect my network, but it's good to know other nc's can now breathe a sigh of relief.

[Jen]

My CSS on a test site for a Client has not been restored. This is truly frustrating and makes me and Ning look bad, with my Client and your Pro Plan Customer.

Do we know when this will be corrected?
Thanks,
Jen

[Patrick Mee]

Jen,

Can you tell me the name of your test site so I can look at it?

Thanks,

Patrick

[Jen]

I'll need to private email this, due to Client disclosures. Or, will the ticket number help?
re:[ ref:00D8cCLt.5004BTtjJ
5004000000BTtjJ

Thanks Patrick! It's the one with the #4 in the URL. The #6 appears to be okay. If you are able to access this ticket information, notice that the test site for #4 has no advanced CSS, completely empty. If I didn't at least have 85% backed up, I really don't know how I could recover this work. It's a lot.

Thanks again!

[Jen]

Patrick,
I just wrote Support again. This is really ridiculous to go this long and have your work wiped out, and no fix. You can't imagine how bad this makes me and Ning look with this Client. I just asked the Support Team should I start trying to rebuild my work?
Best,
Jen

[Patrick Mee]

Jen,

I'm trying to work out what happened during this event and what the worst case scenario is. I am less familiar with the ops side of things.

Are you still having issues adding CSS to the #4 network? It's not clear from your ticket.

Thanks,

Patrick

[Jen]

Hi Patrick,
Not sure how it's not clear, sorry. I do not have the custom CSS that I worked on for almost 2 weeks. When I shutdown my computer night before last (technically 4am August 5th), I had made a good deal of changes to the CSS. I tried to back up the code as I always do. I kept receiving time outs, thinking it was my Internet connection.

The last support ticket correspondence I received stated that Ning Engineers had restored my CSS. It's not even close to my custom CSS. In fact if you look at the Advanced CSS code, it is completely empty.

So to summarize: my custom CSS has not been recovered. The box is empty.

Best Regards,
Jen

[Patrick Mee]

Jen,

I agree that the field is currently empty. What I was trying to work out if you are still having issues adding CSS. That is the part that is unclear to me.

Sorry for the confusion.

Patrick

[Jen]

I have not tried Patrick, waiting to see if Ning will recover my code that Ning removed.

I see what you mean about a comment being confusing - - when I tried to add the CSS I have backed up, via Firebug, it would not work. It was very odd. And I even removed code via Firebug that I knew could interfere.

I simply want Ning to restore my work as it was when I left it.
Thanks,
Jen

[Jen]

Hi Patrick,
Thanks for trying. I must move forward and try to salvage my reputation on this project. Please read the support ticket for my final comments. I'm definitely not impressed or happy with this situation.

Best,
Jen

[Ian Sullivan]

Mike,

I think I may have found a bug in the process.
Can you take a look at my other post:

http://creators.ning.com/forum/topics/unable-to-leave-comments-on

Other than the possibility of another strange side-effect of the process you were describing above, I've no idea on why in the past week an a half after upgrading to Ning Plus, none of my Network Members can post comments on anyone's profile page except their own.

Thanks.

Respectfully,

Ian
MSSpace.org (msspaces.ning.com) Admin