For several hours, Network Creators and Members were unable access many
of their Ning Networks. Users were seeing 404 errors across a large
swath of all Ning Networks. I will explain what happened, how we are
fixing it, and how we are going to avoid the issue in the future.

What happened

Early this week, we were seeing issues when deploying new software to our
production environment. As you know, we have been furiously deploying
new features and enhancements to Ning since we launched Ning Mini, Plus,
and Pro. Specifically, we were noticing that some of our servers were
not functioning properly with new code. To resolve this issue, we made
some operational changes in how we maintain these servers. Part of that
change was deleting old software deployments. Unfortunately, one of the
side effects of this was an errant process that started deleting some of
your Ning Networks. Note however, that this script only impacted the
PHP code for some of your Ning Networks. It did not delete any content.

Our incident team was immediately engaged and online. Once we found the
root cause, we stopped the errant process and begin restoring your Ning
Networks.

How we fixed it

We keep backups of all your Ning Networks both hourly and daily. We ran a
script that restored the deleted code. This script was able to work very
fast and restored many Ning Networks in a few hours. As Ning Networks
were restored from the backup, they were immediately available online.

How we will prevent this issue from occurring in the future

We are avoiding this in three ways:
  1. Strengthen our production review process. All production changes go through a review process at Ning. We make
    dozens of changes in our production environment per week without
    incident. We are further strengthening this review process with deeper
    reviews of all one-time scripts and long-running scheduled jobs/scripts
    in our production environment.
  2. Perform an audit of all our existing jobs. This is something we do regularly but will be doing immediately in light of the incident.
  3. Put safeguards in our system against this. Even if another errant script runs in our environment, we are putting
    safeguards in our system to prevent this kind of deletion or alteration.

Views: 49

Replies are closed for this discussion.

Replies to This Discussion

Hi Eric,

Please let me know the promblem I commented above is a part of ning issue, if you can confirm I can patiently wait to see ning come back with normal service as soon as possible and can take in confidence my members who are talking against.

Please note I have submited a ticket on this, We started getting problem after doing domain mapping.

Patience is art of hope , your reply will give me and my members hope.
You may want to create a new discussion so we can answer you specifically. Many of these comments here are not related to this actual discussion. It is hard to answer every off-topic post effectively.
Eric,
I have created a new discussion with my concern
Link :http://creators.ning.com/forum/topics/my-network-is-down-since-i
I expect ning teams reply there

if you need you can delete my postings in this forum or if you say i shall do it.
Please let me have a reply from ning team about my site, iam loosing....
My site is still not accessible!!!! www.openreel.ning.com

Very frustrating as I have had no reply from either of my tickets over the last 2 weeks now!!!

Please do something or you will lose a customer!
Hi, Openreel. Your situation appears to be a little different than what this discussion focuses on. You may want to open a separate discussion.
Did not affect my site. Thanks for the information.
Hello, I've been trying to contact in regards to the error 404 problem. Finally my account was approved so I can post on the board.

Three of my networks are still down. Is there a way to get them up sooner? I upgraded to the pro account so it's too bad I am faced with this problem. And two of my clients are really bugging me and I don't know what to tell them.

My accounts are:

www.your240sx.ning.com (www.youdrift.net)
www.jerdevelopment.ning.com (www.jerdevelopment.com)
www.ithgraplingleague.ning.com (www.ithfights.com)

I really hope the problem can be resolved soon. I really enjoy the ning platform, it has been great so far.

-marco

RSS

What's Up With Ning 3.0?

Click here to share your thoughts on what features we should develop next! - Survey will close Friday, 4/25

Projects that the Ning 3.0 Team is currently working on: 

Articles - a new way of organizing longer content pieces. Released 4/16.

Activity Feed Updates - options for more content in the activity feed.

Forum Notifications - ability for members to opt into receiving notifications of new discussions in forums. Released 4/9. 

APIs - The ability to create new members and content programatically.

For more information on what's next, visit the Product Roadmap

Latest Activity

Fire-Tech replied to Phil McCluskey's discussion 'Introducing Email Notifications for Blogs and Forums'
"Hey Anam. What theme are you using?"
1 hour ago
Evandro Dalben replied to Thiago Santos de Moraes's discussion 'Exemplos de comunidades inspiradoras usando o Ning 3.0' in the group Criadores Brasileiros - Brazilian Creators
"Estou a mais de 2 meses solicitando a migração e nada ... tomem cuidado pois fui…"
1 hour ago
Evandro Dalben replied to Evandro Dalben's discussion 'Suporte NING é bem ruim hein ?' in the group Criadores Brasileiros - Brazilian Creators
"Obrigado pela sugestão, estou aberto a novas pois irei remover minhas redes do ning uma vez…"
1 hour ago
soaringeagle replied to Monica Diaz's discussion 'MIGRATING TO NING 3.0 STATUS [UPDATE #2]'
"you most likely never responded to the migrzation confirmaztion email thats whats stopped most…"
1 hour ago
Evandro Dalben replied to Monica Diaz's discussion 'MIGRATING TO NING 3.0 STATUS [UPDATE #2]'
"I have been requested this since Feb 13 .. we are in April 19 and NING didn't migratred my…"
1 hour ago

NC for Hire
SweetPotato replied to SweetPotato's discussion 'Tip: Insert Adverts in Content after the First Paragraph'
"Glad it worked. SP"
5 hours ago
francesco pillinini replied to francesco pillinini's discussion '3rd Party Integration > Facebook'
"Elshara Thank you very for replying to my message still can't see what im looking for on…"
5 hours ago
Kevin Kelly replied to Chris Purington's discussion 'Paid Memberships & Recurring Payments in Ning 3.0'
"That would be a great feature to add.  There's bound to be lots of small membership…"
7 hours ago

© 2014   Created by Ning.

Badges  |  Report an Issue  |  Terms of Service