Robots.txt file restricting access to my Site

[Strength Performance]

How can I get Google (and all search engines for that matter) to index all pages within my Ning site www.strengthperformance.com?  Right now, non of the content located on the pages within my site are appearing in Google searches.

 

According to Google Webmaster, most of the links (pages) on my Ning site are restricted by robots.txt.  Content across all pages on my site is visible to non-members.  The robots.txt file is on the default setting within my Ning settings.

 

I understand that a robots.txt file restricts access to my site by search engine robots that crawl the web.  I know that I need a robots.txt file only if my site includes content that I don't want search engines to index.

 

I also understand that if I want search engines to index everything on my site, I don't need a robots.txt file (not even an empty one).

 

How can I get Google to index all pages within my site?  Thanks!

 

[Phil McCluskey]

Hi there,

Realistically you don't want Google to index all your pages. You don't want it to try to index your sign in and sign up pages, and you don't want it to index search results pages. if you have a look at the kinds of pages that are restricted by robots.txt, you'll see that for the most part, it's endless variations of the sign in and sign up pages that will forward to other pages in your network after the sign in or sign up process is complete. You don't want those in Google.

Right now it looks like you have about 3,300 pages of your site indexed by Google. 

[Mochajewl]

Hi Phil, I'm new to Google ads and I am getting error messages on Google Adsense, stating that "crawler could not access my pages" in order to advertise.  I have a 2 sentence robots.txt code that I was told I need to insert on my network.  Do I? if so, where?  Although I don't want open access to my network pages, however, I do want to run Google ads effectively on my network.  Are you able to check my network settings and assist me with this, please? 

[Strength Performance]

Thanks.  Makes sense.  I recently (5 days ago) opened up my Ning network so that everyone can view all content on all pages of the site regardless of whether or not they are signed in.  I did this because I was told that the pages would not be indexed by Google if they were restricted to members only.  Is this true?  DId the 3,300 pages you listed below just start to appear in Google search when I opened up the site to everyone?  Or were they there even when my site was restricted to members only?  I hope that by opening up the site to all visitors that my pages will pop up in Google searches more often.  Will this occur?

Thanks again.  Really appreciate your feedback!