if you were skeptical about the importance of a proper sitemap read this

[soaringeagle]

over the past couple months ive had a hard time keeping my sitemap updated with random issues interfering in the update process.

then superstorm sandy knocked out my power and messed up my pc (i had managed to get 1 update almost complete just before the power went out that was the only update in several weeks at least)

after that  my ram use climbed to 995 and stayed there so i upgraded to 32 gigs and it still  used all my ram  so the next few weeks i had tried to do updates i got 1/2 way through and had to reboot.

so finaly i reinstalled windows and everything today and am finaly on the way to a good update

so heres what ive noticed since not being able to update as often as i should

 

googlebots activity on my sites down by 505

black arrows are peaks after the last 2 complete updates there was no peak after the incomplete update when sandy hit most likely because there were fewer urls in the sitemap then the previos update (bout 200k less)

red arrow is the average activity level when i am able to update regularly

 

now how its affecting trafic;

my trafics been on a steady rise slow but sure

well its just taken a huge dip 1000 less visits a day just the past few days

dropping below levels it hasnt been below in the past 6 months

(no graph included just cause its hard to see the trend unless you know what to look for)

 

ps the sitemap generator i recomend is Sitemap Creator 4if you have a windows machine to run it on

also interesting point  during  the same time period the number of number 1 ranking pages and pages on page 1 of google results ..varied but the averages didnt change enough to explain the drop

however  the number of pages indexed  did go down by a big factor (200k) and the freshness of pages was reduced (meaning pages showing up in top 5 positions within an hour or 2 of posting)

 

i cant stress enough the importance of an updated fresh  and proper sitemap

its the cheapest easiest seo you can do

it wont replace on or offsite seo work but should supliment it and  will give ya the most bang for the buck

since its a 1 time charge to get the software then u just run it over and over and watch the trafic increase

you can spend hu8ndreds a month  on seo  doing page optimization and link building

but see very lil and extremely slow results (especialy with link building)
but having a freshly updated sitemap  and having it ping the search engines when its updated will get those bots to come running to see whats new  and get it indexed faster  and more dependably.

 

most likely you will see a 505 increase within weeks  and a steady increase from there  as well as faster indexing (alreasdy very busy sites might not see 505 but should still see a noticable increase)

 

 

just a few more words about sitemaps;

• ning sites are huge  a brand new site with almost no content would be around 2k urls the ning sitemap for that site might list 3 or 4..maybe 5 the max any ning sitemap would list is around 10 urls...only the features (forums blogs photos) nothing within the features a "proper sitemap" listes every single page you want crawled (every page not blocked by robots.txt)
• creators is huge 24,000 members and so many discussions added a day more then 1/2 never get a reply google lists 300k pages of creators my site is under 10k members  its active  and big but nowhere near as active and fast moving as creators.. google indexed 3 million pages of my site
• imagine going to a city youve never been to, a million or more criss crossing streets  zig zagging  every wich way now imaging this entire city with  5 million streets 100 million addresses and only 4 road signs  can you find the coffeehouse you heard about or  the musuaem or hell can you even find your way home or out of the city for that matter? a sitemap is like a map of thesite or city it tells the bots where to go for the best treats where to avoid  unless looking for a seedy dark back alley, where te attractions are and whats not at all important to see

since upgrading to Sitemap Creator 4with the new video sitemaps ive only run it once and that was the incomplete run but in the past couple weeks ive noticed twice as many vids indexed and several dozen more indexed daily (i have vids set to a semi low priority and change frequency since its a way lower activity on y site ..50k photos and aboput only 2k videos so set pics to 0.9  change freq always and vids to 0.3 and weekly

 

note the vids were indexed as pages already but not as vids

so including all content types in the sitemap not only every page but  every file embeded in every page etc makes a big diference

 

now that my pc is runing better again ill be updating my sitemap weekly (takes about 5 days each run) once i have several done fairly steadily instead of just now and then my trafic should bounce right back and increase again  tye google bot activity should increase as well

and once again should see pages posted show up at the top of searches within minutes to hours instead of days to weeeks

(when the site was smaller and it only took 24 hours to run an update i had seen pages show up in top 3 positions within 3-5 minutes even)

 

pss (i know i rambled alot) see my older sitemap tips and my google domination tips for how to do a proper sitemap on ning since nit only does nings sitemap fall very short from being a proper sitemap, but ning also makes it harder to do one properly..in fact Sitemap Creator 4 has a url replacement feature (the only sitemap creator i know of that has it) that i had them add to it just to make it work with ning (no root access means you have to use a subdomain or other domain and use url replacement to 'fix" your sitemap indexes)

 

[Anthony]

I agree, the traffic is not great.  Something wrong with the schematic infrastructure of the network design. For starters, the forwarding of the domain from mysite.ning.com to mysite.com is a problem with Google webmaster, and secondly, having so many similar network on the same server, or root domain can't be good. 

If one site got banned from Google, it probably affected the other networks, specially if that network is a porn network.

Prior to the sitemap you introduced, Google would only indexed 80 pages, and everyday I have to sort out errors. While your sitemap indexed 80,000 pages, they are not ranking high, they seems to be at the bottom of the search page.

Most of our pages are SEO...Now. My Google Webmaster showing over 540 server error message,  and 8,777 not found pages. All these server error, and page not found can't be good for the network.

I think the only way our network will progress is to be on a open source scrip which we working on.

[soaringeagle]

dont worry aboit those thopse are normal when the server goes down or into maintenance mode  or discussions are deleted or people leave or get baned your gouing to have those

no theres no issues with the forwarding  or the domains on the same ip

yes havingyour own ip gives u a tiny tiny advantage but its so lil to not be worth mentioning

if you say the pages are seo maybe the seo youve doe isnt working or is working against you?

google bans by domain

ip only i believe if theres malware or spamming widespread by that ip

[Manuel France]

Hello,

How do you indicate into webmaster tools the sitemap link???

I got "www.domain.com/"... already written...

Thanks a lot

[soaringeagle]

you would verify the subdomain and do it there also in your robots.txt put this line 1st before any useragent
Sitemap: http://sitemap.yourdomain.com/sitemap.gz (or xml) the url of your sitemap

[Manuel France]

24 hours for 300.000 pages... it's gonna take a week to crawl... but I'm excited to test the effect of this sitemap ;-)

[soaringeagle]

thats actualy really fast are u getting timeouts? i crawl at 200k pages in 24 hours and thats bout fastest i can go with accurate results

[Manuel France]

No timeout... as I''m writting : 32:40:00 -> 411700...

[soaringeagle]

ok lil fast but not too bad how many threads and what crawler delay

[Manuel France]

I've set everything like you show on your post image sitemap...