SEO Problems and Crawler Indexing Issues?
Hello after being away from here for a while I came back because I am having a possibly major issue with my website and search engine traffic. It seems bots and crawlers are having issues with crawling my website. I have summed up most of it in the quote below.
I apologize for the links as this quote is just a direct copy/paste off of my forum.. I just recently purchased an IPB license a little over a month ago and finally made the move to IPB, but this issue seems to have been going on longer than that.
Quote:
I think something is wrong somewhere or broken that is keeping search engines from being able to properly index the website. We technically should at least be seeing a fair bit more guests on at once even if they don’t register given the age of the site now.
As most we get between 10 – 30 guests on at once, sometimes less, half the time it’s the result of a crawler, crawling the pages..
I posted this poll to see how many found the site via a search engine, and only one person/member found the site via a search engine…
http://www.xtemu.com/forum/?showtopic=763
I would appreciate if someone familiar with SEO can help check if something is wrong somewhere, check meta keywords and descriptions and other general SEO stuff to help me optimize results more efficiently.
Also the web archive wayback machine should typically be showing more archived pages/cached versions of the forum than it currently is. This has led me to believe that something somewhere is breaking search engine crawlers from the ability to properly or efficiently index the pages.
http://web.archive.org/web/*/xtemu.com
http://web.archive.org/web/*/http://www.xtemu.com/forum
The wiki brings up no matches on the wayback machine..
Something is wrong somewhere and given the date of the last cached pages on the wayback machine, might give an idea what could of happened or went wrong.
As you can see one member total came from a search engine on the poll, barely any content for my website is being archived in the web archive wayback machine with five pages total that have been archived. Barely any guests on the site at all, at most we get between 10 and 100 guests, but half or more of guests are made up of crawlers, unknown crawlers so relatively few guests on at a time.
Also whenever a crawler is on the site, it usually sits on the board index not crawling any of the pages or content at all. Also my sitemap generator script is having trouble crawling the site, the end result of a sitemap is unusually small like only a couple hundred kilobytes which is quite small given the amount of content, topics and posts we have.
The pages themselves total about 7MB. So I have came to a conclusion that something is wrong or broken somewhere that is making it difficult or nearly impossible for search engines to properly index the site/forum, which also hurts search engine results for my site. As to what it is that is causing this problem baffles me and I am unable to fix it myself without an idea on what it could be.
Thanks if anyone can help me.
Well, if you type
site:www.xtemu.com
into google, it shows 3,570 results, so it seems everything is ok. I wouldn´t worry about the wayback machine, they generally do not cache everypage on the site and is not used as a benchmark for indexing content.
However, you have no meta descriptions. This is the snippet of text that google shows in its results page. This could be affecting click-through rate, as in the absense of a meta description, google will show a snippet it thinks would be good, which more often than not is not very good. Think of your meta description as an advert for a particular page on your site, it should give the searcher an overview of what the page is about.
Also, your meta page titles are bad, perhaps you should add some keywords to do with your community into the metatitle, and this will get you showing up better in searches for these keywords.
That´s just my opinion after a quick 2 minute look, however I would recommend hiring a specialist if your are very worried.
Well, if you type
site:www.xtemu.com
into google, it shows 3,570 results, so it seems everything is ok. I wouldn´t worry about the wayback machine, they generally do not cache everypage on the site and is not used as a benchmark for indexing content.
However, you have no meta descriptions. This is the snippet of text that google shows in its results page. This could be affecting click-through rate, as in the absense of a meta description, google will show a snippet it thinks would be good, which more often than not is not very good. Think of your meta description as an advert for a particular page on your site, it should give the searcher an overview of what the page is about.
Also, your meta page titles are bad, perhaps you should add some keywords to do with your community into the metatitle, and this will get you showing up better in searches for these keywords.
That´s just my opinion after a quick 2 minute look, however I would recommend hiring a specialist if your are very worried.
References :