The Google Problem, or why I’m starting to use MSN and Yahoo more.
This weekend has been a bit of an introspective for me on why google is still the primary search engine I use. I know, I’ve been a big “fan(?)” of google for quite some time, I’ve obviously incorporated many of their products into my pages and used Google for 99% of my web searching. In recent months though, I’ve certainly had frustrations from the “site owner” side of the Google relationship. My North Carolina Genealogy site had traditionally been hosted as a subdirectory of averyjparker.com and had always enjoyed the lions share of traffic, so when I gave it it’s own domain, I did a 404 page not found for those following outdated links and I added an automatic 5 second redirect to the northcarolinagenealogy.net page. I soon learned that was a mistake, as the site vanished from Google around the first of December.
It was a dissapointment. Not a disaster, but a great frustration. Realistically, it had been earning maybe 50 cents a day, maybe as much as a dollar, but not a ton of money. What was more frustrating though was the fact that now it’s next to impossible for me to find pages on my own site. (And of course, the “if you’re not on google you don’t exist” feeling.) I used Google site search which HAD done very well for finding pages within the site. Anyway, I corrected the 404 redirect and did a reinclusion request and have waited and waited. I saw a brief glimpse of re-emergance last week. The site had pagerank of 4 according to the google toolbar and had a few pages indexed. Wow, I thought, it’s finally coming back. I checked my logs a few days later to see if it had been crawled… it HAD. I did another search to try and find a post on a family I’ve researched, with a name that’s somewhat unique. No pages found?? ok… so I returned to a site:northcarolinagenealogy.net search to see what pages were indexed. NONE so, it’s fallen BACK OUT of google. (heavy sigh)…. (Yes I have a site map and there are no reported problems with it.)
So, that was one area of frustration of the last 6 months, but not the only one from a “site owner” point of view. Here’s the next. I was checking (after the Big Daddy update) to see if a few of the sites I host had gotten in the index. I found that the 3 that were on a shared IP address with my hosting provider were in the index but with INCORRECT cache information and site summary. VERY bad. So, I contacted google. About 2 weeks later I got a response to please check and ensure the text is not on the pages and, for that matter “we couldn’t find the domains you mentioned”. So, I looked – sure enough they were gone, vanished, vaporized from Google. After this, I learned that my hosting provider has tagged this as a bug at THEIR end, although it’s related to the way Google is now crawling (keeping the connection alive, moving to a different domain, the shared hosting server sends up the previous page…. or something along those lines.) What’s odd to me here is why NO other search engine seems to be showing this problem. But, ok, we might spot google a gimme here and say it’s the software of the hosting provider. In my discussions with my host, two of the sites were in the index (out of three) and those two got upgraded to fixed IP’s, the only thing to do is wait for a recrawl (who knows when…) The third site, re-appeared sometime in the last couple days (and has the correct cache information.) While, the incorrect site showing up may be a problem at the hosting company, the VANISHING and reappearing without good reason is certainly something at Google’s end and is frustrating the daylights out of me.
Here’s another “site owner” frustration…. I started a site onlineradiotv.com in January. I had such a list of stories specific to either shortwave radio, or getting the broadcasters (shortwave or tv simulcast) online, that I thought it deserved it’s own site so that this site wouldn’t be cluttered with what many would see as not relevant posts. So, I registered the domain, got things going and it’s still not indexed in Google. (According to my toolbar it has a pagerank of 1), but it’s just not indexed. It’s very thoroughly indexed on Yahoo and MSN though, as are all of my sites. In fact, it was well indexed with those two search engines within a week or so of starting out.
Ok… here’s another “site owner” gripe…. January was a fantastic month Between 600-2600 visits per day and I might saw some good adsense earnings. Between February 1-4 those visits plummeted to a maximum of maybe 200 a day. (minimum of 60). So, I just had 10% of the traffic I was getting previously. For a few weeks I had too much on my hands with other issues to investigate what might have happened. But, I soon did and discovered Big Daddy. Google had done a major “infrastructure” update (supposedly not an algorithm update?), it was called big daddy and many said it greatly improved search engine results. I was pleased to see my pagerank had gone from 2 – 4 on the main averyjparker.com page, unfortunately, on most of the search terms that had brought me as many as 2000 visits a day, I had dropped from page 1 to page 4 or worse. Almost overnight. Ok OK – so they are weighting things a bit differently not a big problem, it hurt MY results, wahhh. If it were JUST a drop in my search engine results I wouldn’t be quite as annoyed as some of the really peculiar results I’ve seen. Doing a site search in the last week, (site:averyjparker.com) brought a cached version of my page up which was almost a year old (June 2005). Today, site:averyjparker.com presents me with a November 8, 2005 cache and searching for site:www.averyjparker.com get’s me a cached page that’s May 7th. I thought they were fixing the canonization issues with this update and …. I wouldn’t have ever known about “canonization issues” if I hadn’t run across it while investigating the northcarolinagenealogy.net evaporation of December 2005. At that time, I did 301 redirects on all domains to rewrite EVERYTHING to WWW.averyjparker.com a redirect that Google was supposed to recognize, but doesn’t seem to. For both of those searches it’s frustrating to me that the “main” domain comes in second to an article in the search results. What’s even MORE frustrating though is the difficulty I have finding my MAIN averyjparker.com in a true search. IF I go to google and type in “avery j parker” (without the quotes), as I suspect anyone I’ve given a business card MIGHT do…) the first result is southcarolinagenealogy.org which IS ONE of my sites, but I would have expected averyjparker.com would be more relevant, given that both sites use essentially the same template and the MAIN difference is that averyjparker.com actually has my name in the domain name. After 2 results from my south carolina site, the next thing I see is…. an entry from “kellysearch.com” which gives a link to my page (a link to averyjparker.com is more relevant than averyjparker.com itself???) What’s worse from a business standpoint is that the REST of the contact information is NOT correct at kellysearch.com. I would have to look, it may be that of my hosting provider as they are in Utah, but …. the entry is supposedly for me… (not googles fault obviously). After that entry, the others devolve into pages where avery j parker do NOT appear together on the page, but pages where avery may be seperated by several words from j parker, etc. By the bottom of page 2 and 3 I start seeing sites where I’ve either been referred to (and linked) from another site, where I’ve posted to a forum, pingbacked on someone else’s site, etc…. It is actually page 5 before averyjparker.com is listed in a search for avery j parker. What’s worse… a secondary page in the site is the “main result”, the main domain link is a secondary result. Actually, enclosing “avery j. parker” in quotes improves the results a bit. South Carolina Genealogy is still the top result, and the places where avery and j parker are seperated are filtered out, so you essentially get sites where I’ve either commented, posted, pingbacked, or someone has mentioned me or one of my sites…. but averyjparker.com is still MIA on page 1….After a few pages of sites that link to mine, I finally show up on page 4. Well… on the above, let’s see how MSN and Yahoo do… MSN search, avery j parker ( no quotes), top results averyjparker.com and one sub-page, northcarolinagenealogy.net and a subpage, southcarolinagenealogy.org and a subpage and then a couple sub-pages from onlineradiotv.com. (And then a site from averyjparker.com with a different subdomain that I forgot was still there.) With quotes it’s essentially the same list. (The only exception is spots #9 and 10). On yahoo, the search without quotes gets me averyjparker.com followed by a subpage, then norhtcarolinagenealogy.net and then a southcarolinagenealogy.org subpage, northcarolinagenealogy.net subpage, southcarolinagenealogy.org main page. With quotes, the yahoo search is the same except for places 9 and 10. This is the kind of consistency I used to see from Google, maybe a little variation here and there, but I could at least consistently find my own site without having to resort to site: advanced search bits. Obviously of the above, I like the msn search results as being the “most relevant” to this search.
I spent a lot of time musing over this Friday and Saturday, thinking back over the last few weeks. At one point, I saw google return 1300 pages from averyjparker.com and another in the range of 300. Another reason that this above really annoys me is because the last few weeks as a search engine user, I’ve been more and more frustrated with the search results I’ve seen from google. Many of the articles I do on this site, I do as reference for myself as much as anything else. That way I can go to google and type in “search phrase that I know appears on my site” and search and usually, I’ve been good enough at knowing HOW to search with google that I can find my page in the top ten and reference my article in a fix. This way I don’t have to remember EVERYTHING, just the way I would describe an issue…. For example. “windows desktop upside down fix” (Without the quotes). I have an article on this that SHOULD be indexed in google, I’ve referenced it to someone before. But, it doesn’t show up in the top ten, I added site:averyjparker.com and it STILL doesn’t show up, even site:www.averyjparker.com it fails to appear. If you do a search at yahoo [windows desktop upside down fix site:averyjparker.com] I immediately find my article. (Without the site: directive it’s past page two on yahoo…) On MSN, as well, the site: directive gives the article, without the site directive though my article shows up 5th with is nice, but I’m much happier about being able to find it at all.
But, I’ve thought, maybe my site is appearing spammish. I’ve got sitemaps for all the sites now and there are no messages there of anything to “clean up”, although I understand that they may not ALWAYS want publishers to fix content. So, the webmaster guidelines is all I have to go on. I can’t seem to find any reason my sites are being penalized. And I have seen many sites that will just post a single link to an article elsewhere, no other text and ads… Then, it’s not just about my being able to find MY OWN site, but being able to find other sites as well. I spent 3 days and I don’t know HOW MANY varaitions on terms that I KNEW existed on a page that I referenced some months back. The way I phrased the search, they should have been easily found, but it took more permutations of my terms (and who knows they may have been vanishing/reappearing as well.) I finally found the site, but after much frustration. One of the things about doing computer tech support. For ages, I’ve used google as the massive knowledge base of everything, but my ability to find things the last couple of months has felt hamstrung. I’ve run across more pages that 1) only describe a problem and no solution (unless you register) a tendency that 2) forum posts where the problem is repeated in the “thread” title several times throughout the page and sites that 3) have the search terms and ads, but not much else…. now #3 seems to be much improved the last 2 weeks. FOr that matter, maybe it’s my imagination and I just have a bit of “sour grapes” given what’s happened to my search results lately with google, but… I’ve spent a lot of time Reading threads like this discussion in the google sitemaps group, It was titled Reports of index coverage changes–we’re listening!, posted May 10th and there are now 134 posts. (Which isn’t many in the big scale of the web, but… consider how many people pay attention to their search rankings? For many I think they just call up their webmaster when traffic on the web site plummets and they aren’t hearing from anyone anymore… or when they NOTICE that and get around to mentioning it to someone.) Of course, there are 183 posts in a thread entitled Post here if you have lost index pages.
Among the common threads that seem to have been ascertained in one of the above posts is that there seems to be a common issue of canonization problems (treating www.domain.com as different from domain.com), site:domain.com is not consistently returning ALL pages in the site now. One of the symptoms one person pointed out is the main domain site showing up behind a sub-page in the rankings and there is a mysterious disappearance of backlinks. This lead some to speculate that maybe there is something broken with googles backlink calculation now. CIO Insight is asking if Big Daddy is “choking” google…. And essentially, for me, this has been going on to some degree for at least 3 months now.
So, I reflect back on all of this and say. In spite of the fact that I do web site design and development I don’t really want to HAVE to think about “canonization issues”, backlinks, pagerank. I just want my site to be found, by myself and others and I want to be able to easily find the information I’m searching for. I’d also LOVE to be able to do a search one day after another and get at least RELATIVELY similar results. From a search engines users point of view, I feel like I’ve been rolling dice when I’ve gone to google lately because the results have been so unstable. So, that’s why I’ve started using MSN and Yahoo a bit more, right now MSN is the preferred of the two for some reason. I still hope that google can get things sorted out and get back to a point where I feel like they’re as useful as they once were as a search engine. From a site owners point of view, I’ve not had to THINK about how to design my site to do well in MSN and Yahoo searches, it just seems to have worked, they’ve indexed within a week (even a brand new domain) and haven’t seemed nearly as fickle as google. Unfortunately, that doesn’t make people start using them more to find my site, so I’m still in light traffic mode, but I may be changing the site search function very soon across all of my sites. I know so little about msn search that I am not sure IF you can do an MSN site search on your own page, I suspect so and will be looking. In other words, I’m tired of trying to figure out WHY google doesn’t like a site and then trying to fix it, it’s sapping my time and energy.
It’s interesting though, I’m really thinking that they way they do search really MAKES a market for SEO’s (Search Engine Optimizer’s) and really makes the market for people trying to design pages to a search engine instead of relevant pages for site visitors.
Anyway, lengthy complaint/whine session/whatever you wish to call it/ over…. back to other tasks.
–update–
As you might notice, I’ve added an msn site search on the averyjparker.com search page and will soon be adding that to the others. (I’ve left the google search on there.)