Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Tom Harbin
July 17, 2004, 9:34 pm
rate this thread
I've been lurking hereabouts for several weeks and have learned a lot.
I've been doing web design on and off for about ten years but up until a
year ago it was mainly for intranets or "hobby" sites. To get to the
point, I have totally mucked up the standings for our web site with Google
thanks to the desire to improve the site coupled with a complete ignorance
of search engine technologies. The main culprit was the use of header
rewrites from the old pages instead of 301 redirects when I moved or
renamed a page. MANY pages were renamed as I moved to the use of SSI for
most of the important pages. The site gets a lot of direct traffic so I
did not notice the problem for many months. When I took over the site a
search on Google for most of our important keywords would find us within
the first page (is this what you refer to as SERP?) at least once,
sometimes several times. Now we are rarely in even the first half dozen
pages. We get a very small percentage of referrals from Google but we do
OK from Yahoo and we have a number of inbound links, magazine advertising,
and email subscriptions that keep us from going dry. I need to fix this
immediately and prevent future issues since I have hundreds of pages that
still need to be converted to other technologies, mainly PHP and SSI.
I have learned how to use a .htaccess file and have worked with our ISP to
be compliant with our TOS.
Now my questions:
I have redirect 301s for the main pages that have been replaced and for a
number of secondary pages that show up in the error logs. The list could
obviously get very long by the time I finish converting all of the pages.
How long should I leave a 301 redirect in the .htaccess file before it
would be safe to discard it?
Since the server is doing a redirect I don't believe I will have a way to
know if people and robots are still accessing the old link. Will I?
Do most of the robots "learn" the new link and drop the old link after a
visit if I use 301s?
We often have entire directories that get retired because we have decided
to not work with a particular client. Currently a user gets a 404 if they
try to access one of these directories. I am thinking about using 410s for
these links but don't know if there are any downsides. Are there? What
about if several months later we decide to re-implement the link. Will the
robots see the "new" link or ignore it based on the previous 410?
Thank you for your time,
I do not know, but it appears to be a long long time. :(
I am thinking that you can tell from looking in your we server log files.
Here's an Apache logfile entry showing a redirect:
126.96.36.199 - - [18/Mar/2004:07:27:25 -0700] "GET
/technical_library/silencers.shtml HTTP/1.0" 301 348 "-" "Mozilla/4.0
(compatible; MSIE 6.0; Windows 98)"
Unfortunately, no. It can take many months for the robots data to get
back into the search engine databases.
Let's look at the logfiles:
188.8.131.52 - - [17/Jul/2004:18:17:46 -0600] "GET /equipment/manufacturers.shtml
HTTP/1.0" 301 349 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
That redirect went into place on March 30th.
184.108.40.206 - - [17/Jul/2004:18:18:22 -0600] "GET
/technical_library/nuclear_war.shtml HTTP/1.1" 301 353
"Mozilla/4.0 (compatible; MSIE 6.0; W
indows 98; Hotbar 220.127.116.11)"
That redirect went into place on February 17th.
18.104.22.168 - - [17/Jul/2004:18:18:24 -0600] "GET /hollywood.shtml
HTTP/1.1" 301 344
"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)"
That redirect went into place on January 2nd!
I've sworn off site re-design.
<snip of a lot of good information>
I guess it doesn't matter if I can see the redirects (although I'm glad I
can)in the log if the crawlers never learn.
On second thought it does matter. I can use that data to track activity on
a given redirect. When the usage looks low, I can check how well the new
page is doing in the search engines and if it looks OK, change the 301 to
a 410 and forget about it for awhile. I would think the crawler would
eventually give up on the old link if it can no longer get a response.
Thinking about it it may not be totally gloomy. Since the crawler SHOULD
come in through index.shtml now (DirectoryIndex entry in .htaccess), it
should follow the current links and find all of the replacement pages.
Most of our traffic is from direct access anyway since most people that
like our site really like it and visit several times per week. We have a
pretty loyal user base and good word-of-mouth advertising. Of course we
are still after those people that are using the search engines and don't
know we exist.
I don't really have the option of not doing a redesign (will I guess I do
really but it's not my style). I'm the third webmaster for the site, the
site was originally designed using FrontPage, then the same team switched
to Page Mill. The second team did all kinds of different things. In all
there are at least seven different designs in use for the car pages alone.
There are nine CSS sheets, etc.. Site maintenance is a minor nightmare.
- » The "Freebie" SEO Site-Augmentation Bookshop Package
- — Previous thread in » Search Engines