By Robin Nobles
08.25.03

Congratulations! Your hard work and persistence in optimizing your pages have resulted in a dramatic increase in traffic to your site, which is just what you wanted.

But, are you analyzing that traffic and using it to strengthen your site? Or, are you quickly glancing at your log files every week or so to see how much traffic you're getting, and letting it go at that?

Before we go on, let's define exactly what log files are.

When someone visits your website, server software counts and tracks, i.e. "logs", that visit. It also keeps a record of it for a certain period of time. Part of the saved information is called a referrer log.

Referrer logs can help you analyze the traffic to your site. Though each referrer log program provides slightly different data, some of the more common information includes:



- Which engines have sent you traffic;

- What keywords were used to find your site;

- Which pages were accessed the most or the least;

- Who are the visiting spiders;

- User profile by region;

- Average length of time someone remains on your site;

- Average number of user sessions or page views per day;

- Top entry and exit pages;

- Top referring sites;

- Summary of activity by day;

- Server errors;

- Bandwidth, which is the measure (in kilobytes of data transferred) of the traffic on the site; and,

- Type of technology used by your visitors.

But why is it so important to study your traffic? Isn't it enough to know that your traffic is increasing, without having to spend valuable time analyzing it?

Think about it this way. If you know which engines are sending you the most traffic, you can boost your optimization strategies for those engines by creating additional pages for other relevant keyword phrases. This could increase your traffic even more.

Or, if you know that you're not getting any traffic at all from a particular engine, you'll be able to consider strategies for findability on that engine.

Through your referrer logs, you'll probably discover that you're getting found through keyword phrases that you haven't even considered before. In that case, you certainly don't want to change those pages and lose the traffic. By the same token, if you're getting found under a keyword phrase in one engine, wouldn't it be worth creating pages for the other engines for that same keyword phrase to see if you can bring in some additional traffic?

You can also find out through which pages you are losing visitors. This begs the questions...why are you losing visitors? ...and what changes can you make to keep them from leaving?

Simply put, a referrer log can give you an enormous amount of information and can serve as a road map for future changes to your site.

So, how can you view your referrer logs?

Ask your Internet Service Provider (ISP) to provide referrer logs in extended log format. If they don't provide referrer logs, you're missing out on some extremely valuable information about your web site. You may even want to consider changing ISP's.

However, even if your provider captures referrer information, you may want to get a program to read it, since the raw data can be a little cumbersome to analyze.

Here's an example of such an entry:

- - [15/May/2000:23:03:36 -0800] "GET /index.htm HTTP/1.0" 200 3956 "http://www.altavista.digital.com/cgi- bin/query? pg=aq&text=yes&d0=1%2fnov %2f99&q=email+marketing%2a +AND+email marketing%2a&stq=30" "Mozilla/2.0 (compatible; MSIE 4.0; SK; Windows 98)"

Not exactly easy reading, is it?

However, through the above entry, you can learn:

- The IP address of your visitor --

- The date and time of the visit -- [15/May/2000:23:03:36 -0800]

- The first file requested -- "GET /index.htm HTTP/1.0"

- The fact that the request was completed -- 200

- The number of bytes that were transferred -- 3956

Where your visitor came from and the keywords used to find your site -- "http://www.altavista.digital.com/cgi-bin/query? pg=aq&text=yes&d0=1%2fnov %2f99&q=email+marketing%2a +AND+email marketing%2a&stq=30"

Browser and operating system of the visitor -- "Mozilla/2.0 (compatible; MSIE 4.0; SK; Windows 98)"
Though the information in the raw data provides invaluable information to you, you can easily see that it requires a little effort to analyze it yourself.

 

If you're serious about analyzing your traffic, consider obtaining a quality log analysis software program. Examples of programs are listed at the bottom of this article.

When analyzing your referrer log information, what should you look for?

Number of page views per day

Page views (or page impressions) refer to how many "hits" your site has had to HTML pages only. A hit is an action on a web page, such as when a user views a web page.

Number of user sessions per day

User session refers to the activity of one visitor to a web site.

How long users are staying on your pages

How long did the users stay at your website? Are they finding what they're looking for, or are they getting frustrated and leaving?

If visitors are immediately clicking out of your site, maybe it's time to set up an onsite search engine. After all, once you get visitors to your site, you want them to be able to find what they're looking for. SearchButton.com is an excellent service. It even provides statistics on who is searching your site, the most popular searches, and will also alert you to searches that produced no results.

Ginette Degner, Professional Optimizer with SearchEngineServices.com, adds,

"The time users spend at a site tells me if they are actually reading the site or just clicking in only to leave immediately. Perhaps I am not conveying the purpose of the site well enough to make them stay. Or, I am listed under the wrong phrases. If I am buying traffic from GoTo or another engine, I like to know if the words I chose are valuable to me or just a waste."

Most requested and least requested pages

Degner further explains,

Gateway laptop battery

"Which page is attracting the most visits and how long are they there? This helps me decide what areas of a site need to be expanded upon and what areas can be dropped. For an example with a sports picks site, we found that the least visited page was the record the handicapper used to show everyone his win/lose ratio for picks and the chat room. So we dropped the page and spent the programming money on live scores and a sports news page."

Top entry pages

How are people first coming in to your site? Which pages are bringing you the most traffic? What about some of your other pages? What can you do to make them "top entry pages" too?

Top exit pages

Exit pages are another very important area of a log file, according to Degner. She goes on to ask and then elaborates on the answers...

"Where are they leaving? What off site links are they clicking on the most? If this is an intro page or another sub page that is a doorway, I may need to get rid of it or use a redirect. It tells me where I am losing visitors."

"As an example, a client insisted upon having a second intro page that played their radio commercial, so you clicked on the index page to enter and were stuck in a second media-enhanced page. We could see half of our traffic leaving right there and going no further into the site, which was a really big clue that it was a turn off to the surfing public."

"A lesson to corporate sites - a website is the wrong place to be vain. Serve up your product or service immediately or pay the price with an impatient dotCom'r."

"On other sites, we have discovered that a screen shots page made a huge difference in converting sales, and that was where the most orders came from (exit link was the order link)."

Single access pages

Which pages are being viewed by themselves, where visitors aren't even clicking to go to another page? Again, look at these pages carefully to see what you can provide on the page to keep up the interest of your visitors. You're losing them, and you need to figure out why.

Errors, such as 404 pages

If your visitors encounter too many error messages when visiting your website, they'll assume that you don't do your "house cleaning," and the professional image of your site will plummet several notches.

Most active countries

If you want a corner of the international market, study this data carefully. How many user sessions are being generated for each country that's important to your business? How you can beef up efforts to improve those numbers? Are you creating highly targeted information pages for your international keywords?

Top referring sites and URLs

Top referring search engines

Do you have some top ranking pages in certain search engines, but you're not seeing coinciding traffic through those engines? If so, you may need to rethink your keyword strategy, because a truckload of #1's won't do you any good if traffic doesn't follow.

Keywords that searchers are using to find your site

If you're being found under a particular keyword in one engine but not another, boost efforts in the other search engine and try to bring in more traffic. Also, study this section closely for any holes in your keyword-thinking strategies.

Remember that search engine positioning strategies begin with a simple keyword or keyword phrase. If you're having problems finding a keyword phrase that will bring you more traffic, visit WordSpot.com and sign up for their free trial service, or WordTracker, which also has a free service. Also most search engines have "related search" results that can give you some clues, don't over look that information.

Browsers used by your visitors

Check this section periodically to make sure that the technology offered at your website can be used by the majority of your visitors. In other words, if many of your visitors are accessing the web using older browsers, you will want to be careful about using technology that will prevent them from fully utilizing your website.

Visiting spiders

Which search engine spiders have visited your site recently? After submitting your pages to the engines, be sure to monitor this section closely for spider activity.

What do the experts feel are the most valuable parts of a referrer log?

Charlie Morris, Managing Editor of Web Developer's Journal, -- http://wdvl.internet.com/Internet/Management -- looks at the list of most popular pages first.

"This is something that's easy to act on - whatever content is most popular, simply produce more of the same sort of thing. The list of 404s should also be one of the first sections to look at, as sometimes (though not always) it's easy to find and fix these errors, greatly improving short- term traffic and your long-term reputation."

"The most important thing of all, however, is simply to compare a site's traffic from month to month, to measure how well your promotional efforts are working. It's surprising how many sites don't do this carefully."

So, take the time to analyze your traffic, and then put that valuable information to work on your website and reap the benefits of even more traffic!

Log Analysis Programs

WebTrends

FlashStats

Funnel Web

Analog (free)

Webalizer (free)

eXTReMe Tracking (free)