Author: George Nguyen
Your site’s technical health is a crucial part of your search visibility. If you don’t maintain it, search engines like Google may not properly crawl and index your pages.
This means that the content you’ve worked so hard to create might never make it to the search results, which can mean fewer potential customers for your business.
Server logs can help you avoid this by highlighting areas to troubleshoot and better ways to optimize your site. With the help of Wix’s log data reports and some familiarity with server logs, you’ll be able to remedy problems that could be affecting Google’s ability to efficiently crawl your site (i.e., crawl budget).
What are server logs?
How search engine bots work
Server logs are records of every request a server receives, including requests from bots (automated programs that sites like Google use to crawl your pages) and human visitors. These requests represent attempts to view a page on your site.
Each time a user or bot makes a request, the server records:
The page that was requested
The request date
The response the request got
Server responses are shown as status codes, which are grouped into five classes:
1XX – Informational response. Request received and understood.
2XX – Success. Request received, understood, and accepted.
3XX – Redirection. An additional action was taken to complete the request.
4XX – Client error. There was an error; the request could not be completed.
5XX – Server error. The server could not complete the action.
What server logs can tell us
For SEO purposes, we’re going to focus on the log data from bots, which can reveal insights you can use to improve your organic visibility. These include:
Which pages are or aren’t getting crawled by search engines
Which pages may be getting crawled too often or not enough
Errors on your site that search engine bots are encountering
For example, if you’ve created a new product page but it’s not receiving traffic, your bot log reports could tell you whether search engine bots have crawled it. If they haven’t, that may explain why no users have visited the page.
Wix’s bot log reports
Wix users can access their bot log data by navigating to the Analytics & Reports section of their dashboard, and then selecting Reports. In the Marketing & SEO section, click the “show more” button to reveal the Bot Traffic over Time, Bot Traffic by Page, and Response Status over Time reports.
These reports only include requests from bots, which is what you need to troubleshoot technical SEO issues. In addition to viewing this information in the Wix dashboard, you can also download the reports as Excel, CSV or image files to use later.
Now, let’s take a closer look at each report to see how you can use them to improve your site’s SEO.
Bot Traffic over Time
You can use the Bot Traffic over Time report to view which bots have crawled your site or to get an overview of bot traffic trends over time. This information can help you:
See whether Google or another search engine crawled your site (or a specific page) recently
Estimate the time between when a search engine crawls a page and when the page gets indexed
This report is visualized as a split bar chart (shown below).
Highlighting a segment of a bar will show you how many requests a bot made on a given date. In the example above, we can see that Bingbot made 11 requests on February 11.
Filters can help you find the data that matters most
By default, this report shows request data from the most common bots, but you can add or remove bots using the Select bots option. You can also segment the data to find exactly what you’re looking for by using the following filters:
Entry page – This shows the report for a specific page (or set of pages) on your site
Response status classes – This shows just the data for pages that are returning 2XX, 3XX, 4XX, or 5XX status codes, which can be useful for troubleshooting.
Response status codes – This enables you to go into more detail and view the report for just the pages that are returning a specific status code (such as 404s, for example).
Bot Traffic by Page
Your Bot Traffic by Page report shows how frequently your pages are crawled, which can help you see:
Which pages on your site are getting crawled the most
Whether search engines are crawling your pages more or less often
You can view this report as a bar chart (shown below) or as a customizable table by selecting the Table option above the chart.
Use the table view for more detailed analysis
To get an easier, yet deeper understanding, switch over to the table view of your report.
In this view, you can add or remove columns, such as Date, Bot, Entry page, Unique pages, and so on, enabling you to create an overview with just the information you’re looking for.
Response Status over Time
Your Response Status over Time report is where you’ll go to learn about errors that may be preventing your content from reaching users. It can help you:
See whether redirects are working correctly
Identify missing pages (404 status code) that can be redirected to improve user experience
You can view this information in two ways:
Unique pages – This shows the number of different pages on your site that received a response status code.
Hits – This shows the number of times a specific page was visited by a bot.
You can also filter this data by bot, entry page or response status code. And, the table view for this report enables you to toggle columns to show more or less data.
Now that you’re familiar with bot log reports, the next step is to check in on them regularly so that you can continue to fix errors and find more ways to improve your SEO.
Consider scheduling each report so that they automatically get sent to your inbox as often as you like. You can do this by selecting the Subscribe button at the top-right of each report.
Monitor bot log reports to identify issues and opportunities
Log data is no different from your regular user analytics in that they both allow you to monitor crucial site functions and provide you with areas to troubleshoot or optimize. Now that you understand how to interpret this data, take a look at your reports to establish a baseline. Then, subscribe to your reports so that you’re always up to date with your site’s technical health and can address problems as they arise.
George Nguyen is the Director of SEO Editorial at Wix. He creates content to help users and marketers better understand how search works. He was formerly a search news journalist and is known to speak at the occasional industry event.