SEO Spider Features

Screaming Frog’s SEO Spider is one of the industry’s most popular website crawlers and offers a free and paid version. When you utilize this tool, you’ll be able to use your own powerful, flexible site crawler that allows you to see and analyze your site in real-time.

This will allow you to gather key information about on-site data that may be impacting your SEO and user experience. You’ll be able to use this data to make informed decisions on different site changes that will impact your site’s performance.

You can crawl 500 URLs with the free version, but the paid version will allow you to access unlimited URLs if you have a larger site or want more advanced features, such as scheduling.

While there are many features, it’s best to break them down into three main groups:

• Page insights
• Broken link and error finder
• Validation and checks

Page Insights

As your site grows, you’ll find that some of the pages that you created may no longer be working or may not have the best SEO measures in place. Once you go from a handful of pages to hundreds of pages, it can be very difficult to keep track of everything without using a site crawler.

You can gather great page insights using SEO Spider, such as:

• Analysis of all page titles to ensure that they are SEO optimized and still reflect the content on the page
• Complete analysis of metadata to ensure that it is still accurate and optimized properly
• Review all of your meta robots and directives so that you know if there are any issues with search engines crawling your site

You can also generate XML sitemaps that allow you to alert Google and other search engines about the pages on your site. This is a great option if your site is not being crawled, and it’s done automatically for you based on what the crawler finds.

Broken Link and Error Finder

One of the biggest issues that site owners have is that over time, the links that they have on their sites may break. If these are external links, the issue is not that major, but when it involves internal links, this can lead to negative impacts on your search engine optimization.

Using SEO Spider, you’re able to find broken links and errors that would otherwise go missed by most site owners.

You can use this information to find redirect issues that may be occurring on your site. A lot of site owners use .htaccess files to create redirects, or they may use plugins to help control the redirects on a site.

Over time, if multiple people start updating the site and making changes, it’s not uncommon for these redirects to start creating errors. All it takes is one person to create the wrong redirect or to create a redirect loop that continually tries to redirect to the same page for errors to pop up.

You will be able to find all of these issues and correct them, which can have a major impact on not only your search engine optimization efforts, but also your user experience.

Validation and Checks

Validation and checks are part of the entire program, but there are some features that are more important in this respect than others. The main validation and checks that are offered include:

• Duplicate content checker to ensure that you do not have the same content across multiple pages on your site. If you do have duplicate content, this may be impacting user experience and optimization efforts.
• Spelling and grammar checks are available in the premium version that allow you to improve your user experience and make sure you are using proper grammar, punctuation and spelling to improve the professionalism of your site.

SEO Spider incorporates a lot of additional features into their premium version of the platform. These features are great for larger site owners, and they also include the ability to schedule and customize the way your site is crawled.

You’ll be able to integrate this data into your Google Analytics and Search Console. You’ll also be able to integrate with PageSpeed Insights to better understand how the changes you made have impacted your site.

SEO Spider also gives you the ability to store and view the HTML that is rendered on your site, search for custom source codes and also create custom robot.txt for your site.

If JavaScript rendering does occur, the platform is able to render it and extract information from these pages. This is very important for dynamic pages that may change and for user generated content.

SEO Spider is a tool that is used by some of the largest brands in the world to make sense of their site’s performance and data.