SEO Specialists’ Must-Have Tool
Screaming Frog, when it was first launched in 2011, started as an advanced web crawler designed specifically for Search Engine Optimization.
Fast forward to 2020; the platform has evolved into a multi-faceted website crawler with diverse analytical and auditing tools. To no surprise, Screaming Frog continues to be one of the top players in the industry.
For beginners, the tool could be tremendously overwhelming. As extensive as its possibilities are, Screaming Frog is also sophisticated enough to confuse you in the beginning. Knowing the right settings and features is crucial for you to take the maximum advantage of this tool.
With the help of EssayPro we have created a guide that will help you establish the basics of the Screaming Frog tool.
The simple aim here is to clear the mist and let you in on how to boost your productivity with the tool. We will start with the basic settings first and break it down further below.
Your Basic Menu Kit
Most of the menu is self-explanatory; however, the features listed below are the most important and beneficial.
The File menu contains settings that allow you to save crawl reports. You can save them, and if you did not, you also get access to the six latest coral reports from this menu. Furthermore, you can also schedule your SEO crawl from the ‘File’ button.
Undoubtedly, this is where you set all the preferences for your web crawls. The most important settings here are as follow.
- Include and Exclude. If you want to include or eliminate any specific webpages, you can post the URLs here.
- API Access. When you are integrating your crawl with any analytical tools such as Google Analytics, here is where you can find the settings.
When you want to export detailed data from the crawl, Bulk Export will give you choices of export addresses. You can find them with response codes, anchor texts, images, and a few other options as well.
The difference between Bulk Export and Reports is that you receive an overall report in this section. It will also contain relevant take on data such as canonical errors or redirect chains.
Choosing The Memory
There are two storage modes to facilitate crawling and store the data. Depending on how extensive the crawling is and your machine specifications, you can choose between memory storage and database storage.
Memory storage mode allows for fast, flexible, and efficient crawling for settings. As machines typically have less RAM than the hard disk space, memory mode is optimal for sites with under 500k links. This mode is also recommended for machines without an external SSD.
On the flip side, Database Storage is suited for those who have access to SSD, as well as want to scrape at a large scale. You can cover more URLs at the default of 5 million links.
Users can also automatically save the crawls in this mode, without having to do it manually. As the files are loading in .seospider extension, it is also easier to open the stored crawls.
Screaming Frog highly recommends using an SSD and doing crawls in database mode. It makes for the most efficient crawling features and supports. If you are unable to invest in an SSD, you can adjust the memory used for crawls at:
Configuration > System > Memory.
You can begin by assigning 4gb, as the SEO spider will use that first before moving the data to the disk. It is best to start with this, so you will leave enough RAM for other applications as well.
Screaming Frog comes with an extensive set of resources. It goes beyond the merit of this article to elaborate on every single feature. So, we will elaborate on how to do a crawl of an entire site, how to filter your crawl reports, and how to create an XML sitemap.
By the end, you will be equipped with all the necessary guidance on how to begin crawling with Screaming Frog.
How to Crawl a Site
Before starting a crawl, it is recommended to narrow down the kind of data you would like to obtain. It is equally important to recognize the size of the website as well as the specific pages you need to access.
Scraping an entire site will not only take resources but will also cost you time. Screaming Frog comes with a number of restrictions you can apply to crawl to get the exact specific information you require. This will help you manage the file sizes easier.
To Include All Subdomains
The default settings in Screaming Frog scrape only the specific subdomains you choose. Any other domains the tool crawls will be considered as external links. If you want to add any other subdomains, here is how you do it.
- Go to Configuration Menu > Crawl.
- Check “Crawl All Subdomains” under the Crawl Behaviour.
- Check “Crawl Outside of Start Folder.”
This will make sure that the scraping starts from the subdomain you have specifically mentioned and expands to the other parts as well.
Choose What to Crawl
The memory consumption is also based on the volume of data you collect. Screaming Frog lets you consider options for a lighter crawl by specifying the exact parameters you need.
You can do this from the Configuration menu, as we mentioned earlier. Here, you can select and deselect the data you want, such as:
- External links;
To Exclude Specific Subdomains
Apart from controlling the type of data, you can also set the SEO crawler to limit any unnecessary features from taking up space.
You can control what the SEO crawler covers, from entire sections to few particular URLs.
All you need to do is listed below.
- Go to Configuration > Exclude;
- Add any specific URL that you would like to eliminate from SEO crawling.
After this step, you also have the option to test that the exclusion is actually working while you perform the crawl.
To Limit the Parameters
Let’s say you want to sample a number of pages, but not scrap it entirely. It will help you maintain control over the memory as well as get a report on multiple pages across multiple sites. Here is what you can do:
- Go to Configuration > Spider > Limits;
- Set Limits to Number of Pages, Depth, Number of query strings, and more.
When you are using the Memory mode, do not forget to save your crawls regularly.
You only have to merely stop the crawl and click ‘File > Save’ to achieve this. Once the saving is complete, you can hit ‘resume’ to crawl.
Setting up a Proxy
If you are crawling a competitive site, then you might want to avoid any bot-blocking. Screaming Frog allows users to set proxy of any kind, to ensure the optimum usage of the tool. You can set this at ‘Configuration > Proxy.’
While doing this, ensure that the proxy does not hinder the speed and efficiency of the crawl, or else it could make the crawling process last for a considerably longer time.
Finding Site Content on Your Crawl
With the amount of data available, the next step is to sort it. If you prefer to do the filtering in a CSV file, you can download it from the “Bulk Export” menu.
- If you want to sort pages based on Content:
Go to Internal Tab > filter by HTML > Choose Word Count.
- If you want to find images on a specific webpage:
Select the page > click on Image Info > find the images listed.
Another way to do this is to crawl only the specific URL you want.
- If you want to filter the CSS files:
Before crawling, go to Spider Configuration menu > Crawl > Check CSS.
Before crawling go to
How to Analyze Meta Descriptions
Mets descriptions play a significant role in a site’s technical SEO and thus needs to be audited as well.
You have to regularly crawl meta descriptions to avoid any duplicate content, optimize it with keywords, and maintain its relevance.
- If you want to filter Page Titles or Meta descriptions
Go to Page Titles tab > filter for titles less than 70 characters.
Google does not display over 70 characters in these titles, so looking for longer ones is a waste of time. You can also do the same with ‘meta descriptions’ or URLs, and filter them for duplicity as well.
- If you want to find pages with Meta Directives
Go to ‘Directives’ tab > choose the type of directive and apply filter.
- If you want to find Schema Markups
In this case, you will need to set custom filters.
Go to Configuration menu > Custom > Search.
Here you can search for Schema item types or narrow down with any specific markup you are looking for.
How to Do a Link Audit With Screaming Frog
Among all the site content, links are particularly difficult to audit. However, they are also a big reason for performing the crawl.
Here is how you can set a link audit on with SEO spider.
- Begin the crawl and continue until it’s finished.
- Go to Bulk Export > Export CSV of “All Links”.
This will give you data of all link locations, along with the respective locations, anchor text directives, and other data.
Note that this report could be quite large, depending on the extent of the crawl. If you want to know the overall number, go to the “Internal” tab and filter by ‘Outlinks,’ or ‘Inlinks.’
How to Find Broken Links
- Begin the Crawling of the website by entering the URL;
- Go to the ‘Response Codes’ tab, and filter by ‘Client Error (4xx)’.
You do not have to wait until the entire crawling is finished. You can view the 404 broken links while crawling. You can also see it in the right-hand window and by clicking ‘Client Error (4xx)’. This panel gets updated as the crawling progresses.
- Go to ‘Inlinks’ tab at the bottom to see the source of the broken links of the selected URL. You can use the data to identify the internal links to be modified.
- Choose ‘Outlinks’ if you are looking for broken outbound links.
- Go to ‘Bulk Export’ > Response Codes > Choose the type of filter you want for the broken code. If you want an overall report, then Client Error (4xx) will cover your bases.
Sometimes it is also possible that the source is, in fact, direct. In order to find the original source page, it is recommended to choose Reports > Redirects > All Redirects.
This way, you will find the 404 URL in the final address tab, and the original source under the ‘Source’ tab.
How to Create XML Sitemaps
Sitemaps are enormously helpful to rank in Google, as they point out to the updates of your webpages. Screaming Frog has developed its own unique way to use a sitemap. You can do it in the following steps.
- Finish crawling the site
- Click ‘Sitemaps’ menu > XML sitemap.
- Choose your settings in the ‘XML Sitemap Configuration’
Here, you can filter the pages you need based on response codes, modification dares, priority, frequency, and images. SEO Spider covers only 2xx links by default. If you need more, you have to select it manually.
Users can also create a new XML sitemap by uploading links or manually pasting them.
Auditing an Existing Sitemap
If you have already completed a sitemap, you can also check it for any discrepancies or errors on your site.
Go to ‘Mode’ menu > List > Upload.
Here, you can either download the Sitemap or Sitemap Index and enter the sitemap URL to start the crawl.
If you find that certain sections of your site aren’t indexed, you can find them at:
Configuration > Spider > Autodiscover XML sitemaps through robots.txt. Sometimes, you might have to enter such URLs manually.
Screaming Frog has many more specific tools and capabilities that can help you get precisely what you are hoping for. You can also integrate it with analytical tools to ensure an even better experience.
Hopefully, this article has helped you establish basics on how to get started with the SEO Spider of Screaming Frog. When used correctly, it can be an invaluable tool in decoding a website.
How many URLs can I crawl using the SEO Spider?
The SEO Spider has the potential to crawl millions of URLs; however, it also depends on the hardware and configuration. The range would differ based on the type of memory. In the database mode, with 8 GB RAM and an SSD, you can crawl approximately 5 million URLs.
How many people can use the licensed version of Screaming Frog SEO Spider?
The licenses are provided per individual user. If you have four people working with SEO Spider, then you will need four different licenses. But you can use the license to install the tool on multiple devices.
Which operating systems are supported by SEO Spider?
You can run SEO Spider on Windows, macOS, and Linux.
How can I integrate Google Analytics with SEO Spider?
Screaming Frog’s latest version 4.0 comes with Google Analytics integration. You can grant the permission at
Configuration > API Access > Google Analytics.
Is my data protected when using SEO Spider?
The tool cannot see the data or the webpages you are crawling. The entire information is stored on the machine you use.