If you are a SEO you know of Screaming Frog, a tool used to analysis a website by crawling each page on the site as a search spider would. Screaming Frog just released version 10, a major enhancement that just may make it my new favorite toy.
Embracing the Concept of Indexable
One of my biggest beefs with Screaming Frog has been that it didn’t have the concept of indexability. Let me illustrate what I mean by this with an example.
When you crawl a website with Screaming Frog, it organizes the results into tabs that align with on page elements important for SEO. The titles tab lists the URLs and the title tags found on the page which then can be filtered for potential SEO issues such as duplicate title tags.
It’s frustrating to see a list of pages with duplicate titles only to investigate and find out they have been addressed with the canonical tag. If only Screaming Frog understood the concept of indexability and showed me only duplicate titles that have not been addressed. Other crawlers such as Deep Crawl and OnCrawl do this, why not Screaming Frog?
In version 10 Screaming Frog has addressed this shortcoming. In most of the tabs you’ll find two new fields, Indexability and Indexability Status. The first field is set by the crawl to either “Indexable” or “Non-Indexable”, the second field will tell you why Screaming Frog considers the page Non-Indexable. For example the status field could be set to “Canonicalised” (Screaming Frog is a British company, hence the “s” instead of “z”) indicating that the page is not indexable because there is a pointer to a “canonical” (or representative) page of a group of pages.