Using screamingfrog to create a page list with SEO properties

UPDATED 20th April 2021 for new version.

Screamingfrog is a website crawling tool and it's one of our most used tools in our SEO work. There is a free and a paid version. There is a WINDOWS, MAC and UBUNTU version.

It will create a list of every page (and many other properties) on your website so you can see URLs, Page titles, Meta descriptions, Headings and more, in one place on a single report/spreadsheet.

It's free for up to 500 server calls, so most sites under 100 pages should be fine, even sites up to 200 pages may be OK. A paid license is £149 a year at time of writing.

NOTE: You can also run this on your competition to see what they are doing.

IMPORTANT: This will show you the data that is on your site when you run the program. NOT what's indexed in Google, as this is delayed until Google reindexes your website.

Here's how to use Screamingfrog

1. Go here and download the software https://www.screamingfrog.co.uk/seo-spider/ It's available for PC and Mac.
2. Install the software and open Screamingfrog.
3. Grab your website URL, add it to the top bar and click on START
4. When the crawl is complete click on CRAWL DATA > INTERNAL > HTML in the right window. This shows a list of all the pages on your website.
5. In the top left click on EXPORT and save the file in your required format, for this exercise I am using Excel .xlsx but CSV and Google Sheets formats are also available.
6. Now open it in your spreadsheet application. Here's what we do:
   1. Make the top TITLES row bold
   2. Freeze the top row
   3. Delete all rows after the last H1 column unless you want this information but it's quite advanced. then remove in this order rows N, M, F, E, D, and B
   4. Widen the ADDRESS, TITLE 1 and META DESCRIPTION columns so you can see the data in the cells.
7. There is a lot of useful data in the columns after the H1 but for simplicity and on page SEO we are just using the columns left.

SEO Properties

You can now see all your SEO page titles, meta descriptions and heading in one place.

Now you have an overview you can apply your keyword research to each page and create amazing ON PAGE SEO for your website.