Screaming Frog just released a feature called the Content Cluster Diagram. This new tool uses AI to look at all your website pages and shows you how they’re related to each other. Think of it like a map that groups similar content together and spots pages that don’t quite fit with the rest of your site.
How to Access the Content Cluster Diagram
Here’s how to set it up and use it:
Step 1: Set Up AI Access
- Open Screaming Frog and go to Configuration > API Access > AI
- Choose your AI model
- Enter your API key and click “Connect”
Step 2: Configure the Embedding Prompt
- Go to Configuration > Prompt Configuration > Add from Library
- Select the prompt called “Extract embeddings from page content”
Step 3: Enable HTML Storage
- Navigate to Configuration > Spider > Extraction
- Make sure both “Store HTML” and “Store Rendered HTML” are turned on
Step 4: Turn On Embeddings
- Go to Configuration > Content > Embeddings
- Check “Enable Embeddings functionality”
- In the “Embedding Prompt” dropdown, choose the prompt you set up earlier
Step 5: Run Your Crawl and View Results
- Start your website crawl as usual
- Once finished, go to Visualizations > Content Cluster Diagram
- You’ll see a visual map of your content relationships
What This Helps With for SEO
Find Content Gaps
The diagram shows you which topics you cover well and which areas might need more content. If you see empty spaces between content clusters, those could be opportunities for new pages.
Spot Duplicate Content
Pages that appear very close together might be too similar. This helps you find content that could be competing against itself in search results.
Identify Off-Topic Content
Pages that appear far away from your main content clusters might be confusing search engines about what your site is really about. You can decide whether to improve these pages or remove them.
Better Content Planning
See how your content naturally groups together. This helps you plan new content that fits well with what you already have, making your site more focused and authoritative.
Improve Site Structure
Use the clusters to inform your website navigation and internal linking. Content that’s semantically related should probably link to each other.
Conclusion
Screaming Frog’s Content Cluster Diagram takes the guesswork out of understanding how your content relates to each other. Instead of manually reviewing hundreds of pages, you get a visual overview that shows content relationships, gaps, and opportunities. This makes it much easier to create a focused, well-organized website that search engines can understand and rank better.
