Web Crawler MCP connector

The Web Crawler lets AI agents run technical SEO and site checks on any public URL through InsightfulPipe's hosted MCP server at https://crawler.insightfulmcp.com/. It has 33 read-only checkers, from meta tags and structured data to robots.txt, llms.txt and AI crawler access, plus a sitemap generator that crawls a whole domain.

Web Crawler MCP server reference

Endpoint: https://crawler.insightfulmcp.com/
Server name: crawler-mcp
Actions: 33 read-only actions.

Add it from a terminal, then click Authorize access in the browser window that opens:

claude mcp add --transport http crawler-mcp https://crawler.insightfulmcp.com/
gemini mcp add --transport http crawler-mcp https://crawler.insightfulmcp.com/
codex mcp add crawler-mcp --url https://crawler.insightfulmcp.com/

What agents can do

On-page SEO: meta tags, headings (H1 to H6), canonical URLs, hreflang, language, URL structure, content quality, keyword density, images and alt text, and a full SEO audit.
Technical checks: status codes and redirect chains, indexability directives, page weight and load time, security headers, SSL certificates, character encoding, deprecated HTML, AMP, favicons, and mobile viewport and responsive rules.
Links: extract all internal and external links, find broken or low-quality links, and map internal linking.
Structured data and social: validate Schema.org markup, Open Graph and Twitter Card tags.
AI and crawler access: parse robots.txt, check which AI training bots are allowed, parse llms.txt, and generate robots.txt content with optional AI bot rules.
Brand and site mapping: extract brand colors, fonts, buttons and logos, and generate an XML sitemap for a domain.

Before you connect

No platform account is needed. You only need an InsightfulPipe workspace.
Members with the User role need an explicit grant for the tool.

Add the Web Crawler

Add the MCP server to Claude, ChatGPT or another client.
Click Authorize access in the browser.
Ask the agent to check a URL.

How it works

Each checker analyzes one URL. Agents run checks across several pages by calling a checker once per page.
The sitemap generator crawls the domain and returns an XML sitemap, which you can then audit page by page.
All actions are read-only.

Limits

There is no multi-page audit report; results are per URL.
The robots.txt checker analyzes a site's rules. The crawler doesn't claim to follow robots.txt or crawl-delay when it fetches pages.

Example prompts

"Run an SEO audit on https://example.com/pricing and list the top issues."
"Check the heading structure and meta tags of our five main landing pages."
"Which AI crawlers does https://example.com allow in robots.txt?"
"Generate a sitemap for example.com, then check the status code of each URL."

Troubleshooting

A page can't be fetched: check that the URL is public and doesn't require a login.
Results differ from another tool: checks run against the live page when you ask, so recent changes show up immediately.