Summary
Checking robots.txt and sitemap indexes is the difference between finding the sitemap and finding one of them.
Large sites split their sitemap into an index pointing at many child files, and a naive fetch of the obvious path finds a fraction of the URLs. Following the index and reading robots.txt for the declared location is the unglamorous part that makes the result complete. Reach for the combined call when you just want the list; use the two separate tools when you need to see which sitemap a URL came from.
What it is
A server over SitemapKit's API, for getting the complete URL list of a website without crawling it page by page.
What you get
- discover_sitemaps — find a domain's sitemap files, checking robots.txt, the common paths and sitemap indexes
- extract_sitemap — pull every URL out of one sitemap file
- full_crawl — discovery and extraction across all of them in a single call
Requirements
An API key from their site, and npx to run it.
Setup effort
One command plus a key — npx sitemapkit-mcp, then supply credentials