Armory
Source
Browse
MCPs

jitsmaster-web-crawler

Configurable web crawler that extracts structured content from websites while respecting robots.txt rules and offering customizable settings for depth, delay, and concurrency.

Score
41.0822 signals
Evidence
1 star · 2 forks
Last commit
as last read from GitHub; most reads are from 2 Sep 2026 or later
Listed

Install

No one-command install. Set it up from its source.

Alternatives · MCPs

  1. jasonjmcghee-webmcp789 stars · 52 forks96.941
  2. tinyfish-agentql178 stars · 45 forks94.369
  3. just-every-read-website-fast161 stars · 26 forks93.326

What it is

MCP server Web Crawler, catalogued on PulseMCP. Configurable web crawler that extracts structured content from websites while respecting robots.txt rules and offering customizable settings for depth, delay, and concurrency.

When to use it

Configurable web crawler that extracts structured content from websites while respecting robots.txt rules and offering customizable settings for depth, delay, and concurrency.

Notes

Listed from the PulseMCP registry. The registry does not state a license. Check it before production use.