Basic Web Scraper

A lightweight, ethical web scraping skill for AI agents that need to extract information from the web. EXTRACTION CAPABILITIES: • Page title, meta description, canonical URL • All text content (cleaned of HTML tags, scripts, styles) • Links with anchor text, href, and rel attributes • Images with src, alt text, dimensions (when available) • Open Graph metadata (og:title, og:description, og:image) • Twitter Card metadata • JSON-LD structured data • Tables as structured arrays • Headings hierarchy (H1-H6) • Publication date detection FEATURES: • Respects robots.txt — skips disallowed paths • User-Agent rotation to avoid blocks • Built-in rate limiting (1 req/sec default, configurable) • Timeout handling (30s default) • Redirect following (max 5 hops) • Gzip/deflate decompression • Cookie handling for session-based sites • CSS selector support for targeted extraction USAGE: ```bash mcplug install basic-web-scraper-mcplug # Scrape a page mcplug run basic-web-scraper-mcplug --url "https://example.com" # Extract specific elements mcplug run basic-web-scraper-mcplug --url "https://example.com" --selector "article h2" # API POST /api/v1/install/basic-web-scraper-mcplug {"url": "https://example.com", "extract": ["text", "links", "images", "meta"]} ``` RESPONSE: ```json { "url": "https://example.com", "title": "Example Domain", "meta_description": "...", "text": "This domain is for use in illustrative examples...", "links": [{"href": "https://www.iana.org/domains/example", "text": "More information...", "rel": ""}], "images": [], "og": {"title": "Example Domain"}, "headings": [{"level": 1, "text": "Example Domain"}], "scrape_time_ms": 245 } ``` Ethical scraping: respects robots.txt and rate limits by default.