web-scraper

v2.1.0 MCP-compatible dataweb MIT updated 2026-08-14

Extract structured data from web pages with CSS-selector rulesets, pagination handling, and polite rate limiting. Respects robots.txt by default and never renders JavaScript — what you select is what you get.

Installation

$ skillbank install web-scraper@2.1.0
# pins exactly 2.1.0 · verify with: skillbank show web-scraper

Pin the version in production. The 2.x range is stable; 1.x rulesets are not forward-compatible (see v2.0.0 breaking change below).

Configuration

The skill reads a ruleset.yaml next to your task. Minimal example:

# ruleset.yaml
ruleset:
  base_url: "https://example.com/listings"
  selectors:
    title: "h1.listing-title"
    price: ".price[data-currency]"
  paginate:
    next: "a[rel=next]"
    max_pages: 10
  rate_limit:
    requests_per_second: 2
    respect_robots_txt: true

notedelay_ms was deprecated in v2.1.0 in favor of rate_limit. Old keys still parse but emit a warning.

Inputs

Input	Type	Required	Description
`ruleset`	string (path) / object	yes	Path to a `ruleset.yaml`, or an inline ruleset object.
`seed_urls`	string[]	yes	One or more starting URLs. Overrides `base_url` when both are set.
`output`	"json" \| "csv" \| "jsonl"	no	Output format. Default `json`.
`user_agent`	string	no	Custom User-Agent. Defaults to a Skill Bank identifier with contact info.
`dry_run`	boolean	no	Validate selectors against the first page without crawling. Default `false`.

Version history

v2.1.02026-08-14
- Added retry with exponential backoff on HTTP 429 and 5xx responses (max 5 attempts).
- Deprecated delay_ms in favor of the rate_limit block; old key warns, still parses.
- Fixed pagination when the "next" link is relative and the base tag is absent.
v2.0.02026-05-30
- Breaking: rulesets moved from JSON to YAML; skillbank migrate converts old files.
- Breaking: selectors are now scoped to the article container by default; add scope: page to restore old behavior.
- robots.txt is now respected by default (was opt-in).
v1.2.12026-03-12
- Fixed UTF-8 mojibake when exporting CSV from pages declaring windows-1252.
- Reduced default connection pool from 20 to 8 to be gentler on small sites.
v1.0.02025-11-02
- Initial release: selector rulesets, single-page extraction, JSON output.

← back to catalog