@jayyuen66/dsh-scrapling

dsh plugin --profile web add @jayyuen66/dsh-scrapling

<p align="center">

<img src="https://cdn.jsdelivr.net/gh/JayYuen666/dsh-scrapling@main/icon.svg" width="96" height="96" alt="dsh-scrapling logo" />

</p>

<h1 align="center">dsh-scrapling</h1>

<p align="center"><em>Scrapling web scraping for dsh: JavaScript rendering, structured extraction and a background crawler, behind one resident Python sidecar.</em></p>

<p align="center">

<a href="https://www.npmjs.com/package/@jayyuen66/dsh-scrapling"><img src="https://img.shields.io/npm/v/@jayyuen66/dsh-scrapling?style=flat-square&logo=npm&logoColor=white" alt="npm version" /></a>

<a href="https://www.npmjs.com/package/@jayyuen66/dsh-scrapling"><img src="https://img.shields.io/npm/d18m/@jayyuen66/dsh-scrapling?style=flat-square" alt="npm downloads (total)" /></a>

<a href="https://github.com/JayYuen666/dsh-scrapling/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/JayYuen666/dsh-scrapling/ci.yml?branch=main&style=flat-square&label=ci" alt="ci" /></a>

<img src="https://img.shields.io/badge/license-MIT-4c1?style=flat-square" alt="MIT" />

<img src="https://img.shields.io/badge/node-%3E%3D24.9-4c1?style=flat-square&logo=nodedotjs&logoColor=white" alt="node" />

<img src="https://img.shields.io/badge/dsh-%5E0.2.1--rc.5-658EFF?style=flat-square" alt="dsh" />

<a href="https://github.com/deepseek-ai/deepseek-harness"><img src="https://img.shields.io/badge/powered_by_dsh-4D6BFE?style=flat-square&logo=deepseek&logoColor=white" alt="powered by dsh" /></a>

</p>

<p align="center">

<img src="https://skillicons.dev/icons?i=typescript,python,react,nodejs,vitest,githubactions" alt="tech stack" />

</p>

<p align="center">

<a href="https://github.com/JayYuen666"><img src="https://img.shields.io/badge/GitHub-JayYuen666-181717?style=flat-square&logo=github&logoColor=white" alt="GitHub" /></a>

<a href="mailto:cqwkkj@gmail.com"><img src="https://img.shields.io/badge/Email-cqwkkj@gmail.com-EA4335?style=flat-square&logo=gmail&logoColor=white" alt="Email" /></a>

<a href="https://twitter.com/JayYuen666"><img src="https://img.shields.io/badge/Twitter-X%2FJayYuen666-000000?style=flat-square&logo=x&logoColor=white" alt="Twitter/X" /></a>

<a href="./README.zh-CN.md"><img src="https://img.shields.io/badge/lang-简体中文-658EFF?style=flat-square" alt="简体中文" /></a>

</p>

A web-scraping extension for dsh: Scrapling as agent tools — JavaScript rendering, CSS/XPath extraction with self-healing selectors, sessions with login state, a background crawler and page Q&A. You supply the interpreter and the browser engine; everything runs in one resident Python process.

<img src="https://api.iconify.design/lucide/file-text.svg?color=%23658EFF&height=22" height="22" width="22" align="absmiddle" alt="" /> Contents

• [What it does](#what-it-does)
• [Install](#install)
• [Runtime packs (macOS arm64)](#runtime-packs-macos-arm64)
• [Verify it is live](#verify-it-is-live)
• [What it does not provide](#what-it-does-not-provide)
• [Enabling it in dsh](#enabling-it-in-dsh)
• [The tools](#the-tools)
• [Replacing the built-in fetch and search](#replacing-the-built-in-fetch-and-search)
• [MCP mode](#mcp-mode)
• [Settings](#settings)
• [Proxies](#proxies)
• [Security](#security)
• [Data and privacy](#data-and-privacy)
• [Known boundaries](#known-boundaries)
• [FAQ](#faq)
• [Development](#development)
• [License](#license)

<img src="https://api.iconify.design/lucide/shield.svg?color=%23658EFF&height=22" height="22" width="22" align="absmiddle" alt="" /> What it does

dsh ships web_fetch and web_search. The built-in web_fetch is deliberately honest and minimal: a plain HTTP(S) GET with no JavaScript execution and an honest User-Agent. That is a good default — and it means a single-page application yields only its loading shell, and there is no way to ask for "just the product table". Measured on a page whose content exists only after JavaScript runs:

Pipeline · Text obtained

raw HTML to text (no JS) · SPA Shell / LOADING_PLACEHOLDER

Fetcher (curl_cffi, still no JS) · SPA Shell / LOADING_PLACEHOLDER

browser render (patchright) · full rendered DOM, Markdown table, structured rows

This package adds what the built-ins structurally cannot do:

• Browser rendering — JavaScript execution via patchright.
• Structured extraction — CSS/XPath selection returning data, not a wall of Markdown.
• Self-healing selectors — Scrapling's adaptive mode relocates an element after the site's markup changes; a selector that returns None after a class rename returns the relocated element with adaptive on.

Everything runs against one long-lived Python process. That is not an optimisation: Scrapling's session classes keep cookies, the browser engines keep a page pool, and adaptive keeps a SQLite fingerprint database — a process per call would throw all of that away. Tools are registered conditionally: the plugin probes what the environment can actually do at startup and registers only what will work, so the model never burns a turn on a tool that was guaranteed to fail. Every tool renders its result row through the official tool-call view slot. scrapling_answer is the only tool that spends another model call — see [The tools](#the-tools).

<img src="https://api.iconify.design/lucide/package.svg?color=%23658EFF&height=22" height="22" width="22" align="absmiddle" alt="" /> Install

dsh plugin --profile web add @jayyuen66/dsh-scrapling

Remove with dsh plugin --profile web remove @jayyuen66/dsh-scrapling. The package is on the public npm registry, so installation needs no credentials and no scoped-registry line.

• Requires Node.js >= 24.9. This is an ESM-only package, and the build entry points (build-client.ts, build-host.ts, build-cli.ts) are executed by node directly, so the floor has to sit where TypeScript type st

…(truncated)