dsh-web-automation

🇬🇧 English | 🇷🇺 Русский

A DeepSeek Harness plugin that gives a local model a local-first web stack — no paid search API, no third-party data broker, no cloud required for the keyless engines.

It bundles four capabilities into one installable plugin:

Capability	What it does	Keyless?
Multi-engine search	The web_search tool, routed across DuckDuckGo + Bing (keyless) and Exa / DeepSeek / Jina (opt-in, when their API keys are present). Fallback or fuse (parallel + RRF) routing, cooldowns, BM25 enrichment.	✅ DDG + Bing
Cached fetch	The web_fetch tool backed by a SQLite page cache with ETag/Last-Modified revalidation. Repeats within the TTL make no network request.	✅
Web platforms	The web_platform_search tool: search a specific platform (GitHub, Reddit, YouTube, Bilibili, V2EX, RSS, …) via its own public endpoint. New platforms are added without code via config or versioned rule packs (hot-reloaded).	✅
History / stats / cache	web_history, web_search_stats, web_cache_clear — read the shared local store; no network.	✅

All state is local: the store is $DSH_HOME/web.db. Outbound traffic for the keyless engines is limited to DuckDuckGo and Bing.

Quick start

Prerequisites:

• Node.js 22.19+ or 24+ — the same requirement as DSH itself; the plugin uses the built-in node:sqlite store.
• DeepSeek Harness — either via npx @deepseek-ai/dsh (no install) or a source checkout (pnpm install && pnpm run build, then pnpm dsh ...).

1. Install the plugin into a profile (the bundle overlay is applied automatically):

   dsh plugin --profile tui add git+https://github.com/stelmakhdigital/dsh-web-automation.git
   

2. Boot the profile:

   dsh --profile tui    # TUI
   dsh web              # Web UI at http://127.0.0.1:3080
   

   At boot the plugin loads with the host: the web seam is pinned to the multi search / cached-http fetch providers, web_fetch is enabled, and web_platform_search plus the history tools are registered. No extra configuration is needed — every field is defaulted (keyless DuckDuckGo + Bing).

3. Verify — run the smoke test from a DSH session (e.g. web_search "hello world").

Replace tui with your profile name. To inspect the composed profile tree: dsh --profile tui --dump-config.

Install

Plugins are installed into a DSH profile with dsh plugin — each profile is its own pnpm workspace under $DSH_HOME/profiles/<name>. At boot, DSH symlinks the host's @deepseek-ai/* packages into the profile's node_modules, so the plugin's peer dependencies resolve to the host's own copies.

This package is a DSH bundle: its package.json declares dsh.bundle, so installing it automatically applies the shipped local-web.cordis.yml overlay — it pins the web seam to the plugin's providers, enables web_fetch in the host tool-web row, and registers the plugin. No manual --patch needed:

dsh plugin --profile tui add git+https://github.com/stelmakhdigital/dsh-web-automation.git

(Replace tui with your profile name. The seam pin is required: without it the seam sees two usable search providers (the deployment default plus multi) and fails with WEB_PROVIDER_AMBIGUOUS.)

Manual alternative — if you want to tweak the config before applying, apply the overlay yourself:

dsh --profile tui --patch "$PWD/local-web.cordis.yml"

Optional: browser automation

The dsh-web-browser sub-package adds local Chromium (Playwright) automation behind the browser_* tools (browser_open, browser_navigate, browser_snapshot, browser_click, browser_type, browser_screenshot). It is separate because it pulls in Playwright + a Chromium download.

It is a sub-directory of this repo, and pnpm cannot install a sub-directory of a git repo directly. Install it from a local clone as a tarball (npm pack): unlike a link: install, the tarball is unpacked into the profile's node_modules, so the package's own dependencies (playwright) are installed and its @deepseek-ai/* imports resolve to the host's packages:

git clone --depth 1 https://github.com/stelmakhdigital/dsh-web-automation.git ~/dsh-plugins/dsh-web-automation
cd ~/dsh-plugins/dsh-web-automation
npm pack ./browser  # → dsh-web-browser-0.3.0.tgz (lib/ is prebuilt in the repo)
dsh plugin --profile tui add ./dsh-web-browser-0.3.0.tgz
# one-time: install the Chromium binary
dsh plugin --profile tui exec playwright install chromium

The browser package is a bundle too — its patch (cordis.patch.yml in the tarball) registers the browser plugin row automatically. To update: git pull in the clone, npm pack ./browser again, and dsh plugin add the new tarball.

Configure

Add a row to your deployment's cordis.yml (or an overlay applied with dsh --patch ...). See cordis.yml.example for the full reference.

- id: web-automation
  name: 'dsh-web-automation'
  config:
    search:
      engines: [ddg, bing, exa, deepseek, jina]   # tried in order
      mode: fallback            # fallback | fuse
      region: ''                # region/market hint (DDG kl, Bing setmkt)
      # exa:      { apiKeyEnv: EXA_API_KEY }      # or apiKey: '...'
      # deepseek: { apiKeyEnv: DEEPSEEK_API_KEY }
      # jina:     { apiKeyEnv: JINA_API_KEY }
    fetch:
      revalidate: true          # conditional revalidation for fresh-but-expired pages
    platforms:
      tool: true                # register web_platform_search
      maxResults: 20
      # platforms:              # override built-ins by id, or add new platforms
      #   - id: my-site
      #     name: My Site
      #     format: json
      #     searchUrl: 'https://my-site.example/search?q={query}'
      #     fields: { items: 'data.results', url: 'link', title: 'title' }
    history:
      history: true             # web_history
      cacheClear: true          # web_cache_clear
      stats: true               # web_search_stats

Every field is defaulted, so an empty config: {} (or no config at all) enables the full local web stack with the keyless engines.

Relationship to DSH's built-in web packages

This plugin is an externalized, standalone copy of DSH's internal web packages (web-search-multi, web-fetch-cached, web-platforms, web-store, web-browser, tool-web-history) and is currently ahead of upstream (SearXNG engine, news-mode freshness, embedding re-rank, LRU eviction, SSRF guard, inline screenshots).

• Mutual exclusion: the plugin and the built-in packages register the same provider ids (multi, cached-http) and tool names. A deployment that loads both fails at startup with WEB_DUPLICATE_PROVIDER — keep one. If you use this plugin, do not apply DSH's examples/web-local overlay (or its preset copies), and vice versa.
• The tools come from the host: web_search and web_fetch are registered by the host's tool-web plugin; this plugin registers the providers behind them (plus web_platform_search and the history tools). The overlay above enables web_fetch in the tool-web row.
• Upstream drift: because the plugin evolves independently, its behavior may diverge from the built-in packages over time. The module headers in src/ mark the upstream package each module mirrors.

API keys (optional)

The keyless engines (DuckDuckGo, Bing) work with no configuration. To opt in to Exa / DeepSeek / Jina, provide their API keys either:

• in the launch environment (EXA_API_KEY, DEEPSEEK_API_KEY, JINA_API_KEY), or
• in the plugin config (search.exa.apiKey, etc.), or
• via the DSH credentials domain (a key written to the credentials store takes effect per-search, without a restart).

Usage

Once installed and configured, the model can:

• Search the web — web_search "query" (multi-engine, enriched snippets).
• Fetch a page — web_fetch <url> (cached; repeats within the TTL make no network request).
• Search a platform — web_platform_search { platform: "github", query: "..." }.
• Review history / stats — web_history, web_search_stats, web_cache_clear.
• Drive a browser (with dsh-web-browser) — browser_open, browser_navigate, browser_snapshot, browser_click, browser_type, browser_screenshot.

Privacy model

• Queries go to the configured search engines only — unavoidable with any search engine. With the default engine list and no API keys, outbound traffic is limited to DuckDuckGo and Bing; inference stays local.
• No credentials required for basic use.
• All state is local: the store is $DSH_HOME/web.db, nothing is sent anywhere else.
• Caveat: scraping public SERPs may violate a search engine's terms of service; the provider sends an explicit product User-Agent, rate-limits itself (1 req/s per engine by default), and cools down blocked engines. Use responsibly.

Security

• SSRF guard (on by default): requests to loopback, private, link-local, and otherwise reserved network targets (IPv4 0/8, 10/8, 127/8, 172.16/12, 169.254/16, 192.168/16; IPv6 ::1, ::/128, fe80::/10, fc00::/7) are blocked. The check runs on the literal host and after DNS resolution (against rebinding), and for web_fetch/enrichment it re-checks every redirect hop (max 5). Guarded paths and their flags:

  Path	Flag
  web_fetch (cached fetch provider)	fetch.allowPrivateNetworks
  search enrichment (page fetches for snippets)	search.allowPrivateNetworks
  web_platform_search fetches (incl. RSS feed URLs)	platforms.allowPrivateNetworks
  browser_navigate (Playwright)	allowPrivateNetworks in the dsh-web-browser config

  Set the relevant flag to true only in a trusted, network-isolated environment.

• Browser approval (fail-closed): browser_open/browser_navigate require approval per the dsh-web-browser approval setting (never | navigate | all — navigate gates navigation and script evaluation, all adds clicks and typing). If the approval service is unavailable or the call has no agent to route it through, the action is denied, not silently allowed.

• Cache eviction (LRU by usage): the store keeps at most fetch.cacheMaxPages page records (default 500) and search.cacheMaxSearches search records (default 1000), evicting the least-recently-accessed beyond the cap after each write. This keeps web.db bounded over time.

Known limitations

• HTML SERP parsing is brittle; markup changes degrade to zero results until the parser updates (block detection converts silent empties into cooldowns).
• The plugin runs in the host DSH process with the host's privileges (a trusted static package); it is not sandboxed. Run DSH as a normal user, and in a network-isolated container/VM if the plugin may reach sensitive targets.
• Browser automation: one tab per agent session. Screenshots are saved to a file by default; pass inline: true to browser_screenshot to get base64 in the model context.

Examples

Keyless-only (no API keys, no SearXNG)

dsh-web-automation:
  search:
    engines: [ddg, bing]   # keyless only
    enrich: true

Full (all engines + SearXNG)

dsh-web-automation:
  search:
    engines: [ddg, bing, exa, deepseek, jina, searxng]
    searxng:
      endpoint: http://localhost:8080   # your SearXNG instance
    embedding:
      endpoint: http://localhost:11434  # Ollama (or any /embeddings server)
      model: nomic-embed-text

News mode (time-filtered)

dsh-web-automation:
  search:
    engines: [bing]   # Bing supports the freshness filter
    freshness: 24h    # 24h | week | month | year

Smoke test

After installing and applying the overlay, verify the stack end to end (in a DSH session):

1. web_search "hello world" — returns sources (DDG/Bing keyless).
2. web_fetch https://example.com twice — the second call is a cache hit (no network; check web_search_stats).
3. web_platform_search { platform: "github", query: "schemastery" } — returns GitHub sources.
4. web_history — shows the searches/fetches above.
5. web_fetch http://127.0.0.1/ — fails with WEB_SSRF_BLOCKED (the SSRF guard).
6. (with dsh-web-browser) browser_open → browser_navigate https://example.com → browser_screenshot → browser_close — the screenshot file appears in the temp dir.

Developer workflow

For working on the plugin itself (build, test, typecheck, local install):

git clone https://github.com/stelmakhdigital/dsh-web-automation.git
cd dsh-web-automation
npm install --legacy-peer-deps   # @deepseek-ai/* peers are provided by the host DSH at runtime
npm run build && npm run build --prefix browser   # bundle src/ → lib/ (both packages)
npm test                          # vitest (103 tests)
npm run typecheck                 # tsc against the local @deepseek-ai/* stubs
DSH_HOST=/path/to/deepseek-harness npm run typecheck:host   # strict check against the real host types

lib/ and browser/lib/ are committed and CI enforces they stay in sync with src/, so git/tarball installs work without esbuild. To try a local copy in a profile:

npm pack                                   # → dsh-web-automation-0.3.0.tgz
dsh plugin --profile tui add ./dsh-web-automation-0.3.0.tgz

Troubleshooting

Symptom	Cause	Fix
ERESOLVE peer conflict on install	Peer deps absent outside DSH deployment	npm install --legacy-peer-deps
Cannot find module '@deepseek-ai/...'	Plugin installed without DSH's packages	Install DSH first (peers resolve to host's versions)
WEB_PROVIDER_AMBIGUOUS at startup	The web seam sees two usable search providers	Add the web seam pin row (searchProvider: multi, fetchProvider: cached-http) — see the overlay
WEB_DUPLICATE_PROVIDER at startup	Both the plugin and DSH's built-in web packages are loaded	Keep one — remove the built-in rows (or the plugin row); see "Relationship to DSH's built-in web packages"
web_fetch blocked (SSRF)	Target is loopback/private/link-local	Set fetch.allowPrivateNetworks: true (trusted env only)
SearXNG returns non-JSON	JSON API not enabled on the instance	Add search.formats: [html, json] to SearXNG's settings.yml
Embedding re-rank falls back to BM25	Embedding endpoint unreachable	Check the endpoint URL + model name; BM25 is the fallback
Browser: Chromium not found	Playwright browser not installed	npx playwright install chromium
web.db grows large	Cache eviction caps too high	Lower fetch.cacheMaxPages / search.cacheMaxSearches

License

MIT — see LICENSE.