dsh-auto-review

English | 简体中文

LLM auto-review approval answerer for DeepSeek Harness: when a session's approval policy is 'auto', sandbox escalations beyond workspace-write (sandbox_permissions: "danger-full-access" + justification) are decided without a human prompt — a human is asked only when the reviewer is unsure, and every error path fails closed.

Two decision layers, in order:

1. Deterministic filter — hardline (always-deny) + deny/allow regex rules over a normalized command (~/$HOME expansion, comment stripping, whitespace collapse, macOS private-path folding). A hardline or deny match rejects immediately; an allow match grants immediately; neither runs the LLM. Deterministic denials are FINAL — operator policy is never appealed.
2. Clean-context LLM safety review — the gray zone goes to a small reviewer model with a fixed constant system prompt, temperature: 0, a single-word verdict (ALLOW / DENY / UNSURE), and a tight output cap. The user message carries ONLY the tool name, the justification, and the XML-wrapped command — never the conversation.

Appeal mode (context: 'appeal-user-prompts', the default): an LLM-voiced DENY is re-judged ONCE with the session's genuine user prompts attached as intent evidence — extracted deterministically (never LLM-summarized) from user/message events whose source kind is exactly 'user'. Appeals can only RELAX a denial, never tighten an allowance: they reduce false positives; they are not a detection upgrade.

The answerer registers with { prepend: true } on approval/request, so it runs before any human/machine channel, and delegates with next() whenever the effective policy is not 'auto' or the request carries no action. The reviewer is defense-in-depth against an honest-but-wrong model, never a sandbox-widening mechanism: workspace-write remains the boundary until a grant lands. An optional consecutive-denial circuit breaker (denialBreakerThreshold) short-circuits retry loops before any further LLM cost.

⚠ Requires a patched harness core

This plugin cannot run on stock upstream deepseek-harness. It needs three patched-core surfaces, shipped in core-patches/ as two git patches with a rebase guide (authored against upstream base commit 47f943859b, merge PR #2519):

• 0001-approval-auto-policy-and-escalation-action.patch — the 'auto' approval policy, ApprovalRequest.action, ApprovalService.effectivePolicyFor, and the tool/sandbox/permission-presets wiring that carries the escalation action.
• 0002-llm-auto-review-request-purpose.patch — the 'auto-review' LLM request purpose.

Apply them with git am (or git apply) in order after syncing upstream; if upstream conflicts, resolve and regenerate — see core-patches/README.md.

On vanilla upstream the plugin is deliberately inert: the policy gate never resolves to 'auto', so every escalation lazily delegates (next()) to the normal human answerers. Nothing breaks, nothing is auto-decided — but you get no auto-review until the patched core is in place.

Install

Prerequisites: a patched harness build (above), and the profile's permission presets must include the auto-review preset (sandbox: workspace-write + approval: auto) — patch 0001 adds it to the harness's permission-presets package.

From GitHub (builds on install via its prepare script; pnpm will ask you to allow the build once):

dsh plugin --profile web add github:accpowered/dsh-auto-review

If pnpm prints an allowBuilds prompt, add the printed key under allowBuilds in the profile's pnpm-workspace.yaml and re-run the add — this is install-time code execution permission, so only allow sources you trust (pinning a commit, github:accpowered/dsh-auto-review#<sha>, is recommended).

From a local checkout:

dsh plugin --profile web add ./dsh-auto-review

Restart dsh web (or your profile). The bundle patch (cordis.patch.yml) inserts one host row with a default reviewer route (deepseek-official / deepseek-v4-flash). To use a different reviewer, restate the row in your profile's cordis.patch.yml — provider and model are required and a patch replaces the row's whole config:

- id: auto-review
  name: dsh-auto-review
  config:
    provider: my-provider      # any registered LLM route
    model: my-small-model

Configuration

All knobs (composition = the cordis.patch.yml row config; user = live override through the auto-review settings namespace):

Key	Default	Composition	User-overridable	Purpose
provider	— (required)	✓	✓	Reviewer route (a registered provider)
model	— (required)	✓	✓	Reviewer model id on that route
hardline	shipped list	✓	—	Always-denied regex patterns; never bypassed while policy is auto
deny	shipped list	✓	✓	Configurable deny regex patterns
allow	shipped list	✓	✓	Configurable allow regex patterns (conservative, side-effect-free only)
maxTokens	512	✓	—	Reviewer output-token cap (single-word verdict + headroom)
timeoutMs	15000	✓	—	End-to-end reviewer deadline (ms)
onError	deny	✓	✓	Reviewer-failure fallback: deny (fail closed) or ask (human)
onUnsure	ask	✓	✓	UNSURE-verdict fallback: ask (human) or deny
denialBreakerThreshold	0 (off)	✓	✓	Consecutive-denial circuit breaker
context	appeal-user-prompts	✓	✓	appeal-user-prompts or minimal (command alone, v1 behavior)
appealMaxPrompts	0 (unbounded)	✓	✓	Appeal budget: newest user prompts attached (first prompt exempt)
appealMaxChars	0 (unbounded)	✓	✓	Appeal budget: total user-prompt characters attached

hardline, maxTokens, and timeoutMs are composition-only by design. The shipped filter defaults are documented in src/index.ts (DEFAULT_HARDLINE_PATTERNS / DEFAULT_DENY_PATTERNS / DEFAULT_ALLOW_PATTERNS).

Live user-layer overrides

User-tunable keys live in the auto-review settings namespace (registered through installSettingsSection; the namespace name stays auto-review so existing user settings documents keep applying). The composition entry is the base layer; a committed user-layer change re-resolves and recompiles the rule patterns on the very next decision — no restart. A write carrying an invalid regex is refused at the settings seam, so a bad pattern can never reach the compiled filter.

Uninstall

dsh plugin --profile web remove dsh-auto-review

The answerer and the settings section detach with the plugin fiber. Log-only auto-review/decision events already on session logs are left untouched, and sessions on the auto-review preset fall back to the human answerers.

Development

pnpm install
pnpm build       # tsc declarations (lib/types) + tsdown (lib/index.js)
pnpm test        # vitest: filter/verdict/appeal flows, settings lifecycle (local approval stub)

Layout: src/index.ts (the whole plugin — answerer, config schema, settings section, exported pure functions), tests/auto-review.spec.ts + tests/stubs.ts (the stub replicates the patched ApprovalService's dispatch semantics; the npm rc.1 package predates the 'auto' policy), core-patches/ (the two required core patches + rebase guide).

License

MIT