dsh-plugin-vision

Auxiliary vision for DeepSeek Harness (dsh): analyze images through an external OpenAI-compatible vision endpoint and get a text answer back. Works with any main model — including DeepSeek, whose own API is text-only.

Why

dsh's built-in read_image tool injects the image into the main model's context, which requires the main model to declare image input. The DeepSeek adapter does not (inputModalities: ['text']), so read_image refuses to run with it. This plugin takes the complementary auxiliary vision route: it sends the image to a separate vision model and returns the answer as text. No main-model vision support needed.

	read_image (built-in)	vision_analyze (this plugin)
Image goes to	the main model's context (native)	an external vision endpoint
Main model must be vision-capable	yes	no
Result	image block the model sees	plain text answer

How it works

1. Read the image — a local file path or an http(s) URL (size-capped, magic-byte MIME sniffing).
2. Encode it as an inline base64 data URL.
3. POST an OpenAI-compatible chat/completions request with [{type:text}, {type:image_url}] content.
4. Return the vision model's answer (falls back to reasoning_content for thinking models).
5. Transcription cache (new in v0.1.1): each (image, question) pair is transcribed once — keyed by image content sha256 (deduplicated across paste / read_image archive copies / resumed history) plus question sha256 — and later requests reuse the cached text. The vision endpoint sampling temperature is pinned to 0 (temperature) for deterministic output. The result:
   • once an image is in the conversation history, later turns no longer call the vision endpoint again → time-to-first-token (TTFT) stays flat;
   • the transcribed text never changes → the main model's (DeepSeek) prompt-prefix cache is not broken by transcription variance, so the cache hit rate no longer decays.

Capabilities

1. vision_analyze tool

Send an image (local path or URL) to the configured vision endpoint and get a text answer — works with any main model, including text-only DeepSeek.

2. Image-attachment transcription (pasting images in the Web UI)

The Web UI accepts pasted/dropped images, but the DeepSeek adapter rejects image content (UNSUPPORTED_CONTENT). This plugin hooks agent/pre-step — the documented seam for replacing the messages that enter a step — and, when a user message carries image blocks, calls the vision endpoint to describe them, then swaps each image block for a [User-attached image description] text block before the request reaches the model. DeepSeek only ever sees text, so image attachments just work. When the active model route declares image input, transcription is skipped and native vision takes over.

Toggle with attachImages (default true). Transcription failures degrade to an explicit note instead of blocking the turn.

attachMode controls how the vision model is asked about attached images:

• auto (default): when the user wrote text with the image, that text is passed to the vision model verbatim as its question — "who is this?", "translate the text", "what is wrong on this page?" are answered directly, plus a one-line image summary for follow-ups. Without user text, a generic description is produced.
• describe: always use the generic description prompt, ignoring user text.

3. deepseek-vision provider route (pasting images in the Web UI, root fix)

The Web UI's upload preflight rejects images when the selected model does not declare image input — so pasted images never even reach the agent on the plain DeepSeek route. This plugin registers a deepseek-vision provider: a DeepSeekAdapter subclass that declares image input (preflight passes) and transcribes attached images to text at request time before delegating to the real DeepSeek chat-completions endpoint. The main model is still DeepSeek — same endpoint, same key, same models. Select "DeepSeek (vision via plugin)" in the model picker, then paste/drop images as usual.

Toggle with deepseekVision.enabled (default true); the provider id is deepseekVision.providerId (default deepseek-vision). Transcription goes through the content-hash cache, so once an image is in the history every later turn reuses the cached text instead of calling the vision endpoint again.

4. opencode-vision provider route (OpenCode Go + image transcription)

The same wrapper idea over OpenCode Go (opencode.ai/zen/go), which is a pi-ai built-in provider in dsh. This plugin registers an opencode-vision provider: a PiAiAdapter subclass that serves the full opencode-go catalog (deepseek-v4-flash/pro, qwen3.7-max/plus, kimi-k3, glm-5.1/5.2, minimax-m3, grok-4.5, …), declares image input so the Web UI upload preflight accepts pasted images, and transcribes attached images to text at request time for text-only models. Models the catalog marks as natively image-capable (e.g. qwen3.7-plus, minimax-m3) stream the image through untouched. Select "OpenCode Go (vision via plugin)" in the model picker, then paste/drop images as usual.

Toggle with opencodeVision.enabled (default true); the provider id is opencodeVision.providerId (default opencode-vision). The credential is the same OPENCODE_API_KEY as the base opencode-go route, so store the key once (via the web Models page or OPENCODE_API_KEY) and both routes work.

Models the installed pi-ai catalog does not describe yet (your account may support newer ones) can be added outright with opencodeVision.extraModels — the wire protocol is required, endpoint and caps default to the OpenCode Go gateway:

config:
  opencodeVision:
    extraModels:
      - id: glm-5.3
        api: openai-completions
        contextWindow: 1000000
        maxTokens: 131072
      - id: qwen3.8-max
        api: anthropic-messages

To also enable the plain (non-vision) opencode-go route with all its models, add a llm-pi-ai section to $DSH_HOME/settings.yaml (hot-reloaded, no restart):

# $DSH_HOME/settings.yaml
llm-pi-ai:
  providers:
    opencode-go:
      apiKeyEnv: OPENCODE_API_KEY
      displayName: OpenCode Go
      # Omit `models` to serve the full installed catalog instead.
      models:
        - id: deepseek-v4-flash
        - id: deepseek-v4-pro
        - id: qwen3.7-plus
        - id: grok-4.5
        # …all opencode-go models you want to use

No-second-port setup: load the plugin into your existing GUI

The vision API key resolves through the harness credentials service first (the Web UI's stored keys) and falls back to the environment — no export needed in the GUI. Write the plugin into the home-level user patch, which applies to every profile (including the web GUI you already run):

# $DSH_HOME/cordis.patch.yml
- insert:
    - id: vision
      name: 'file:///<path-to-plugin>/lib/index.js'

Store your vision API key in credentials (or export VISION_API_KEY), then restart the GUI once. Use the built lib/index.js — the published CLI loads .ts entries but cannot resolve their .js-suffixed sibling imports from src/.

4. Image archive (auto-save pasted images + index.json)

Every pasted image is automatically saved to ~/.dsh/image-archive/ with a date-numbered name (2026-08-14_120331_001.png), recorded in an index.json manifest (path, sha256, size, source, optional note), and its location is annotated for the model ([图片已存档: …]). Two tools complete the flow:

• image_archive — the agent archives an important image (user preferences, receipts, key data) into a named folder with an optional note: it saves <archiveDir>/<folder>/<name-or-date>.png and updates index.json.
• image_archive_find — search the manifest by name, folder, or note.

Configure with archive.enabled (default true) and archive.dir (default ~/.dsh/image-archive). Deduplicated by attachment id.

Try it

Install the bundle into any profile (the prepare script builds on install):

dsh plugin --profile web add github:Tianbaidi/dsh-plugin-vision

Store the vision API key in your credentials or environment (ALIBABA_CODING_PLAN_API_KEY), then restart the GUI. Paste/drop an image and ask — or use the vision_analyze tool directly. To use the opencode-vision route, also store OPENCODE_API_KEY (the web Models page accepts both).

Prefer a dev overlay instead? Point it at your local checkout:

- insert:
    - id: vision
      name: 'file:///<path-to-plugin>/lib/index.js'

Windows note: plugin paths in overlays must be file:// URLs (file:///D:/...%20...), not bare D:/... paths — the ESM loader rejects the latter as scheme d:.

Configuration

Key	Default	Meaning
baseUrl	(empty — required)	OpenAI-compatible chat-completions endpoint base URL.
apiKeyEnv	VISION_API_KEY	Env var (or stored credential) holding the API key.
model	(empty — required)	Vision model id on the endpoint.
visionEndpoint	(empty)	Active transcription endpoint by id from visionEndpoints. Empty (or an id not in the registry) falls back to the top-level baseUrl / apiKeyEnv / model single endpoint.
visionEndpoints	[]	Named transcription-endpoint registry. Each entry needs id and may override any subset of baseUrl / apiKeyEnv / model / timeoutMs / maxImageBytes / temperature / seed / attachMode; unset fields inherit the top-level value. The id also namespaces transcription-cache entries, so switching endpoints never reuses another model's text.
timeoutMs	120000	Per-call timeout (thinking vision models need headroom).
maxImageBytes	8388608 (8 MB)	Hard cap on image size.
temperature	0	Vision endpoint sampling temperature. 0 (default) keeps transcriptions deterministic so the main model's prompt-prefix cache stays stable.
seed	(unset)	Optional fixed random seed (if the endpoint supports it).
attachImages	true	Transcribe pasted images to text for text-only main models.
attachMode	auto	auto: pass the user's own prompt to the vision model; describe: always generic description.
transcriptionCache.enabled	true	Cache transcriptions keyed by image content sha256 + question sha256: each (image, question) pair calls the vision endpoint once; later requests reuse the cached text. Fixes TTFT inflation and prompt-prefix cache breaks.
transcriptionCache.file	~/.dsh/vision-transcription-cache.json	Cache persistence file (atomic writes, survives restarts).
transcriptionCache.maxEntries	1000	Cache entry cap; oldest entries are evicted first.
deepseekVision.enabled	true	Register the deepseek-vision provider route (DeepSeek + image transcription).
deepseekVision.providerId	deepseek-vision	Provider route id shown in the model picker.
opencodeVision.enabled	true	Register the opencode-vision provider route (OpenCode Go + image transcription).
opencodeVision.providerId	opencode-vision	Provider route id shown in the model picker.
opencodeVision.apiKeyEnv	OPENCODE_API_KEY	Env var (or stored credential) holding the OpenCode Go API key.
opencodeVision.displayName	OpenCode Go (vision via plugin)	Display name shown by selectors.
opencodeVision.extraModels	[]	Uncatalogued models to merge into the route (account models newer than the installed pi-ai catalog); each entry needs id + api (anthropic-messages / openai-completions / openai-responses), optionally baseURL / contextWindow / maxTokens / input / reasoning.

Any OpenAI-compatible vision endpoint works. The defaults are deliberately empty so no provider is assumed; pick one:

Provider	baseUrl	model	Notes
Zhipu GLM (free tier)	https://open.bigmodel.cn/api/paas/v4	glm-4.6v-flash	Free registration, zero cost out of the box
Alibaba DashScope (incl. token plans)	https://dashscope.aliyuncs.com/compatible-mode/v1 (or your plan's endpoint)	qwen3.7-plus / qwen-vl-max	Your own plan's endpoint if you have one
Ollama (local, offline)	http://localhost:11434/v1	qwen3-vl:4b	No API key needed
Any OpenAI-compatible gateway	your gateway's /v1	the gateway's vision model	—

Configure per deployment (e.g., your profile's cordis.patch.yml or the plugin row's config):

- id: vision
  name: dsh-plugin-vision
  config:
    baseUrl: https://open.bigmodel.cn/api/paas/v4
    apiKeyEnv: VISION_API_KEY
    model: glm-4.6v-flash
    timeoutMs: 120000

Multiple transcription endpoints (registry)

When one endpoint is not enough — e.g. a token plan that may expire, with OpenCode Go vision models as the fallback — register several named endpoints and pick the active one with visionEndpoint. Every field a preset omits inherits the top-level value, so a preset usually only names what differs:

- id: vision
  name: dsh-plugin-vision
  config:
    # Active transcription endpoint (switch here when the plan expires).
    visionEndpoint: aliyun
    visionEndpoints:
      - id: aliyun
        baseUrl: https://token-plan.cn-beijing.maas.aliyuncs.com/compatible-mode/v1
        apiKeyEnv: ALIBABA_CODING_PLAN_API_KEY
        model: qwen3.7-plus
      - id: opencode-kimi
        baseUrl: https://opencode.ai/zen/go/v1
        apiKeyEnv: OPENCODE_API_KEY
        model: kimi-k3
        temperature: 1   # required: kimi models reject the default temperature 0
      - id: opencode-qwen
        baseUrl: https://opencode.ai/zen/go/v1
        apiKeyEnv: OPENCODE_API_KEY
        model: qwen3.7-plus

OpenCode Go candidates verified against the live endpoint (2026-08-16, real key, image payload): of the catalog's image-capable go models, only kimi-k3 and kimi-k2.7-code work as transcription endpoints today, and both require temperature: 1 — the plugin's default temperature: 0 is rejected with "invalid temperature: only 1 is allowed for this model". mimo-v2.5 answers but intermittently returns empty text; minimax-m3 answers but emits its reasoning inline (<think>…) into the transcription. kimi-k2.6, qwen3.6-plus, and qwen3.7-plus currently fail with upstream 503 "Endpoint is unavailable" (text and image, direct and proxied) — not a client issue; retry when the gateway recovers. Switching endpoints never reuses cached text from another model: each preset's id namespaces its transcription-cache entries.

Develop

pnpm install        # installs the published @deepseek-ai peer packages
pnpm typecheck
pnpm test           # 58 vitest cases: MIME sniffing, payload, parsing, source loading, execute, transcription cache, provider routes, endpoint registry

Changelog

v0.3.0 — transcription endpoint registry

New visionEndpoints / visionEndpoint config: register any number of named transcription endpoints and switch the active one with visionEndpoint. Each preset inherits unset fields from the top-level config, so a preset usually only names baseUrl / apiKeyEnv / model. The preset id namespaces transcription-cache entries — switching endpoints never reuses another model's cached text. activeVisionEndpoint(config) resolves the active endpoint and is exported. Legacy single-endpoint config (baseUrl / apiKeyEnv / model) keeps working unchanged (no id → default cache bucket, on-disk entries stay valid).

v0.2.0 (2026-08-16) — opencode-vision provider route

New opencode-vision provider route: serves the full pi-ai OpenCode Go catalog under one picker entry, claims image input for the Web UI preflight, and transcribes attached images at request time for text-only models (natively image-capable models such as qwen3.7-plus / minimax-m3 stream the image through untouched). Built as a PiAiAdapter subclass over the installed pi-ai catalog, so all three wire protocols (openai-completions, anthropic-messages, openai-responses) and the provider's reasoning special-casing keep working. Same OPENCODE_API_KEY credential as the base opencode-go route. opencodeVision.extraModels merges uncatalogued account models (e.g. glm-5.3, qwen3.8-max) into the route with an explicit wire protocol. 7 new tests (catalog listing, extra-model merge, image claim, transcription, native-vision passthrough, replay-state alignment, registration).

Fixed in v0.2.0: the delegate rewrites model.provider to the catalog route at wire time, which made pi-ai stamp the session replay state with opencode-go while the harness recorded the message source under opencode-vision — a resumed session then failed with "invalid pi-ai replay state: provider does not match assistant source". The adapter now realigns every finish chunk's replayState.provider to the route id. Sessions written before the fix can be repaired by rewriting each message's replayState.provider to its source.provider in the exported JSONL.

v0.1.1 (2026-08-16) — Major fix: TTFT and cache hit rate

Two performance issues in the deepseek-vision provider, located from real session telemetry:

• TTFT inflation: once an image entered the conversation history, every request re-transcribed it synchronously (measured TTFT degraded from ~1s to 22–69s). Added transcriptionCache (image content sha256 + question sha256, persisted to ~/.dsh/vision-transcription-cache.json): each (image, question) pair is transcribed once, later requests reuse the cached text.
• Cache hit rate decay: the non-deterministic transcription text permanently broke the main model's prompt-prefix cache from the image position onward (measured hit rate decayed from 99.8% to 91.1% and kept falling). Added temperature (default 0) and optional seed for deterministic output.
• Added 5 tests covering cache hits, content-hash keying (paste vs read_image archive copy), persistence, eviction, and deterministic payloads.

v0.1.0

Initial release: vision_analyze / vision_reask / image_archive tools, image-attachment transcription, deepseek-vision provider route, image archive.

Known limitations

• Image sources are file paths (resolved against the harness cwd) or plain http(s) URLs. Remote URLs are fetched directly without SSRF hardening — restrict this tool to trusted networks if you use it with URL inputs.
• The image is sent to the configured endpoint as-is; oversized images are rejected rather than downscaled (no Pillow dependency). Compress first for very large screenshots.
• Token costs of the vision call are charged to the configured endpoint's plan.
• The transcription cache is keyed by image content sha256 + question sha256: a new question about the same image triggers one fresh vision call (which is then cached). To force a refresh, remove the matching entry from (or delete) ~/.dsh/vision-transcription-cache.json.
• Once an image is in the conversation history (pasted by the user, a read_image tool result, or a resumed session), every later request carries it; the cache ensures the vision endpoint is only called on the first occurrence, but the transcribed text still occupies tokens in the history.

Publish

This project is a bundle (dsh.bundle.patch). Install with dsh plugin add, share on GitHub with the dsh-plugin topic, or npm publish. See PUBLISH.md in the companion scaffold repo for the full checklist.

License

MIT