dsh-model-tester

English | 简体中文

A DeepSeek Harness plugin that adds a one-click model availability tester panel at the bottom of the Models settings page. It sends one real streaming request to every added model and reports availability, output speed (TPS), first-token latency, and total elapsed time.

Features

• Test all at once: real streaming calls (llm.stream) with 3-way concurrency and a 30-second per-model timeout
• Test one channel: per-provider「测试本渠道」button runs only that provider's models
• Three metrics on one line: available / TPS (tok/s) / first token (s) / elapsed (s)
• Actionable failures: failed rows show the reason, error code, and HTTP status
• Per-model retest: each model row carries its own prominent Test / Retest button
• Smart filtering:
  • Inactive / unconfigured providers are hidden
  • Auto-registered entries whose model id or name contains vision (e.g. modlens vision) are hidden
• Model list source: models you explicitly configured in settings win; otherwise the adapter catalog (llm.listModels) is used

Install

dsh plugin --profile web add github:masknull/dsh-model-tester

Restart the web profile (restart the dsh web process) after installing, then open Settings → Models — the tester panel sits at the bottom of the page.

Uninstall

dsh plugin --profile web remove dsh-model-tester

Metric definitions

Metric	Meaning
Available	The streaming call finished with stop / max-tokens / tool-calls
TPS	usage.outputTokens ÷ (stream finish time − first token time), tok/s, 1 decimal; shows — when the decode window is shorter than 50 ms (response arrives in a single chunk — no meaningful rate can be measured)
First token	Time from request start to the first output chunk (text / reasoning delta), seconds, 2 decimals
Elapsed	Total time from request start to stream finish, seconds, 2 decimals

The test request deliberately sends no maxTokens — DSH omits the field entirely when it is undefined, so the upstream provider applies its own default output budget. Passing any value there is forwarded as max_output_tokens, where a small budget competes with a reasoning model's chain-of-thought tokens and makes the provider truncate early with response incomplete: max_output_tokens (an intermittent, model-dependent false failure). TPS relies on provider usage reporting; it shows — when the provider does not report usage.

Architecture

• Host half (index.js): injects llm / settings / timer / webServer / connection and registers two authenticated HTTP routes (auth via the client-connection service's requestRejection — the same trust boundary open-in-app uses):
  • GET /model-tester/list — configurable-provider directory × settings profiles × live-state join
  • POST /model-tester/test — one real streaming test against {provider, model}
• Browser half (client.js): a hand-written closure-factory bundle following the window.__ModuleLoader__ client-bundle contract (react resolved through the platform module table), registered into the settings.models.footer slot, talking to the host routes over fetch

Local development

# Install from a local checkout (link mode; changes apply after a profile restart)
dsh plugin --profile web add <path to this repository>

No build step is required: index.js and client.js are plain, directly loadable JavaScript.

License

MIT

---

中文

DeepSeek Harness 插件：在「设置 → 模型」页底部提供模型可用性一键测试面板。对每个已添加的模型发起一条真实的流式小请求，给出可用性标记、输出速度（TPS）、首 token 延迟与总耗时。

功能

• 一键测试全部：3 路并发逐个真实调用（llm.stream），单模型 30 秒超时
• 按渠道测试：每个渠道标题行带「测试本渠道」按钮，只测该渠道下的全部模型
• 单行三指标：可用 / TPS（tok/s） / 首 token（s） / 耗时（s）
• 失败可诊断：失败行显示原因、错误码与 HTTP 状态
• 单模型重测：每个模型行带独立的显眼「测试 / 重测」按钮
• 智能过滤：
  • 未激活 / 未配置的提供方不显示
  • 模型 id 或名称含 vision 的自动注册条目（如 modlens vision）不显示
• 模型清单来源：优先读取你在设置中配置的模型，未显式配置时回退到适配器目录（llm.listModels）

安装

dsh plugin --profile web add github:masknull/dsh-model-tester

安装后重启 web profile（重启 dsh web 进程），打开 设置 → 模型，页面底部即为测试面板。

卸载

dsh plugin --profile web remove dsh-model-tester

指标口径

指标	含义
可用	流式调用以 stop / max-tokens / tool-calls 正常终结
TPS	usage.outputTokens ÷（流结束时刻 − 首 token 时刻），tok/s，保留 1 位小数；decode 窗口不足 50 毫秒（响应一次性到达）时无法测得有效速率，显示 —
首 token	从发起请求到收到第一个输出分片（text / reasoning delta）的耗时，秒，2 位小数
耗时	从发起请求到流结束的总量耗时，秒，2 位小数

测试请求刻意不传 maxTokens：DSH 在该值为 undefined 时会完全省略此字段，由上游提供方使用自身默认输出额度。任何传入值都会被转成 max_output_tokens 下发，而小额度会与推理模型的思维链 token 争抢同一份额度，导致上游以 response incomplete: max_output_tokens 提前截断（偶发、随模型而异的假失败）。TPS 依赖提供方返回 usage；未返回时显示 —。

架构

• 宿主半（index.js）：注入 llm / settings / timer / webServer / connection，注册两个鉴权 HTTP 路由（鉴权走 client-connection 的 requestRejection，与 open-in-app 同一信任边界）：
  • GET /model-tester/list — 可配置提供方目录 × settings 剖面 × 激活状态联接
  • POST /model-tester/test — 单模型真实流式测试
• 浏览器半（client.js）：按 window.__ModuleLoader__ 工厂契约手写的客户端 bundle（react 走平台模块表），注册进 settings.models.footer 槽位，通过 fetch 调用宿主路由

本地开发

# 从本地 checkout 安装（link 方式，改动即时生效，重启 profile 后可见）
dsh plugin --profile web add <本仓库本地路径>

无需构建步骤：index.js 与 client.js 均为可直接加载的纯 JS。

License

MIT