dsh-plugin-midscene

English | 中文

Midscene-based AI UI automation for DeepSeek Harness (DSH): one ctx.midscene capability seam, two providers, and two model-facing tools — android_ui (one ADB-connected device) and web_ui (the active page of an already-running Chrome). Each tool is a single tool whose action parameter branches internally (tap / act / input / query / assert / boolean / back).

Install

With the dsh CLI:

dsh plugin --profile mysetup add dsh-plugin-midscene

The bundle's default layer registers the android_ui and web_ui tools; the tools read ctx.midscene opportunistically, so they appear (and the schema catalog stays stable) even before a provider row is configured — calling one without a provider fails with a message naming the missing row.

Then add exactly one provider row to your profile's cordis.patch.yml (both providers cannot own ctx.midscene in the same context):

Android (one ADB-connected device)

- insert:
    - id: midscene-android
      name: dsh-plugin-midscene/android
      config:
        deviceId: ''                  # empty: first device from getConnectedDevices()
        aiActionContext: ''           # free-form context for aiAct planning

Requirements: adb devices shows the device; Midscene's model env vars are set (see below).

Web (an already-running Chrome)

- insert:
    - id: midscene-web
      name: dsh-plugin-midscene/web
      config:
        browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/<id>'
        aiActionContext: ''

Start Chrome with --remote-debugging-port=9222 and take the webSocketDebuggerUrl from http://127.0.0.1:9222/json/version. The provider connects, never launches; on teardown it disconnect()s — never close()s — so your Chrome keeps running. On the web side back() triggers the active page's browser-history back.

Model configuration

Midscene's vision model is configured through @midscene/*'s own conventions (.env / MIDSCENE_MODEL_NAME, MIDSCENE_OPENAI_BASE_URL, MIDSCENE_OPENAI_API_KEY, ...), not through DSH's ctx.llm.

Design boundary: no policy, no recovery

The seam is deliberately thin: no retry, no precondition checking, no automatic recovery from unexpected UI state (stray popups, unwanted navigation, re-login). Callers that need that behavior build it on top — for example a constraint/harness layer that checks app state before each write action.

Known limitations

• One target per provider instance — one device or one browser per context; fan out with isolated compositions.
• No reconnect — a mid-session disconnect surfaces as a rejected call; the provider does not detect, retry, or reconnect.
• Pinned SDK versions — @midscene/android and @midscene/web are pinned at exactly 1.11.0; upgrading is a deliberate version bump.
• puppeteer is a peer (web) — the deployment's pnpm resolves it; Chrome itself is supplied by the deployment, never downloaded by this plugin.

Development

pnpm install   # build scripts of native/browser deps are denied by default; tests mock the SDKs
pnpm test      # wiring-only: mocked @midscene/*, puppeteer, and a stub seam behind the real tool registry
pnpm build     # tsc emit to lib/ (also runs as `prepare` on git installs)

Tests verify wiring only — argument forwarding, device/endpoint selection, teardown, tool routing, cross-field validation — never real device or browser interaction.

Community and support

Feel free to submit feedback or bug reports through GitHub Discussions. This repository carries the dsh-plugin topic for discoverability.

License

MIT