dsh-model-jury

Structured cross-model peer review for DeepSeek Harness.

Models deliberate. Code counts. Dissent survives.

Blind independent answers → anonymous peer critique → revision → deterministic verdict.

No permanent chairman. The judge is code.

Compatibility: verified on DeepSeek Harness 0.1.2-alpha.2 at commit 0a53fb55bea101816fa226bb964ae2bed71c343b. DSH is in developer preview and may introduce breaking changes. See COMPATIBILITY.md.

                         One Question
                              │
            ┌─────────────────┼─────────────────┐
            ▼                 ▼                 ▼
      GPT via Codex          GLM             DeepSeek
    ChatGPT subscription     API                API
            │                 │                 │
            └──────── Blind Round ─────────────┘
                              │
                         P1 / P2 / P3
                              │
                    Anonymous Peer Review
                              │
                           Revision
                              │
                  Deterministic Aggregation
                    ┌─────────┼─────────┐
                    ▼         ▼         ▼
                Majority    Dissent    Risks

Release validation: 3 providers · 3 rounds · 9 real model calls · 0 identity leaks · 6 ms maximum Round-1 dispatch skew.

The full redacted provider transcript, vote, dissent, risk, and timing record is in REAL_DEMO.md.

Install in 30 seconds

Install the prebuilt package from npm into the DSH Web profile, inspect the composed layer, and boot DSH:

dsh plugin --profile web add dsh-model-jury
dsh --profile web --dump-config
dsh web

Alternatively, download the versioned tarball from the GitHub Release, verify its published SHA-256 checksum, and install it directly:

dsh plugin --profile web add ./dsh-model-jury-0.1.0.tgz

The tarball and npm package contain compiled lib/ output. Installation does not require permission to run a package build script.

Setup

Model Jury uses DSH's provider-neutral public services. It does not implement private HTTP clients or proxy provider traffic.

GPT through native Codex authentication

The GPT seat uses DSH's first-party Codex app-server provider and the user's native ChatGPT/Codex subscription authentication. OPENAI_API_KEY is not required. The bundle registers an isolated model-jury-codex route so GLM and DeepSeek provider configuration cannot replace Codex authentication.

GLM

Configure a GLM route through DSH's @deepseek-ai/dsh-llm-pi-ai provider. Keep the credential in DSH settings or the DSH launch environment:

- id: llm-pi-ai
  config:
    providers:
      glm:
        displayName: GLM
        api: openai-completions
        baseURL: '<GLM OpenAI-compatible base URL>'
        apiKeyEnv: GLM_API_KEY
        models:
          - id: '<your GLM model id>'
            name: GLM jury model

DeepSeek and jury models

The DSH base profile supplies the first-party deepseek-official route. Configure DEEPSEEK_API_KEY through DSH settings or the launch environment, then override the complete model-jury row in the profile's cordis.patch.yml with the model IDs available to your account:

- id: model-jury
  config:
    storageDir: .dsh-model-jury
    maxQuestionChars: 12000
    maxFieldChars: 6000
    codex:
      provider: model-jury-codex
      timeoutMs: 180000
      maxRetries: 1
    glm:
      provider: glm
      model: '<your GLM model id>'
      timeoutMs: 120000
      maxTokens: 16384
      maxRetries: 1
    deepseek:
      provider: deepseek-official
      model: '<your DeepSeek model id>'
      timeoutMs: 120000
      maxTokens: 16384
      maxRetries: 1

Credentials never belong in this package, patch, test fixtures, or run artifacts.

Doctor

/jury doctor

Doctor performs one minimal real request per seat, so it can consume provider quota. It reports provider, model, status, duration, safe diagnostics, and native Codex authentication without printing credential values.

Run

/jury Should this inference runtime prioritize iOS support or ARM SIMD optimization?
/jury --style adversarial Should this inference runtime prioritize iOS support or ARM SIMD optimization?

balanced is the default. adversarial strengthens the critique instructions without changing provider topology or aggregation.

Why Model Jury

Most model councils ask several models, then ask another model to judge them. Model Jury takes a different approach:

1. Models answer independently.
2. Competing answers are anonymized.
3. Models critique one another.
4. Models get a chance to revise.
5. Code computes quorum and vote state.
6. Dissent and critical risks remain visible.

The current reference deployment uses GPT/Codex, GLM, and DeepSeek. The protocol itself is provider-neutral: CouncilSeat implementations can route other compatible models through the same blind, anonymous, structured state machine.

Protocol

Round 1 sends the same question, instructions, and JSON schema to all three seats concurrently. No seat sees another response. A random per-run mapping then assigns the responses to P1, P2, and P3. These labels identify participant positions only; answer choices such as Strategy A/B remain ordinary question content.

Round 2 gives each model its own first answer plus the anonymized positions. Provider, model, and common product identity terms are scrubbed from redistributed fields. The prompt requests strongest points, weakest points, missing evidence, actual disagreement, and calibrated severity without rewarding consensus.

Round 3 allows each surviving model to revise, merge, remain undecided, or preserve dissent. The deterministic aggregator accepts only P1, P2, P3, hybrid, or undecided; computes vote state and quorum; and keeps named final dissent visible to the user.

Three final seats produce a normal verdict. Two produce an explicit DEGRADED verdict. Zero or one produces a failed report. A single structured critical-risk flag is surfaced prominently but has no veto. Two compatible normalized critical-risk categories trigger CRITICAL REVIEW REQUIRED.

Artifacts

Every run writes owner-only trace files beneath:

.dsh-model-jury/runs/<run-id>/
  request.json
  config.json
  state.json
  prompts/round1|round2|round3/<seat>.txt
  round1|round2|round3/<seat>.json
  redistribution.json
  calls.json
  verdict.json
  report.md

Artifacts contain visible structured responses, safe call metadata, prompts, aggregation data, and the final report. They exclude environment snapshots, request headers, API keys, OAuth state, provider request bodies, and hidden chain-of-thought. Credential-shaped fields are recursively redacted.

Safety

Model Jury is deliberation-only. It never applies a recommendation, edits project files, installs dependencies, commits, pushes, deploys, or performs deliberate network mutations. GLM and DeepSeek receive no tools. Codex uses permissionMode: never plus mutation-prohibiting prompts; DSH's tested public Codex provider does not expose a strict read-only flag, so this remains a documented limitation. Read SECURITY.md before using sensitive workspace content.

Develop and package

pnpm install --frozen-lockfile
pnpm test
pnpm typecheck
pnpm build
pnpm pack

The deterministic fake transcript is in DEMO.md. Provider and host validation is recorded in PRE_FLIGHT.md. Contributions are welcome under CONTRIBUTING.md.

Known limitations

• GLM and DeepSeek require user-supplied provider credentials and model IDs.
• The tested public DSH Codex provider cannot enforce a hard read-only sandbox through its plugin configuration.
• Vote consensus uses explicit structured labels, not semantic clustering of textual hybrid proposals.
• Each Codex call starts an ephemeral app-server thread.
• There is no automatic implementation, research, chairman model, deployment, or five/seven-seat mode.

MIT licensed. See LICENSE.