Cortex — Multi-Model Orchestration & Evaluation for DeepSeek-Harness

English | 中文

Cortex is a DeepSeek-Harness plugin that turns the host's primary (strong) model into a high-leverage supervisor. The supervisor plans, decomposes, verifies and accepts; routed executor models (small/cheap models such as Qwen3.8-9B) and tools do the heavy lifting. With a recursive task tree, dynamic routing, four-stage quality gates, budget control, failure recovery, policy reuse and per-dimension model evaluation, Cortex keeps quality stable while driving the unit cost of successful tasks down.

The supervisor is the host primary model that drives the cortex_* tools (e.g. deepseek-v4-pro in the bundled registry), and executors are the models the dynamic router picks per subtask (L1/L2 tiers, e.g. Qwen3.8-9B).

Suggested repository topics (GitHub): dsh-plugin · dsh · cordis · llm-orchestration · multi-model · model-routing · quality-gate

---

Highlights

Capability	Entry	Notes
Task intelligence	cortex_start	Profile → fingerprint → strategy match (≥0.90 reuse / 0.70~0.90 light-check / <0.70 replan) → budget pool → granularity check
Recursive task tree	cortex_decompose	Safety limits: maxDepth 6 / maxTotalNodes 50 / maxChildrenPerNode 10 / maxReplan 3; DAG depends_on; aggregate nodes merge children by node schema + Gate0, active-model only
Recursive execution	cortex_execute recursive	Ready leaves auto-scheduled, aggregate nodes merged; evaluate:gate2 / multi_vote pass through — same standard as manual batch
Caching	engine built-in	L1 exact + L2 semantic (near-duplicate text similarity, conservative 0.88 threshold, skip-able) + L3 cross-task component reuse
Drift & value	engine built-in	Model drift monitoring vs static baseline (auto routing penalty, surfaced in registry/matrix) + value score & Pareto frontier per evaluation run
Dynamic routing	cortex_execute	Candidate filter → utility scoring (quality/success/risk − cost/latency/failure) → MQC upgrade gate; costFirst / balanced / qualityFirst; multi-model vote
Quality gates	Gate0 deterministic · Gate1 self-check · Gate2 evaluator · Gate3 cortex_review	Hard fails: format_validity / safety / groundedness < 0.5; review decisions carry node quality/Gate1 context snapshots
Recovery engine	cortex_recover	Failure taxonomy → retry / switch / prompt-fix / decompose / upgrade / human + failure-pattern learning
Budget control	engine built-in	4 pools (10/70/10/10), STOP-1..4 (STOP-2 from observed MQC — real ΔQ/ΔC, not hard-coded), 90% high-cost lock, degraded mode
Model evaluation	cortex_evaluate	7-dim scoring per case; per-model × task-type × capability × 7-dim profile (dims_profile, EMA, persisted across restarts); Rule-First deterministic correctness for structured expected; UI one-click run; value score (Q×SR/Cost) + Pareto frontier; drift monitoring (vs static baseline → automatic routing penalty)
Model registry	cortex_models (config/models.yaml)	Static YAML + DSH ctx.llm dynamic discovery + enable/disable management; routing scores (ranking preview)
Policy center	cortex_policy / cortex_strategy_match	Fingerprint similarity, versioned policies, skip-level tree suggestions
KPI	cortex_report	Supervisor calls/task, A-cost breakdown by call kind (start/decompose/review/…) (aByKind), supervisor leverage, saving vs direct primary-model run (negative → orchestration warning), quality, policy reuse rate
Web console	/cortex ui / 8788	Trends / timeline / model registry / capability matrix (4 metrics + 7-dim rows) / evaluation center (case library + results), i18n zh+en, light/dark theme, CNY/USD display, SSE live updates

Install & Auto-Enable

dsh plugin is the supported way: it installs the package into the profile, and because this package declares dsh.bundle.patch (= cordis.patch.yml), it is automatically appended to dsh.profile.bundles — enabled on next boot with no manual profile edits.

From npm (after npm publish):

npx @deepseek-ai/dsh plugin --profile web add dsh-cortex
npx @deepseek-ai/dsh web        # boot — the 12 cortex_* tools are live

From a local checkout (development):

cd <repo-path>                                   # the repo root
npx @deepseek-ai/dsh plugin --profile web add .  # anchors to this dir (or pass the absolute path)
npx @deepseek-ai/dsh web

Remove: npx @deepseek-ai/dsh plugin --profile web remove dsh-cortex.

The shipped bundle patch inserts the cortex row with uiEnabled: true, uiPort: 8788 (console listens on loopback only). stateDir and modelsFile use built-in defaults ($DSH_HOME/storages/cortex and the bundled config/models.yaml) — anything can be overridden by an - id: cortex config row in your profile's own cordis.patch.yml (it applies after bundle layers).

Typical Flow (agent view)

cortex_start → (cortex_strategy_match) → cortex_decompose T0 → cortex_execute [node_ids | recursive]
→ cortex_review (batch accept) → cortex_recover (on failures) → cortex_report (KPI) → cortex_policy (settle)

Operations & Commands

1. Agent tools (12 — the supervisor calls them autonomously)

Tool	Purpose	Key parameters
cortex_start	Create a task (profile → fingerprint → strategy match → budget pool)	goal, profile{task_type, domain, complexity, risk, quality_requirement, budget_limit, input_tokens?, expected_output_tokens?, latency_requirement_ms?, language?, capability?}, input
cortex_strategy_match	Policy-center lookup (3-tier reuse verdict)	profile
cortex_decompose	Split a parent node (safety limits / DAG / replace)	task_id, parent_id?, replace?, children[{goal, task_type, capability, output_schema, quality_target?, tool_allow?, depends_on?, type?}]
cortex_execute	Route & execute (Gate0/1[/2], cache, accounting)	task_id, node_id?/node_ids?, recursive?, mode? (costFirst
cortex_review	Gate3 batch acceptance (decisions carry quality/self-check snapshots)	task_id, decisions[{node_id, status, reason_code?, missing_items?, next_action, recommended_depth?, confidence?}]
cortex_recover	Recovery decision tree (retry/switch/upgrade re-run in-engine)	task_id, node_id?
cortex_policy	Settle a policy from a successful task	task_id, note?
cortex_report	KPI report (per task or global)	task_id?
cortex_models	Model registry / profiles / routing scores / drift	task_type?, capability?, reset?
cortex_evaluate	Model evaluation (7-dim × capability profiles, value/Pareto)	cases?, use_saved?, models?, task_type?, quality_target?
cortex_eval_cases	Eval-case library (page + conversation shared)	action? (list
cortex_capabilities	Capability catalog (append-only)	action? (list

2. Slash commands (/cortex <subcommand>)

Command	What it does
/cortex status	Engine state: tasks (running), nodes, attempts, policies, models, discipline status
/cortex task <id>	Task tree (profile/strategy/budget/nodes)
/cortex trace <id>	Execution trail (attempts + decisions)
/cortex kpi [id]	KPI report (global or per task)
/cortex models [reset]	Registry listing; reset clears evaluation feedback
`/cortex policies [disable	enable ]`
/cortex model-pref [set <id> [reason]|clear|show]	Set/clear the preferred executor model with preference content (reason = which tasks it suits): routing adopts it when the task scene is covered by the content (>=50%) + active + quality <0.9 (user preference is authoritative — capability tags are advisory only, prefCapabilityMismatch audited; failed executions auto-upgrade via the recovery engine); persisted; same as page ★
/cortex rate [refresh]	USD→CNY rate (refresh forces fetch)
/cortex reset	Delete state files (effective after host restart)
`/cortex ui start	stop
/cortex help	Command usage

3. Console REST (127.0.0.1:8788)

GET  /api/overview · /api/timeline?hours=all|24|168 · /api/rate · /api/models
     /api/matrix · /api/evaluations · /api/eval-cases · /api/capabilities
     /api/evaluate/status · /api/task/<id> · /api/events (SSE live)
POST /api/models/<id>            {enabled?: boolean, preferred?: boolean, reason?: string}   // reason = preference content (required on the page)
     /api/matrix/reset           (clear evaluation feedback)
     /api/eval-cases             {action: add|remove|save, ...}
     /api/capabilities           {name, description?}
     /api/evaluate               {models?, cases?, use_saved?} (real background run)

4. Runtime lifecycle

• Mount: apply → registry load (static YAML + persisted overrides + model states) → heartbeat stateDir/mounted.json → optional console → usage-settle timer (60s, zombie sweep every 5 ticks) → dynamic model discovery.
• Per turn: orchestration-discipline context is injected per conversation session (suppressed only when that session has a flow active within 15 min); other sessions' leftover flows never suppress.
• Unmount / restart: effect dispose stops the console (active SSE connections destroyed — no hang) and clears timers; everything durable lives in the state dir (tasks/nodes/policies/profiles/model states/rate) and rebuilds on restart (last-write-wins).
• Self-healing: zombie tasks (running, 4h silent) are finalized automatically; evaluation feedback and drift baselines survive restarts; /cortex reset wipes state files for a fresh start.

Model Profiles (how they update)

• Evaluation write-back (cortex_evaluate / one-click UI run): per task-type quality EMA + success rate + cost/latency + 7-dim capability profiles (dims_profile), persisted in profile_overrides.json (drift baselines anchored at first measurement).
• Execution feedback (round 69): every real node execution (non-cache) updates deterministic counters — attempts/complete/failed + failure-type & task-type distributions, persisted in the same file; when samples >= 3 (EXEC_PROFILE_MIN_SAMPLES), routing uses the measured success rate instead of the static one (the more it is used, the truer the profile, the better the routing). Quality scores still come from evaluations only (Gate1 self-checks never pollute profiles).
• POST /api/matrix/reset or /cortex models reset restores static baselines (evaluation + execution feedback).
• Measured-only scores (round 72): the capability matrix shows a value only for verified capabilities (evaluation/execution write-back keys, tracked in measured); static declarations and defaultQuality never masquerade as verified scores — unverified cells display "no measured data".

Sub-Model Capabilities & Media Support

Executors (routed sub-models) are self-contained worker agents. What they can and cannot do:

Capability dimensions include media tags: image_analysis (declared by vision-capable models — image tasks route & filter precisely by it) and audio_video_analysis (catalog dimension; no model declares it yet — matrix stays empty until a media-capable model/MCP is configured). Decompose image subtasks with capability: ['image_analysis'] (not document_analysis) for exact routing.

Capability dimensions include media tags: image_analysis (declared by vision-capable models — image tasks route & filter precisely by it) and audio_video_analysis (catalog dimension; no model declares it yet — matrix stays empty until a media-capable model/MCP is configured). Decompose image subtasks with capability: ['image_analysis'] (not document_analysis) for exact routing.

Dimension	Supported
Input	text, files (read/glob/grep), images (read_image / MCP browser), URLs (web_search/web_fetch, read-only) — also base64 images (data:image/...;base64 or {image_base64, mime}) auto-decoded to a real file for the executor
Tools (agent mode)	read glob grep read_image + web_search/web_fetch (read-only, default; override via executorExtraTools) + mcp__* (any configured MCP server — e.g. Playwright screenshots) + skill + business-plugin prefixes (default tssdp_)
Policy	mediaPolicy: auto (default — media input forces agent mode + contract), reject (explicit fast-fail with diagnosis), pass_through (hand to MCP)
Output	structured JSON + optional top-level artifacts [{path, mime}] — engine validates existence & magic bytes (png/jpeg/webp/gif/mp4/mp3/wav/pdf); invalid → node failed with audit trail
By default denied	write/edit/shell (pwsh/bash/run_code) and re-orchestration (subagent/workflow/cortex_*/…) — an explicit per-node tool_allow swaps in a custom allow list for artifact-producing tasks
Honest limits	DSH provider modalities are text + image only — audio/video understanding requires a media-capable MCP/multimodal route; otherwise the executor returns the {"unsupported": true, "reason": ...} contract (never invents content), or mediaPolicy: reject fails fast with a clear diagnosis

Example flows: image analysis → cortex_execute on a node with image references (agent mode + read_image/MCP); screenshot/poster generation → Playwright MCP + artifacts declaration (magic-byte validated); audio/video processing → only with a media MCP configured (else explicit diagnosis, classified INPUT_FAILURE).

Preferred-model fallback (round 70/73): when a preferred executor (e.g. a text-only L1 model) is handed an image task, it runs first (attempt is recorded). The executor now pre-checks modality deterministically (modelAcceptsImage via host llm.resolveModelInfo): a model that does not declare image input fails fast with model lacks image modality (no generation — a text model must never be allowed to fabricate read-image evidence; observed in production: invented 800x600 / white-pixel result for a 1x1 red PNG). The recovery engine detects the gap and switches directly to a vision-capable candidate — no unnecessary decompose/upgrade round-trip.

Host prerequisite for vision: the vision model must declare inputModalities: [text, image] in the host's llm-deepseek.models entry (e.g. ~/.dsh/settings.yaml) — otherwise read_image refuses with "model does not declare image input" (the catalog default for undeclared entries is [text]).

Host prerequisite for vision: the vision model must declare inputModalities: [text, image] in the host's llm-deepseek.models entry (e.g. ~/.dsh/settings.yaml) — otherwise read_image refuses with "model does not declare image input" (the catalog default for undeclared entries is [text]).

Cost Model

• Executor models (sub-models): exact TokenUsage from subagent/chat sessions (billed = input + cache read/write), priced from the registry — true token accounting.
• Supervisor (primary model): real session-model usage via DSH tokenMeter/sessionProjections (fallback estimate), plus a "vs direct primary-model run" baseline saving in cortex_report.
• UI-triggered evaluations run without a supervisor (chat path) and are not counted as supervisor calls.

Development

npm test                 # node --test (engine/service/ui unit + smoke) — 141 tests
node tools/verify-post-restart.mjs   # post-restart self-check (5/5)
node tools/smoke-ui.mjs              # UI script syntax + SSE + model toggle

Repository Layout

app-pkg/index.js   entry forwarder (rollout cache-bypass, same pattern as dsh-tssdp)
lib/index.js       plugin entry (name/inject/Config/apply + lifecycle)
lib/service.js     CortexService facade (high-level operations behind the tools)
lib/tools.js       12 agent tools
lib/cli.js         /cortex command group
lib/ui-server.js   local web console (127.0.0.1:8788)
lib/engine/*.js    pure engines (fingerprint/router/budget/quality/recovery/store/executor/kpi/granularity)
config/models.yaml model registry (overridable)
test/              node --test suites
tools/             verification/dev scripts

License

MIT