RE Lab

Reproducible model-behavior experiments and verifiable reverse-engineering evaluations.

简体中文 · DSH chat practice · Architecture · Offline demo · Research review · Strategy landscape · Evaluation protocol

Try it in DSH chat

With the plugin installed, restart the official DeepSeek Harness Desktop, open a chat, and use the model you already selected. Send:

开始逆向体验

The model loads a small WebAssembly reverse-engineering task with re_lab_start, analyzes the sample, and submits trialId, mask, and bias through re_lab_submit. The evaluator checks all 256 inputs and returns the result. You do not need to find a sample or run a model command yourself.

First-time setup from the project root: npm ci, then npm run dsh:install. Fully quit Desktop for installation and restart afterward; the script does not close or restart the app. No handwritten profile configuration is needed. JSON and Markdown reports default to $DSH_HOME/storages/re-lab-practice (~/.dsh/storages/re-lab-practice when unset). See the practice guide for setup, reports, and troubleshooting.

One real Desktop toy-practice smoke test passed on official 0.2.1-rc.3, with the UI selecting DeepSeek-V4.1-Flash 1017 / High. Both tools completed, all 256 inputs passed, and JSON/Markdown reports were generated. Practice records are conversation_submission with notBenchmark: true; this one check does not establish a controlled request, authenticated model snapshot, token usage, model ranking, or general reverse-engineering ability.

Project status

Version 0.1.0-alpha.1; 142 local tests passed, along with type checking and plan validation. The previous prototype has been retired. The repository has recorded execution/replay and native DSH chat practice. An isolated official Cordis/ToolRuntime check verified start/submit with fixture agent identities and a fixture answer, without a model call or user profile. A separate real Desktop smoke test completed the toy exercise with the current chat model, without a nested LLM request or oracle files supplied to its workspace; see the practice guide. The roadmap and offline guide retain the separate recorded-pipeline acceptance gate.

Implemented components include immutable run plans, bounded JSON validation, a pure lifecycle/accounting state machine, five ports, application orchestration, memory/file artifact stores, a recorded provider, a baseline strategy, independent toy-case evaluators, and campaign metrics. The offline CLI composes these components around one authored WebAssembly case.

The two saved responses are synthetic negative and positive controls. They test storage, grading and replay; they are not model outputs, a strategy comparison, a jailbreak-success measurement or an RE benchmark result. The native chat tools reuse the current DSH conversation. A controlled live-provider adapter, strategy-search algorithms, and Ghidra/IDA bridges remain unimplemented.

Research question

Which model, strategy and tool configuration improves verified task completion under a fixed evaluation protocol?

Refusal, behavior compliance, substantive answer coverage and technical correctness are different measurements. A candidate that answers more requests can still become less accurate. The project records refusal separately from task quality, preserves failures and missing outcomes, and requires a named evaluator and denominator for every rate.

The initial RE protocol uses authored programs, known ground truth and independent oracles. Model-behavior strategies will be compared as versioned experimental conditions; there is no universal “strongest jailbreak” claim.

Research before implementation

The review examines StrongREJECT, HarmBench, JailbreakBench, garak and PyRIT, including differences in scoring versions and denominators. The RE protocol draws from CrackMeBench, CREBench, Cybench and other primary sources. METR's RE-Bench measures research engineering, not binary reverse engineering.

The architecture also examines DSH ArmorBreak, reverse-skill packs, GhidraMCP and the official IDA MCP. A skill pack, provider router, analysis bridge and refusal-bypass strategy have different responsibilities; their names do not establish their capabilities.

The strategy landscape reviews GCG, PAIR, TAP, many-shot research and refusal-direction interventions. It separates API search, long-context and white-box access requirements, recording/cost models and generalization limits. These are research references, not implemented algorithms or current effectiveness guarantees.

Design principles

• Host-neutral domain and orchestration. Core, ports and application import no DSH, Cordis, provider SDK, filesystem, process or network implementation.
• Explicit authority. A trusted operator supplies a reviewed plan. Model text and sample content remain observations, including text that claims authorization.
• Independent assessment. A completed run may be refused, wrong, correct or indeterminate. Transport failures are not model refusals; a scorer failure preserves any successfully scored axis.
• Frozen experiment identity. Plans pin case, strategy, model snapshot, evaluator and tool-manifest identities, input hashes, generation settings, retry ceilings and budgets.
• Evidence before claims. Execution verifies artifact bytes and evidence ranges; replay checks journal commitments and recomputes grading without calling a provider. Hashes establish identity, not external authenticity.
• Small, justified structure. One TypeScript repository first. Additional packages or services require an actual dependency or release need.

Current repository

README.md                         English canonical entry point
README.zh-CN.md                   Chinese entry point
CONTRIBUTING.md                   Contribution and documentation rules
src/
  core/                          Contracts, bounded validation, canonical JSON, state machine
  ports/                         Provider, strategy, evaluator, artifact store, RE bridge
  application/
    run.ts                       Port orchestration, journal and terminal settlement
    replay.ts                    Evidence and independent assessment replay
    artifacts.ts                 Bounded observation/envelope/evidence verification
    evaluation.ts                Independent axis-to-event normalization
    metrics.ts                   Scheduled-population reconciliation and rates
    practice.ts                  Session-bound conversation submissions and assessment
  adapters/
    artifacts/                   Memory and persistent file stores
    providers/recorded.ts        Request-bound saved responses; no network
    strategies/baseline.ts       Versioned task-to-message preparation
    evaluators/toy-codec.ts      Fixture loading and deterministic scoring
    experiments/toy-demo.ts      Toy-case composition
    cli/                         Plan validation and offline demo/replay commands
    dsh/plugin.ts                Native chat tools, host wiring and readable reports
scripts/
  install-dsh.mjs                 Build, package and install through the official CLI
  verify-dsh-host.mjs             Isolated official runtime check; no model or profile
cordis.patch.yml                 Bundle registration for the native plugin
locale/                          English and Chinese plugin display metadata
tests/
  architecture/                  Dependency direction
  contracts/                     Domain validation and lifecycle
  adapters/                      Storage, provider, strategy and toy evaluator
  application/                   Orchestration, replay, failure and metric regressions
examples/
  run-spec.json                  Illustrative plan with placeholder hashes
  toy-codec/                     Public WASM/task, synthetic controls, evaluator-side oracle
docs/
  en/                            Architecture, research, protocol, demo, roadmap and ADRs
  zh-CN/                         Corresponding Chinese documents

The RE bridge is an interface only. No empty integration directory is presented as a working tool.

Develop and check offline

Node.js 22 or newer:

npm ci
npm test
npm run plan:check
npm run demo:offline
npm run replay:offline

plan:check validates the example plan's shape and bounds; its hashes are placeholders. demo:offline uses real fixture hashes and writes a new runs/offline-demo directory. It refuses an existing output directory. replay:offline verifies the saved campaign and re-evaluates the saved answers without a model request. See the offline guide for output files, alternate directories, hash anchoring and verification status.

The offline CLI has no runtime package dependencies. The native DSH adapter uses optional host peer dependencies supplied by DSH; its development SDK and tooling are pinned in the lockfile. TypeScript 7's pinned lexical API checks dependency direction; it is an architectural regression gate, not a hostile-code sandbox.

Reading the results

The aggregator keeps all scheduled units in N, including missing or unsuccessful runs. It reports C/N for retained correct assessments and E/N for completed runs with a correct assessment. These differ when one scoring axis succeeds but the other fails. Reports also show G/N answer coverage, C/G conditional correctness, R/N scheduled refusal, R/A assessable-response refusal and A/N refusal coverage. Empty denominators are null; retries are not extra tasks.

The runner's token total is a budget debit, not a bill. Unknown provider usage reserves the remaining allowance. The synthetic recorded demo has no measured model tokens or model cost; a reserved value must not be reported as measured usage.

Tests exercise local contracts, storage, deterministic evaluation and failure handling. They do not establish live-provider behavior, jailbreak effectiveness, general RE ability, hostile-code isolation or a successful Desktop chat interaction. An uncooperative operation may outlive cancellation; terminal records expose whether cleanup settled or remains unconfirmed. The chat-practice guide explains why numeric conversation submissions do not supply benchmark refusal populations or controlled request/usage evidence.

Implementation sequence

1. Recorded vertical slice: one authored case, synthetic controls, baseline strategy, deterministic oracle, persistent artifacts, replay and denominator reconciliation. Local acceptance checks passed.
2. One live provider: explicit endpoint configuration, cancellation, usage/errors and versioned request artifacts. Not started.
3. Controlled strategy comparison: frozen candidates, held-out cases, equal budgets, calibrated scoring and uncertainty.
4. DSH adapter: the native toy tools pass an isolated official host-runtime check and one real Desktop practice smoke test. Controlled evaluation integration remains a separate gate.
5. One RE bridge: explicit artifact/database identity and read capabilities, with separate integration evidence.

Features advance when their acceptance gates pass.

Documentation and license

English is canonical for source, identifiers, comments, CLI messages and normative documents. Chinese versions mirror the same scope and metrics. Translations do not alter schemas or claim unimplemented features.

MIT License. See LICENSE.