Overview · Development · Testing · Grading · Listing · Risk · Ranking

Standard No.DSHIO/T 02—2026TitleTestingVersionv1.0Published2026-11-06Effective2026-11-06StatusCurrent

DSHIO-TEST · Testing

Six gates, from static checks to tool invocation. Each has a firm verdict: pass, fail, or "untested" when it cannot be run.

Why these tests. "Installs" is not "works". Of 122 top plugins we measured, 14% could not install at all; of those that did install, some never loaded (silent failure) and some loaded but their tools could not be invoked.

The six gates climb from "it installs" to "it works". Each gate rules out a class of failure.

The six gates

Level	Name	Checks	Passes when	Network
T0	Static	Manifest validity, required fields, `engines.dsh`	Parses and declares no upper bound	—
T1	Install	`dsh plugin add`	Exit code 0	on
T2	Listed	`dsh plugin list`	Appears in dependencies	off
T3	Mounted	`dsh --dump-config`	Appears in the config tree	off
T4	Boot	Start the profile, observe 12 s	Process alive, no fatal error	off
T5	Tool call	Invoke a declared tool	No throw, no timeout	off

Why T2–T5 run offline. Install needs the network, so T1 has it. Once installed, T2–T5 run inside --network none, so a malicious plugin has nowhere to send data. That boundary is the most important difference between this and install-only testing.

T0 · Static

Check	Consequence if it fails
`package.json` parses	Testing aborts
`dsh.bundle.patch` exists	DSH cannot recognise the plugin
The patch file exists	Load failure
`private` is not `true`	Not publishable
`engines.dsh` has no upper bound	Flagged as `VERSION_GATE` risk

T1 · Install

dsh plugin --profile web add <package>

Run in a clean profile with a 180 s timeout. Passes when the exit code is 0 and the dependency tree is complete.

T2 · Listed

dsh plugin --profile web list

The plugin appears in dependencies. This separates "downloaded" from "registered".

T3 · Mounted

dsh --profile web --dump-config

The plugin appears in the final config tree. This catches silent failure: installed, listed, but never mounted — usually a slot conflict or a manifest problem.

T4 · Boot

Boot the profile and observe 12 s. Passes when the process stays alive with no fatal exception. Catches ESM defects, injection failures and init throws.

T5 · Tool call

The gate that separates "loaded" from "usable".

How it runs

Read the tool names and parameter schemas from --dump-config
Generate one minimal valid argument set per schema
Invoke once with a 10 s timeout
Passes when there is no throw and no timeout

When it cannot run

No tool list → "T5 untested", never "failed". Some plugins provide only UI or background services and register no tools; some schemas cannot be resolved from the config tree. Both are marked untested and do not lower the grade ceiling (see grade C in DSHIO-GRADE).

Treating "cannot test" as "failed" is the common defect of grading systems. We do not do it.

Failure codes

Every failure is classified into one of nine codes. This is where DSHIO differs most from other directories: they report that it failed; we report why.

Code	Meaning	Seen at	Whose problem
`VERSION_GATE`	DSH version mismatch; plugin skipped	T1–T3	Author
`ESM_DEFECT`	ESM loader defect; module judged unloadable	T4	Author / DSH
`SLOT_CONFLICT`	Loader Entry ID duplicates an installed plugin	T3	Author
`SILENT_FAIL`	Installed but never mounted, no error	T3	Author
`ENV_MISMATCH`	Node / pnpm environment mismatch	T1	Author
`NPM_404`	Package absent from the registry	T0–T1	Author
`NO_TOOL`	No tools declared (T5 untested)	T5	—
`T5_ERROR`	Tool invocation threw	T5	Author
`T5_TIMEOUT`	Tool invocation timed out	T5	Author

Report format

{
  "plugin": "dsh-pet",
  "version": "1.2.3",
  "dsh_version": "0.2.2-rc.1",
  "tested_at": "2026-11-06T07:20:00Z",
  "stages": {
    "T0": { "pass": true, "ms": 120 },
    "T1": { "pass": true, "ms": 8600 },
    "T2": { "pass": true, "ms": 900 },
    "T3": { "pass": true, "ms": 2100 },
    "T4": { "pass": true, "ms": 12000 },
    "T5": { "pass": true, "ms": 2400, "tool": "pet_feed" }
  },
  "verdict": "pass",
  "failure_code": null,
  "evidence": "results/dsh-pet-0.2.2-rc.1.json"
}

What we promise

Promise	Not promised
T2–T5 run fully offline	Security certification (a separate system)
Every failure is classified	Functional correctness (only that tools respond)
Untestable is marked untested, not failed	Coverage of every usage scenario
Reports are reproducible	Liability for plugin behaviour

Version history

Version	Standard No.	Published	Change
v1.0	`DSHIO/T 02—2026`	2026-11-06	First release

The version increments on any clause change; older versions remain available.