Validation: LLM routing and cost-quality measurement
Окружение#
| Параметр | Фактическое значение |
|---|---|
| Дата проверки | 2026-07-25, Asia/Ho_Chi_Minh |
| Commit | 68582c97cb6b51804532cbd021290f07f4f74fa5; dirty working tree |
| OS и hardware | macOS 26.4 build 25E246, arm64 |
| Runtime | Node.js 24.14.0; npm 11.9.0; TypeScript 7.0.2 |
| Внешние сервисы | none |
| Входные данные | 3 fixed cases, seeds 20260725–20260727, 20 repeats/strategy |
Результаты по критериям#
| ID | Проверка | Ожидалось | Получено | Evidence | Результат |
|---|---|---|---|---|---|
| AC-01 | Manifest/case parity | same hash/seeds | hashes identical (83cc1063…8f77), 3 cases each |
E-004, E-005 |
pass |
| AC-02 | Machine-readable metric schema | all required fields; unavailable values explicit | usage/bytes/latency/rates present; costs null; quality not_measured |
E-006–E-009 |
pass |
| AC-03 | Guardrails beside usage | deterministic defects visible; fake-model unknowns explicit | rule/state/privacy 0; replay/structured 1; contradiction/facts/retry null |
E-008 |
pass |
| AC-04 | Provider calls | none | fake_local, realProviderCalls: 0; explicit permission gate |
E-009 |
pass |
| AC-05 | Non-inferiority labeling | no invented claim | quality not_measured; non-inferiority not_tested |
E-009 |
pass |
Команды проверки#
node RnD/technical/2026-07-25-llm-routing-cost-quality/prototype/run.ts
node --test RnD/technical/2026-07-25-ai-gm-vertical-slice/prototype/tests/*.test.ts
npm run test:technical
npm run typecheck
npm run technical:benchmark
Метрики#
Один fresh wrapper run; latency не имеет допуска и приведена только для воспроизводимости локального report.
| Метрика | Full-context baseline | Projection/delta candidate | Вывод |
|---|---|---|---|
| Manifest hash | 83cc1063…8f77 |
83cc1063…8f77 |
identical |
| Cases × repeats | 3 × 20 | 3 × 20 | identical |
| Input bytes | 650060 | 136640 | candidate меньше |
| Input token proxy | 162515 | 34160 | synthetic_proxy, не provider tokens |
| Proxy reduction | — | 78.98040180906378% | local payload evidence only |
| Output bytes / proxy | 11500 / 2875 | 11500 / 2875 | identical fake output |
| p50 latency, ms | 0.000458 | 0.000417 | fake in-process only |
| p95 latency, ms | 0.001708 | 0.000500 | fake in-process only |
| Structured-output success | 1 | 1 | deterministic fake fixture |
| Invalid transition / private leakage | 0 / 0 | 0 / 0 | central guardrails |
| Replay consistency | 1 | 1 | central deterministic guardrail |
| Contradiction / fact preservation / retry | null / null / null |
null / null / null |
not measured by fake adapter |
| USD cost per turn/session | null / null |
null / null |
Unknown |
| Narration quality | not_measured |
not_measured |
Unknown |
| Non-inferiority | not_tested |
not_tested |
Unknown |
Integrated npm run technical:benchmark вернул status: pass и те же
proxy/cost/quality summary values, что wrapper (E-011).
Expanded technical suite: 41 tests, 41 pass, 0 fail (E-012).
Ошибки и нестабильность#
No execution errors. Latency varies at sub-millisecond scale and is neither a provider measurement nor a decision threshold. Stable comparison fields are manifest hash, byte/proxy totals and deterministic fixture metrics.
Repository npm run check прошёл: 4 site tests, 41 technical tests, typecheck,
10-study validator, 96-document build и rendered-link validation с 0 failures.
Воспроизводимость#
Wrapper and central benchmark command were executed in the same checkout.
Independent machine/provider repetition was not performed. The exact
machine-readable report is reproducible locally, except generatedAt and
timing samples.
Финальный исход#
- Outcome:
partial - Обоснование: все harness-safety AC выполнены (
E-004–E-011) и local payload proxy materially decreased, но real cost-quality research остаётся Unknown by design. - Что результат не доказывает: provider token/cost savings, cache economics, production latency/retries, structured-output reliability или отсутствие narration quality loss на real models.