# Validation: LLM routing and cost-quality measurement

## Окружение

| Параметр | Фактическое значение |
|---|---|
| Дата проверки | 2026-07-25, Asia/Ho_Chi_Minh |
| Commit | `68582c97cb6b51804532cbd021290f07f4f74fa5`; dirty working tree |
| OS и hardware | macOS 26.4 build 25E246, arm64 |
| Runtime | Node.js 24.14.0; npm 11.9.0; TypeScript 7.0.2 |
| Внешние сервисы | none |
| Входные данные | 3 fixed cases, seeds 20260725–20260727, 20 repeats/strategy |

## Результаты по критериям

| ID | Проверка | Ожидалось | Получено | Evidence | Результат |
|---|---|---|---|---|---|
| AC-01 | Manifest/case parity | same hash/seeds | hashes identical (`83cc1063…8f77`), 3 cases each | `E-004`, `E-005` | `pass` |
| AC-02 | Machine-readable metric schema | all required fields; unavailable values explicit | usage/bytes/latency/rates present; costs `null`; quality `not_measured` | `E-006`–`E-009` | `pass` |
| AC-03 | Guardrails beside usage | deterministic defects visible; fake-model unknowns explicit | rule/state/privacy `0`; replay/structured `1`; contradiction/facts/retry `null` | `E-008` | `pass` |
| AC-04 | Provider calls | none | `fake_local`, `realProviderCalls: 0`; explicit permission gate | `E-009` | `pass` |
| AC-05 | Non-inferiority labeling | no invented claim | quality `not_measured`; non-inferiority `not_tested` | `E-009` | `pass` |

## Команды проверки

```bash
node RnD/technical/2026-07-25-llm-routing-cost-quality/prototype/run.ts
node --test RnD/technical/2026-07-25-ai-gm-vertical-slice/prototype/tests/*.test.ts
npm run test:technical
npm run typecheck
npm run technical:benchmark
```

## Метрики

Один fresh wrapper run; latency не имеет допуска и приведена только для
воспроизводимости локального report.

| Метрика | Full-context baseline | Projection/delta candidate | Вывод |
|---|---:|---:|---|
| Manifest hash | `83cc1063…8f77` | `83cc1063…8f77` | identical |
| Cases × repeats | 3 × 20 | 3 × 20 | identical |
| Input bytes | 650060 | 136640 | candidate меньше |
| Input token proxy | 162515 | 34160 | `synthetic_proxy`, не provider tokens |
| Proxy reduction | — | 78.98040180906378% | local payload evidence only |
| Output bytes / proxy | 11500 / 2875 | 11500 / 2875 | identical fake output |
| p50 latency, ms | 0.000458 | 0.000417 | fake in-process only |
| p95 latency, ms | 0.001708 | 0.000500 | fake in-process only |
| Structured-output success | 1 | 1 | deterministic fake fixture |
| Invalid transition / private leakage | 0 / 0 | 0 / 0 | central guardrails |
| Replay consistency | 1 | 1 | central deterministic guardrail |
| Contradiction / fact preservation / retry | `null` / `null` / `null` | `null` / `null` / `null` | not measured by fake adapter |
| USD cost per turn/session | `null` / `null` | `null` / `null` | Unknown |
| Narration quality | `not_measured` | `not_measured` | Unknown |
| Non-inferiority | `not_tested` | `not_tested` | Unknown |

Integrated `npm run technical:benchmark` вернул `status: pass` и те же
proxy/cost/quality summary values, что wrapper (`E-011`).
Expanded technical suite: `41` tests, `41` pass, `0` fail (`E-012`).

## Ошибки и нестабильность

No execution errors. Latency varies at sub-millisecond scale and is neither a
provider measurement nor a decision threshold. Stable comparison fields are
manifest hash, byte/proxy totals and deterministic fixture metrics.

Repository `npm run check` прошёл: 4 site tests, 41 technical tests, typecheck,
10-study validator, 96-document build и rendered-link validation с 0 failures.

## Воспроизводимость

Wrapper and central benchmark command were executed in the same checkout.
Independent machine/provider repetition was not performed. The exact
machine-readable report is reproducible locally, except `generatedAt` and
timing samples.

## Финальный исход

- **Outcome:** `partial`
- **Обоснование:** все harness-safety AC выполнены (`E-004`–`E-011`) и local
  payload proxy materially decreased, но real cost-quality research остаётся
  Unknown by design.
- **Что результат не доказывает:** provider token/cost savings, cache economics,
  production latency/retries, structured-output reliability или отсутствие
  narration quality loss на real models.