RFC 0007: Diagnostics, Testing, and Release Evidence

Status: Accepted
Areas: Appendix G.16, G.18; Appendix H.1–H.3, H.6, H.14–H.18

Decision: Accepted by the project owner on 2026-10-23. This approves the design direction; implementation and conformance evidence remain outstanding.

Implementation progress

The following slice is present in the source tree, but does not satisfy RFC 0007 as a whole:

• scripts/runtime-c-sanitizers.sh builds and runs the JSON-depth, arena, and secure-random C harnesses with AddressSanitizer and UndefinedBehaviorSanitizer.
• .github/workflows/runtime-security.yml is configured for pushes, pull requests, weekly runs, and manual dispatch. It requires libFuzzer, runs 50,000 bounded parser-fuzz iterations using six version-controlled seed cases, and uploads artifacts after a failed run.
• scripts/promote-http-fuzz-crash.sh records a reviewed minimized input under a SHA-256-derived filename in tests/fuzz-corpus/. The corpus builder includes saved regressions, and the deterministic fallback replays the corpus before running mutations.
• tests/fuzz-regression-pipeline.sh verifies repeat promotion, symlink and size rejection, corpus assembly, and ASan/UBSan replay using a temporary synthetic regression; the security workflow runs this check before coverage-guided fuzzing. No real crash input has been promoted into the repository yet.
• The manual artifact review and promotion steps are documented in the fuzz regression workflow.
• On 2026-10-27, the native compiler added statement-level source ranges for errors raised while compiling a source line. The CLI reports zero-based UTF-8 byte columns; the VS Code extension converts them to UTF-16 editor columns. tests/check-diagnostics.sh checks the JSON range, while editors/vscode-yaza/test/extension-diagnostics.test.js verifies an unsaved buffer with a non-ASCII trailing comment maps to the full statement range. The macOS bootstrap passed three self-hosted stages and 41/41 language tests; the diagnostics CLI smoke passes with normal and ASan/UBSan binaries, and the VS Code extension suite passes. These are statement spans, not token-exact spans; errors raised before statement compilation may still use line-only ranges, imported-source mapping is incomplete. At the earlier two-error checkpoint, multi-error recovery could return up to two errors per scope when the first failure was a recoverable typed initializer, supported return mismatch, or non-Boolean assert; other same-function and global recovery remain incomplete.
• On 2026-10-27, yaza check --format json initially recovered up to two diagnostics per function and start: body for import-free, non-API files up to 96 KiB, 4,096 lines, and 32 functions. It launches isolated self-hosted compiler probes with unrelated function bodies replaced by supported type-correct stubs, then preserves original source URIs and line spans in the aggregate version-1 report. tests/check-diagnostics.sh then verified two initializer errors within one function, a return mismatch followed by a return-placement error, a non-Boolean assert followed by a typed declaration error, preservation of a later use of the recovered binding, diagnostic order, paths, and JSON-only stdout. At that stage, one isolated retry could repair a typed initializer, supported return mismatch, or non-Boolean assert using temporary type-correct code; API files, imports, global declaration errors, unsupported stub return types, and other same-function recovery remain incomplete. The updated macOS bootstrap passed with the smoke running against both normal and ASan/UBSan compiler binaries.
• On 2026-10-27, bounded recovery was extended to up to three diagnostics per eligible function or start: body, under a shared 128-probe budget. Each scope gets at most three isolated compiler probes. Temporary repairs cover explicitly typed let/var initializer mismatches, supported return mismatches, and non-Boolean assertions; they preserve names and line positions and affect only child source. tests/check-diagnostics.sh now verifies three sequential initializer errors, a return mismatch followed by a return-placement diagnostic, an assertion error followed by a typed initializer error, and exact message order. API/import/global recovery and other unrecoverable same-scope error categories remain incomplete.
• On 2026-10-27, yaza audit --security --format sarif began emitting SARIF 2.1.0. The report includes the four stable audit rule descriptors, matching ruleIndex values, warning-level results, line regions when available, percent-encoded relative artifact URIs, and YAZA severity/confidence metadata. Candidate credential values remain redacted. tests/audit-sarif.sh checks finding and empty reports, UTF-8 and reserved-character URI encoding, line-less scan-depth results, rule-index consistency, invalid formats, and secret redaction; the macOS bootstrap runs it with jq. Findings keep the existing nonzero exit status. The audit itself is still heuristic and does not inspect dependencies, perform taint analysis, or verify runtime behavior.
• yaza test sorts directory entries, builds each test from the project root, and runs each executable in a private per-test working directory. A 30-second combined build-and-run budget defaults per test; YAZA_TEST_TIMEOUT_MS can set an integer from 1 to 3,600,000 ms, and yaza test [directory|file] --timeout <milliseconds> takes precedence. A timeout is reported separately, its process group is terminated, and discovery continues. Temporary cleanup rejects paths outside /tmp/yaza-run.<six alphanumeric characters>, verifies ownership and mode 0700, does not follow symlinks, and is bounded to 64 directory levels and 100,000 entries. This isolates relative working-directory files but is not an OS sandbox. yaza test --format json emits a versioned JSON report with one record per test (path, status, phase, exit_code, and timed_out) plus a summary; child output is redirected to stderr. The resilience script verifies pass, compile-failure, runtime-failure, timeout, isolation, outside-path rejection, symlink safety, and continuation. The full 22-test suite and regression script pass. Database-container fixtures remain absent.
• On 2026-10-24, yaza check --format json [file.yz] was added with the versioned yaza.diagnostics schema, percent-encoded file URIs, coarse stable codes (YZA1000 for compiler errors with a mapped line; YZA0001 for CLI/input failures and unmapped errors), zero-based line-level ranges, and a nonzero exit status for errors. tests/check-diagnostics.sh verifies valid input, a type error, missing input, URI escaping, project-entrypoint selection, and JSON-only stdout; it passed with native and ASan/UBSan CLI builds. The 22-test suite also passed after this source change, and two self-hosted compiler stages emitted identical C. At that stage the command reported one fatal error at a time and had no column ranges; the later statement-span update above adds partial column support. Parser failures can still fall back to line 1, so the acceptance criteria for full diagnostics remain incomplete.
• After these changes, yaza test tests passed all 22 current YAZA test programs. This does not establish CI/hosted-runner evidence.
• Local verification on 2026-10-23 passed shell/YAML parsing, all three sanitizer harnesses, 10,000 deterministic sanitizer mutations, and all 15 YAZA language tests. The local macOS Clang lacks libFuzzer, so coverage-guided execution and the hosted workflow remain unverified.
• The workflow does not yet run the compiler/API/PostgreSQL/target-build matrix, benchmark or soak suite, or release provenance checks. Fuzz crashes are artifacts requiring review; they are not automatically promoted to permanent regression tests. Repository branch-protection requirements are also not verified.

Summary

Make compiler/runtime behavior diagnosable and make security and performance claims depend on repeatable evidence. Define diagnostic structure, test categories, fuzzing, benchmarks, CI gates, release artifacts, and the process required before a stable release.

Proposal

1. Compiler diagnostics have stable codes, severity, primary span, related spans, and machine-readable JSON output. The compiler reports multiple independent errors in one run while suppressing cascades that depend on invalid syntax/types.
2. Every token and AST node retains source file, line, and column through imports and generated code. Build failures map to original .yz locations; generated C locations are secondary details.
3. The test runner separates discovery, compile, and execution results, continues across independent failures, supports timeout/isolation, and exits non-zero if any case fails. Test fixtures and temporary directories are isolated per test.
4. CI runs unit, negative compiler, API integration, PostgreSQL integration, sanitizer, parser-fuzz regression corpus, and target-build checks. Coverage-guided fuzzing runs with bounded time on every change and longer scheduled runs; crashes are retained as regression inputs.
5. Benchmarks compare identical workloads and report throughput, p50/p95/p99 latency, RSS, startup time, binary size, CPU/toolchain versions, and connection policy. Results are descriptive and are not a release gate until variance and environment controls are documented.
6. Runtime soak tests include slow clients, large but valid input, malformed input, cancellation, repeated restart, 100,000+ requests, and RSS trend measurement. A memory-growth threshold must account for allocator behavior and test noise.
7. yaza audit emits a machine-readable report with rule identifiers, confidence, evidence locations, suppressions, and tool version. Heuristic findings are labeled; the command does not claim taint analysis or dependency scanning unless those analyzers actually run.
8. Release builds record compiler source revision, bootstrap stage hashes, build flags, dependency identities, SBOM, provenance, and signatures. Reproducible build and independent rebuild checks are release goals, not assumed properties.
9. A public release requires a threat model, vulnerability-reporting process, incident owner, supported-version policy, security response targets, and a documented security gate. No placeholder contact counts as a reporting channel.
10. Appendix H/OWASP mappings point from each claim to source code, test evidence, limitations, and operator responsibilities. Any failed or missing evidence blocks the associated claim.

Security considerations

Tests and scanners provide evidence but do not prove absence of vulnerabilities. Fuzzing scope, duration, instrumentation, and coverage must be reported honestly. Release signatures establish artifact identity, not code safety.

Staged implementation

1. Add spans/diagnostic codes and multi-error reporting to parser/type-checker paths.
2. Add isolated fixtures and machine-readable test results.
3. Run sanitizer and parser fuzz corpus in CI; maintain crash regressions.
4. Add PostgreSQL integration, slow-client, soak, and benchmark reports.
5. Establish release security process and signed provenance before announcing stable support.

Acceptance criteria

• Compiler diagnostics remain stable enough for editor integration and link to original .yz source locations.
• A deliberately malformed source file reports multiple independent errors without crashing or emitting misleading cascades.
• CI evidence is retained with toolchain and environment metadata and can be reproduced by a contributor.
• Fuzz regressions, test failures, sanitizer findings, and signature failures block release jobs.
• Public security claims link to tests/review evidence and list known limitations.

Decisions for the owner-specified open questions (2026-10-24)

Diagnostic and editor interface

yaza check --format json is the stable compiler diagnostic interface, using a versioned yaza.diagnostics JSON schema with stable diagnostic codes, severity, source URI, start/end positions, message, related locations, and notes. Human-readable terminal output remains the default. The language server will translate compiler diagnostics into the Language Server Protocol's diagnostic messages rather than inventing a second editor protocol. yaza audit --security --format sarif emits SARIF 2.1.0 for the current heuristic security findings; compiler JSON remains the direct editor interface. This keeps editor diagnostics easy to consume while allowing security results to integrate with code-scanning systems. See the OASIS SARIF 2.1.0 standard and GitHub SARIF support.

CI platforms and architectures for signed releases

Every pull request must pass on Ubuntu 24.04 x86_64 and macOS 15 ARM64. A signed release must build and run conformance tests natively for every target it claims to support: macOS ARM64 and x86_64, Linux x86_64 and ARM64, and Windows x86_64; WASM/edge, Android ARM64, and iOS ARM64 enter the release gate when those targets are declared supported. Cross-compilation alone does not qualify a target. Until those target jobs exist and pass, the target must not be advertised as release-supported. GitHub Actions is the current reference CI; runner labels and availability must be reviewed when updating the pinned image matrix. GitHub's hosted-runner reference currently lists Linux ARM64 and macOS ARM64/Intel runners, alongside Windows x64 runners (runner matrix). Release artifacts require detached signatures and provenance tied to protected CI identity. Such signatures identify the artifact but do not replace Apple's Developer ID signing or notarization for Gatekeeper acceptance.

Initial memory and latency gates

These are initial engineering gates, not measured product SLOs. Benchmark reports must identify OS, architecture, CPU, compiler mode, workload, connection policy, and revision; run five samples; and only enforce relative thresholds when sample variation is at most 5%.

• CLI profile on a pinned native runner: cold yaza --version p95 startup at or below 250 ms and peak RSS at or below 64 MiB.
• Local native API profile without PostgreSQL: p99 at or below 50 ms for /healthz and 100 ms for in-memory CRUD under 16 loopback clients. After a 10,000-request warm-up followed by 100,000 sequential requests, peak RSS growth must stay at or below 16 MiB, with at most 4 MiB growth during the final 10,000 requests.
• PostgreSQL-backed profile: measure and publish p50/p95/p99 and RSS using a pinned server version and configuration. Enforce a 15% maximum regression against the rolling median of the previous ten successful same-target runs; do not claim a fixed end-to-end latency until the database fixture and runner are stable.
• WASM, edge, and mobile profiles: establish target-native baselines and budgets before declaring release support; do not reuse native server limits as evidence for those targets.

A same-target p99 or memory regression above 15% blocks a signed release when the measurement-variation requirement is met. These gates must be revisited after benchmark and hosted-CI evidence exists; the current 100,000-request RSS observation does not prove all the thresholds above.

← Kembali ke semua dokumentasiVersi dan status rilis →