Proposal: YAZA Base SQL Database

Status: Proposed; accepted by the project owner for proposal work, not yet implemented
Product name recommendation: YAZA Base
CLI surface: yaza db ...
Engine process name: yazad (provisional)

1. Executive summary

YAZA Base is a proposal for a SQL database engine designed and implemented as a first-party part of the YAZA ecosystem. It is intended to cover two different needs over time: a durable embedded database for local applications and a client/server database for API services and cloud deployments.

The project already has a separate proposal for drivers that connect YAZA applications to PostgreSQL, SQLite, MySQL/MariaDB, and other external databases. Those drivers remain useful and are not replaced by this proposal. YAZA Base is the database product whose storage engine, SQL execution, transactions, wire service, and administration tools are owned by YAZA.

The proposal does not claim that a new engine can be more reliable or faster than mature databases before implementation and independent measurements. Its goal is to combine dependable database fundamentals with unusually direct language integration: schema-aware YAZA types, safe prepared queries, migrations, API contracts, and later change streams can share one supported toolchain.

2. Naming recommendation

Recommend YAZA Base as the product name. “Base” communicates a foundational data service and is easy to say in product documentation. Use:

“YAZA DB” is a useful short description, but it is generic as a product name. “YAZA Store” is broad and can be confused with object storage. “YAZA Base” is the clearest of these options for a general-purpose SQL product. A quick public web search did not establish whether the name is legally available; complete trademark, package registry, domain, and app-store checks before a public release.

The command name yaza db avoids introducing a second top-level executable for everyday use. yazad can be shipped as an implementation detail for service managers and container images.

3. Relationship to existing database work

YAZA has two distinct database workstreams:

1. Database connectivity: std.db providers and drivers let applications use databases maintained by other projects. PostgreSQL is the first-party integration priority; SQLite, MySQL/MariaDB, and additional providers follow their own compatibility and conformance gates.
2. YAZA Base: a YAZA-owned relational engine, SQL implementation, storage format, client/server service, shell, backup tooling, and operational model.

The first workstream comes first for API completion. YAZA Base must not block reliable PostgreSQL API delivery. Product CRUD should continue to work with external PostgreSQL while YAZA Base is only a proposal or experimental implementation.

“psql” is PostgreSQL’s interactive command-line client, while PostgreSQL is the database server and engine. YAZA Base would compete as a database product and would include its own SQL shell; it would not be a drop-in replacement for the psql program in the first release.

4. Product goals

• Run a standard relational SQL workload for YAZA API and service applications.
• Provide ACID transactions, crash recovery, constraints, indexes, and safe concurrent access before pursuing advanced query features.
• Offer one supported workflow for local development, automated tests, deployment, schema migrations, and typed YAZA clients.
• Make parameterized query use the natural path and keep SQL values separate from query text.
• Derive YAZA records and API response contracts from an explicit, versioned schema snapshot.
• Protect rolling deployments with schema compatibility checks that account for running application versions and their declared schema fingerprints.
• Keep a compact embedded profile and a client/server profile on a shared SQL and type system, adding each profile only when its storage and concurrency rules pass their own acceptance gates.
• Expose database change events to YAZA Pulse through a documented change-data-capture contract, without requiring applications to poll tables.
• Make backup, restore, diagnostics, and data export available in the CLI instead of requiring private implementation knowledge.

5. Non-goals for the first release

• Claiming superiority over PostgreSQL, MySQL, or SQLite without reproducible benchmark and reliability evidence.
• Supporting every SQL standard clause, vendor dialect, extension, or wire protocol at launch.
• Reimplementing or wrapping another database engine while marketing it as YAZA-owned.
• Building multi-region distributed SQL before single-node transaction correctness, crash recovery, and operational safety are proven.
• Replacing the external database drivers in std.db.
• Automatically switching an application between databases with different transaction, type, locking, or consistency semantics.

6. Proposed technical architecture

6.1 SQL front end

The server and embedded library share the same SQL front end:

1. UTF-8 aware lexer with bounded token and statement sizes.
2. Parser that produces a versioned syntax tree and source-positioned diagnostics.
3. Binder that resolves identifiers, privileges, and parameter types against a schema snapshot.
4. Type checker that rejects incompatible operators and unsafe implicit conversions.
5. Logical relational plan with explicit filters, projections, joins, aggregates, sorting, and limits.
6. Cost-based optimizer that uses table and index statistics and can explain its chosen plan.
7. Vectorized execution operators for scans, filters, projections, joins, and aggregates; a simple iterator execution path may be used first to validate semantics.

The initial SQL profile should cover CREATE TABLE, ALTER TABLE for supported safe changes, DROP TABLE, CREATE INDEX, SELECT, INSERT, UPDATE, DELETE, joins, aggregates, transactions, constraints, and prepared parameters. Unsupported syntax must produce a clear diagnostic rather than being silently accepted or approximated.

6.2 Storage and transactions

The first durable server profile should use:

• A documented, versioned page format with explicit byte order, checksums, and recovery metadata.
• Append-only write-ahead logging (WAL), durable commit ordering, checkpointing, and idempotent crash replay.
• MVCC row versions so readers can use a stable snapshot while transactions update data.
• B+tree primary and secondary indexes with integrity checks and a rebuild command.
• A bounded buffer pool with observable memory limits and pin accounting.
• An asynchronous virtual file system (Async VFS) boundary for page reads, WAL writes, checkpoints, flushes, backup, and recovery I/O. The database and web scheduler must never perform potentially blocking storage operations on the event-loop thread.
• Background vacuum/compaction with cancellation, progress reporting, and safe pause/resume.
• Atomic schema catalog updates and a transactional migration history with checksums. Migration planning checks registered application schema fingerprints and active compatibility leases; it rejects breaking changes that would invalidate a live application version by default. The complete rolling-deployment contract is defined in §6.5.
• Row and transaction limits that are explicit in configuration and diagnostics.

WAL is a reliability mechanism, not a performance promise: commit records must reach stable storage before durable success is returned. Group commit may combine flushes for concurrent commits while preserving this guarantee. PostgreSQL documents WAL as the basis for redo recovery and point-in-time recovery; the proposal adopts that proven design principle rather than inventing a less-tested durability shortcut (PostgreSQL WAL documentation).

The Async VFS owns the boundary between database work and operating-system storage calls. Linux may use io_uring when it is supported and passes the runtime capability checks. kqueue and epoll remain readiness mechanisms for sockets; they do not by themselves make ordinary file reads, writes, or fsync asynchronous. On macOS and on platforms without a safe asynchronous file API, the VFS delegates disk operations to a dedicated, bounded I/O worker pool. The YAZA cooperative scheduler suspends the database task and resumes it from the completion queue. Queue capacity, outstanding bytes, operation deadlines, and overload behavior are bounded; overload applies backpressure or returns a typed error rather than spawning unbounded workers. Durable commit waits for its required flush completion, while unrelated HTTP tasks remain schedulable.

The first transaction profile should document READ COMMITTED and snapshot behavior precisely. Add serializable isolation only after serialization-conflict detection and retry behavior have dedicated correctness tests. A transaction must never silently degrade to a weaker isolation level.

6.3 Server and connection model

• yazad accepts authenticated TCP connections and local Unix-domain sockets where supported.
• The first network protocol is a versioned YAZA Base protocol with TLS required for non-loopback TCP.
• Use bounded asynchronous connection handling with per-connection and per-query budgets, cancellation, backpressure, and fair scheduler integration.
• Prepared statements, bind parameters, streaming rows, and server-side cursors are first-class protocol operations.
• Server-side connection pooling is internal; applications should not need one connection per request.
• Provide a stable C ABI only if required for non-YAZA clients; the first supported client is native YAZA code.
• Publish a protocol specification before promising third-party driver compatibility.

PostgreSQL wire compatibility is a possible later interoperability project, not an initial requirement. It would require protocol conformance evidence and would not make YAZA Base PostgreSQL-compatible in SQL semantics by itself.

6.4 Embedded mode

The embedded profile should reuse the parser, planner, executor, types, and durable file format where practical. It runs in-process without a network server and supports local development, tests, desktop and mobile persistence, and offline-first use cases.

All embedded file access goes through the same Async VFS contract. The embedded database must not call blocking page or WAL I/O directly from a YAZA HTTP handler, scheduler task, or UI thread. The VFS can submit work to its bounded I/O pool and suspend only the task waiting for that operation. This prevents storage waits from monopolizing the cooperative scheduler; it does not make an individual durable commit complete before its required storage flush.

Its first supported concurrency contract should be explicit: many concurrent readers and a single serialized writer per database file, unless a later design proves a stronger contract safe. File locking and crash-safety rules must be tested on each supported filesystem and operating system. The official SQLite guidance distinguishes embedded local storage from client/server workloads and documents the single-writer constraint; YAZA should make similarly clear workload guidance rather than presenting both modes as interchangeable (SQLite appropriate uses).

6.5 YAZA integration

• yaza db codegen generates stable YAZA row shapes and typed query bindings from a pinned schema snapshot.

• Declared YAZA sum types may be persisted as a first-class database type. Each stored type has a stable type identifier and version, and each case has a stable numeric tag. The page encoding uses a tag width selected by the type version (1, 2, or 4 bytes), a null bitmap where needed, and canonical field encodings with specified byte order and alignment rules. The type decoder validates the type ID, version, tag, and payload bounds before constructing the corresponding YAZA variant in its managed arena. It never casts a database page directly to a C struct: persistent pages must remain portable across compiler, ABI, and machine changes. This avoids storing every variant as its display name in a VARCHAR column while preserving safe decoding and schema evolution. Result[T, E] remains a language control-flow type and is not automatically a persistable column; a separately declared serializable variant is required.

• Selected query API direction (project-owner clarification, 2026-10-26): application developers should describe database queries in native YAZA syntax, without authoring SQL strings. The syntax does not need to imitate LINQ or use chained method calls. Use a native, typed query block: developers state the source, conditions, ordering, limit, and selected result fields in YAZA syntax. The compiler parses the block into a typed query tree, resolves fields against the pinned schema snapshot, checks the projection shape, parameter types, and nullability, then lowers the tree into a relational plan for the optimizer and executor. Captured values such as min_price become bound parameters automatically; query text is never assembled by the application. The typed runtime descriptor and PostgreSQL lowering in std/db/query.yz and std/db/postgres.yz are now an initial implementation foundation; the native query-block grammar, schema binding, general predicate tree, and typed result mapping remain unimplemented.

• The native query block is the recommended v1 API. For example, after declaring ProductSummary with id, name, and priceMinor fields:

  shape ProductSummary:
      id: Text
      name: Text
      priceMinor: Number
  
  async fn find_products_above(min_price: Number) -> Result[Rows[ProductSummary], DbError]:
      let results: Rows[ProductSummary] = await db.read Product as product:
          where product.priceMinor > min_price
          order by product.priceMinor ascending
          limit 10
          select ProductSummary:
              id: product.id
              name: product.name
              priceMinor: product.priceMinor
      return results

  This proposed syntax is YAZA code, not a SQL string, and does not copy LINQ's chained-call style. The compiler owns its grammar and translates it to a checked query plan. The example assumes the schema maps products.id to Text and products.price_minor to Number; code generation from the pinned schema snapshot should produce or verify ProductSummary so result fields and database columns cannot silently drift. min_price is captured as a typed bind parameter, never concatenated into query text. ? propagates the declared database error through Result.

• A raw-SQL or tagged-SQL escape hatch is not part of the selected v1 application API. Revisit it only if real YAZA Base use cases show that the native query language cannot express a needed operation. Database-specific operations should first be exposed through typed YAZA APIs so ordinary application code keeps compile-time schema checks and parameter safety.

• yaza db migrate validates ordered migration files, verifies checksums, and applies each migration transactionally.

• yaza db snapshot emits a deterministic, portable schema manifest with explicit type IDs, constraints, indexes, and engine features. It records what the server can preserve and rejects lossy conversions.

• API route contracts may refer to generated shapes, but database storage models and public HTTP response models remain separately versioned.

• YAZA Pulse integration reads committed changes from a WAL-derived stream; it does not poll tables with repeated SELECT statements and does not send each change back through the query planner. As the WAL record is assembled in memory, the engine can stage a reference to that immutable record in a bounded CDC ring buffer before the disk write; the staged record stays pinned while its transaction outcome is pending. Pulse subscribers see the event only after the transaction reaches its configured commit/durability point, so the stream never presents an uncommitted change as durable. After delivery or durable replay checkpointing, the pin can be released. A full or slow queue applies backpressure or pauses publication; durable WAL remains the recovery source, with sequence-based resume tokens and at-least-once delivery. Consumers must be idempotent. This design avoids table polling and can reduce avoidable work and latency, but “near-zero latency” is a benchmark target, not a guarantee independent of commit durability, storage, or subscriber load.

• Schema code generation pins the application to an explicit schema fingerprint and compatibility range. During a rolling deployment, database clients register their application build ID and schema fingerprint with a bounded lease. A migration checks active leases and the configured minimum supported application version before dropping, renaming, or changing the type of a referenced column. Incompatible migrations are rejected by default. Use an expand/migrate/contract rollout: add compatible schema first, deploy code that can work with both versions, backfill, then remove old schema only after old client leases expire and the minimum supported version advances. yaza db migrate --allow-breaking is the explicit override and must print the affected objects and clients; automation must opt in deliberately. Offline applications are covered by the configured compatibility window/minimum version rather than being silently treated as active forever.

• The same native client should serve CLI commands, API services, batch programs, and later mobile applications; no Python or external database runtime is required to install or execute it.

6.6 Security and operations

• Authenticate users with a modern password KDF and support certificate-based service identity later.
• Enforce database, schema, table, and operation privileges; default-deny new users.
• Require verified TLS for remote transport and reject plaintext credentials on non-loopback connections.
• Use prepared statements; enforce statement, memory, result-row, and execution-time budgets.
• Redact credentials and bind values from logs by default; expose query shape and timing separately.
• Support encrypted backups first. Encryption at rest requires a reviewed key lifecycle, rotation, recovery, and memory-clearing design before release.
• Provide checksummed logical and physical backups with restore verification. Point-in-time recovery follows only after WAL archive integrity and restore drills pass.
• Keep an operator-readable audit log for authentication, privilege changes, migrations, backup, restore, and destructive operations.
• Include schema fingerprint, compatibility decisions, active-client leases, and any --allow-breaking override in migration diagnostics and the audit log.

The MVP must not use a custom cryptographic primitive. Cryptographic implementation and provider choices must follow YAZA’s security RFC and receive an independent audit before production claims.

7. CLI proposal

yaza db init ./data
yaza db start --data ./data --listen 127.0.0.1:55432
yaza db shell --url yazabase://localhost:55432/app
yaza db status --url yazabase://localhost:55432/app
yaza db migrate
yaza db snapshot --output schema.yzschema
yaza db codegen --schema schema.yzschema
yaza db backup --output backup.yzbackup
yaza db restore --input backup.yzbackup --target ./restored-data
yaza db inspect --data ./data --checksums --indexes
yaza db explain "SELECT id FROM products WHERE id = ?"

Every command should have --help, stable machine-readable diagnostics, safe defaults, explicit destructive-operation confirmation for interactive use, and a non-interactive --yes flag for automation. Secrets should be read from protected environment/config sources, not command-line arguments that appear in process listings.

8. Delivery phases and gates

Phase 0 — Specification and risk reduction

Define SQL subset, type mappings, isolation behavior, file format, protocol, error codes, security boundary, compatibility policy, and benchmark methodology. Build a parser and execution-plan prototype against in-memory fixtures. No persistent-data compatibility promise yet.

Exit gate: reviewed specs, grammar corpus, deterministic diagnostics, differential SQL semantic fixtures, and a signed-off threat model.

Phase 1 — Durable embedded alpha

Implement tables, primary keys, basic B+tree indexes, single-writer transactions, WAL, crash recovery, backup/restore, and parameterized CRUD in a local database file. Keep the SQL subset deliberately small.

Exit gate: deterministic crash-injection at every WAL boundary, corruption detection, recovery tests, fuzzed parser, bounded Async VFS queue tests proving scheduler fairness during slow disk operations, and repeatable CRUD/migration/backup restore runs on macOS and Linux.

Phase 2 — Single-node server beta

Add yazad, authentication, TLS, bounded asynchronous client sessions, concurrent readers, MVCC, row-level conflict handling, query deadlines, connection reuse, metrics, and administration operations.

Exit gate: concurrency and isolation suite, slow-client and overload tests, privilege tests, TLS verification tests, restart recovery, resource-limit tests, rolling-version migration compatibility tests, and sustained load with recorded p50/p95/p99 latency and memory.

Phase 3 — YAZA API production profile

Add pooled native YAZA client, typed row decoding, schema snapshots/code generation, API integration examples, migrations, and PostgreSQL-to-YAZA-Base migration tooling with validation reports.

Exit gate: the API release gates remain satisfied independently, and YAZA Base passes a production readiness review. This phase starts after API baseline delivery; it does not postpone PostgreSQL API support.

Phase 4 — Replication and cloud operation

Add WAL streaming, read replicas, automated backups, point-in-time recovery, failover tooling, and cloud deployment templates. Define quorum and consistency semantics before multi-writer replication.

Exit gate: documented recovery-point and recovery-time objectives, fault-injection across node/network failures, upgrade rollback, restore drills, and data-consistency verification.

Phase 5 — Broader ecosystem profiles

Add embedded/mobile packaging, offline change synchronization, Pulse CDC integration, search/analytics extensions, and optional object/blob references. Keep blobs in the database only when transactional semantics justify the cost; large media should normally use YAZA object storage with database metadata.

Exit gate: profile-specific size, battery, memory, lifecycle, synchronization-conflict, security, and compatibility evidence.

9. Acceptance and comparison methodology

Compare only equivalent workloads and configurations. Publish source revision, compiler flags, CPU, storage device, operating system, durability settings, SQL schema, dataset, warm/cold cache method, connection count, and benchmark tooling.

Measure:

• point reads and indexed reads;
• inserts and mixed read/write transactions;
• joins and aggregate scans;
• throughput plus p50, p95, and p99 latency;
• CPU, peak and steady-state RSS, disk bytes written, and database growth;
• process startup, crash recovery, backup, and restore time;
• performance with durable commit enabled and with each documented tuning profile.

Use PostgreSQL, MySQL/InnoDB, and SQLite according to the workload they are designed for. SQLite is a distinct embedded use case, so do not compare its single-process embedded latency with a networked server round-trip and present that as a fair general result. MySQL’s InnoDB documentation emphasizes transactions, row locking, MVCC, and crash recovery; those are baseline capabilities YAZA Base must meet before claiming comparable server use (InnoDB overview). PostgreSQL’s MVCC and isolation documentation likewise provides a reference for specified concurrency semantics (PostgreSQL concurrency control).

Reliability gates are release blockers, not benchmark trade-offs: committed data must survive the documented crash model, transaction behavior must match the isolation contract, backups must restore, and malformed SQL or protocol input must remain bounded.

10. Principal risks and mitigations

11. Recommendation

Adopt YAZA Base as the proposal name and reserve yaza db for its user-facing CLI. Keep the name provisional until public release checks. Build the design as a durable single-node relational engine first, with embedded and client/server profiles delivered through separate acceptance gates. Do not start a distributed SQL implementation as the first milestone.

The strongest intended differentiator is not a claim that every subsystem is unprecedented. It is a single tested path from schema to typed YAZA code, API contracts, migrations, query parameters, observability, and later YAZA Pulse change streams—while retaining explicit workload profiles and database-grade durability.

References

• PostgreSQL reliability and write-ahead logging
• PostgreSQL concurrency control
• SQLite: appropriate uses
• MySQL 8.4 InnoDB overview
• MySQL 8.0 InnoDB architecture

← Kembali ke semua dokumentasiVersi dan status rilis →