YAZA Pulse Real-Time and Event Streaming Engine

Status: Accepted design direction — implementation pending (accepted by the project owner on 2026-10-24)
Proposal name: YAZA Pulse
Scope: Typed realtime sessions and a durable, partitioned publish/subscribe event broker for YAZA APIs, services, web applications, and mobile clients

This proposal describes a future YAZA-owned communication engine. It is not an implementation claim. As of this proposal, the runtime has an HTTP/1.1 server and a partial cooperative scheduler, but it does not implement WebSocket, SSE, WebTransport, HTTP/3, or QUIC. See the runtime capability matrix for code-backed status.

1. Goal

YAZA Pulse should make realtime application code and service-to-service event flows use one typed contract across server, browser, and mobile. Developers declare events and authorization rules once; the compiler should validate publish and receive calls and generate reviewed client bindings. The engine should provide connection lifecycle, bounded queues, scheduling, flow control, reconnection, replay, durable event logs, consumer groups, observability, and transport negotiation.

The target is a simpler and more predictable programming model than assembling a hub library, transport adapters, message codecs, reconnect logic, custom buffering, and an independent Kafka-like broker separately. The design aims for low latency and low resource use, but it does not claim to be faster than SignalR, raw WebSocket, Kafka, or other engines until reproducible benchmarks demonstrate that.

SignalR is a higher-level realtime framework with connection management, hubs, groups, and transport fallback; WebSocket and SSE are transport interfaces. Kafka is a durable partitioned event log with replay, independent consumer groups, and replication; simple pub/sub typically means fanout to currently connected subscribers. Pulse should offer the relevant framework and broker capabilities under one typed developer model while preserving these distinct operational profiles and remaining interoperable with established transports.

2. Current position

The current native server handles HTTP/1.1 with a readiness-driven event loop, a cooperative request-reader task per connection, and a bounded worker pool for generated controller handlers and database-backed readiness probes. The pool defaults to four threads and accepts 1–32; only macOS kqueue has been exercised, while Linux epoll remains source-only. PostgreSQL protocol I/O is blocking within workers. YAZA source-language async/await, scheduler-backed business tasks, WebSocket/SSE/WebTransport, durable event delivery, and Pulse itself remain unimplemented.

The first implementation work must therefore establish the language contract and bounded session/scheduler core before promising realtime features. Transport support must be delivered in visible stages. WebSocket and SSE are the practical first network adapters over the existing HTTP/1.1 direction; HTTP/3 and WebTransport follow after a maintained QUIC/TLS provider and target support are available.

3. Programming model

The language should declare the event contract at compile time. The following is illustrative syntax, not currently valid YAZA:

shape RoomMessage:
    room: Text
    text: Text
    sent_at: Number

realtime Chat:
    event Message(RoomMessage)
    event PresenceChanged(room: Text, online: Number)
    command SendMessage(RoomMessage) -> Result[Boolean, ChatError]

    authorize SendMessage with require_member
    join room with require_member

    on SendMessage(message):
        let saved = ChatRepository.save(message)?
        publish Message(saved) to room(message.room)
        return ok(true)

The same event can be published to the transient session layer or appended to a durable topic. A separate broker example:

stream OrderEvents:
    topic "orders.v1"
    partition_by order_id
    retain 7 days or 200 GiB
    delivery at_least_once

    event OrderPlaced(OrderPlaced)
    event OrderCancelled(OrderCancelled)

consumer BillingWorkers from OrderEvents:
    group "billing-v1"
    parallelism 8
    start latest

    on OrderPlaced(order):
        BillingService.charge(order)?
        acknowledge

The exact language syntax and retention units need an RFC. The example illustrates a topic, a stable partition key, bounded retention, a consumer group, and explicit acknowledgement; it does not imply that these features exist today.

The compiler should generate stable event identifiers from the versioned schema, client-side typed subscribe/publish bindings, and transport-neutral API docs. Method and event dispatch should use generated IDs and exhaustively checked variants rather than untrusted method-name strings. Schema changes should be explicit and compatibility-checked.

The server-facing API should separate durable commands from transient events and durable stream records. A command can return a Result; live publish returns accepted, coalesced, backpressured, expired, or disconnected outcomes. Durable append returns an acknowledgement only after the configured persistence policy is met. A successful live publish means the engine accepted the message under its declared delivery policy, not that every recipient has processed it.

4. Layered architecture

YAZA should own the typed contract, session semantics, broker semantics, log format, queue policy, scheduler integration, codecs, diagnostics, and generated bindings. It should use maintained, reviewed platform providers for QUIC and TLS instead of creating new cryptography or transport-security implementations. Consensus and storage formats need public specifications, fault-injection tests, and independent review; an original implementation is not automatically a safer or faster one.

5. Transport strategy

Preferred path: WebTransport over HTTP/3

On supported secure browser and server hosts, use WebTransport for multiple reliable streams, optional datagrams, and isolation between unrelated streams. A control stream can carry session management and acknowledgements; separate reliable streams can carry ordered chat or synchronization data; datagrams can carry lossy state such as cursor position or rapidly refreshed telemetry.

WebTransport APIs are available in current major browsers, but the IETF HTTP/3 binding remains an active Internet-Draft as of 2026-10-24. The adapter must pin and report its protocol version and remain experimental until the specification and the selected provider are stable. It must never silently change delivery semantics when negotiating a different transport.

Compatibility path: WebSocket

WebSocket provides a widely supported bidirectional fallback. It maps Pulse to one reliable ordered byte stream, so independent Pulse lanes share transport ordering and cannot gain WebTransport's per-stream isolation. The adapter must apply Pulse's own bounded queues and explicit backpressure policy because the browser's classic WebSocket API has no built-in backpressure.

One-way path: Server-Sent Events

SSE is a server-to-client stream with browser-managed reconnection and event IDs. It is useful for notifications, dashboards, and server push, but is not bidirectional. Client commands use normal typed HTTP requests or another negotiated upstream channel. The Pulse API must expose that distinction instead of describing SSE as a full-duplex transport.

Long polling may be added only as a compatibility option when a deployment requires it. It should not be the default.

Negotiation

The client and server should negotiate only transports and features explicitly enabled by deployment policy. The connection diagnostics should report the chosen adapter, protocol version, compression, message-size limit, lane capabilities, and downgrade reason. Compression should default off for secrets and small messages; if enabled, its CPU and side-channel implications must be documented and measured.

6. Scheduler, backpressure, and latency design

The engine should use bounded state everywhere: active connections, streams, handler work, per-session bytes, per-tenant bytes, replay storage, pending timers, and broadcast fanout. Hitting a bound must produce a defined result; it must never turn into unbounded memory growth.

The scheduler should provide:

• Readiness-driven I/O and monotonic timers, with a single-owner state model for each connection.
• Fair scheduling across sessions and tenants, so a busy room cannot starve unrelated clients.
• Separate control, interactive, bulk, and lossy-update budgets. Control messages must have a reserved but bounded budget.
• Per-event deadline and overflow behavior: reject, drop-oldest, coalesce-latest, or require acknowledgement. The default should be reject/backpressure for reliable events and coalesce-latest only when the event declaration opts in.
• A producer-visible try_publish result and an awaitable publish operation once source-level async semantics are implemented.
• Cancellation propagated from disconnected sessions and expired deadlines to queued work.
• No hidden unbounded task creation or implicit worker thread per connection.

The first implementation should use the existing cooperative scheduler only after it is integrated with generated HTTP applications and socket readiness. A bounded worker pool may execute blocking database and file work later. The engine should measure queue delay separately from network and handler time so users can see where latency accumulates.

7. Delivery, reconnect, and ordering

Pulse must state delivery guarantees per event:

• Transient: best effort; may be dropped on disconnect or backpressure.
• At-most-once: accepted once by a live session and not replayed.
• At-least-once: replay may deliver duplicates; clients use a stable event ID to deduplicate.

Pulse must not promise exactly-once delivery. Exactly-once effects require application-level idempotency and durable transactional state.

Resumable sessions should use opaque, expiring resume tokens; per-room or per-session monotonic sequence numbers; bounded replay windows; acknowledgement checkpoints; and a snapshot/reset signal when the requested history has expired. Replay storage must have size, age, and tenant limits. A resume token is not an authorization grant: the server must reauthenticate and reauthorize each resumed subscription.

Ordering should be guaranteed only within a declared scope, such as one room or one reliable stream. Global ordering across rooms and processes is not a default guarantee.

8. Durable event broker and pub/sub

Pulse should expose two explicit delivery profiles over the same typed event schema:

Live fanout should be inexpensive and responsive. Durable streams should be independently consumable later, so a disconnected consumer does not make a published event disappear. A stream can fan out to several consumer groups; members of one group share work. This follows the useful distinction found in established systems: Kafka orders records within topic partitions and uses groups for parallel work, NATS JetStream adds persistence and replay to pub/sub, and Redis distinguishes fire-and-forget pub/sub from acknowledged Streams. Pulse should not collapse these into one ambiguous “send” operation.

Broker contract

• Topics and partitions: each durable topic has a schema version, retention policy, partition count, and optional partition key. Records with the same key remain ordered within one partition. There is no implicit global ordering across partitions.
• Offsets: each append receives a stable topic-partition offset. Consumers can start at earliest retained, latest, a specified offset, or a bounded time lookup. An expired offset returns a clear retention-gap result and can require a snapshot/rebuild path.
• Consumer groups: a group tracks progress independently. Consumers in one group split assigned partitions; distinct groups each receive their own view. Partition reassignment must fence stale owners to avoid double commits.
• Acknowledgement and redelivery: a durable group advances its committed offset only after acknowledgement. Unacknowledged records can be redelivered after a bounded visibility deadline. Consumers must tolerate duplicates and use application idempotency keys for side effects.
• Retry and dead-letter handling: retry count, backoff, maximum processing time, and dead-letter routing are declared policies. Poison messages cannot block a partition forever; ordering implications of retrying or skipping are visible to the application.
• Idempotent producer: a producer identity and monotonic sequence can suppress duplicate appends within a documented scope/window. This protects retries after a lost acknowledgement, but does not make arbitrary database or external effects exactly once.
• Retention and quotas: topics support time and byte limits, with an optional count cap. When the disk quota is full, the broker follows a selected policy (backpressure, reject append, or explicitly configured expiry); it never silently discards records advertised as durable.
• Batching: producers and consumers can use bounded batches. Batching can improve disk/network efficiency, but must have explicit maximum bytes and linger deadlines to bound tail latency.

On-disk and failure model

The first durable broker should be a single-node log with an owned, versioned segment format: bounded record lengths, schema/event IDs, key, producer identity/sequence where enabled, append timestamp, payload, and a per-record corruption check. Segment indexes and manifests accelerate lookup but can be rebuilt from valid log records. After a crash, recovery scans to the last complete valid record and truncates only the incomplete tail; corruption in the committed middle of a segment must fail closed and report the affected topic/offset.

The format needs explicit byte order, versioning, upgrade rules, disk quota accounting, free-space behavior, and crash-consistency tests. Checksums detect accidental corruption; they are not authentication or cryptography. Encryption at rest should use reviewed operating-system or storage-provider facilities and a documented key lifecycle, not a YAZA-invented cipher.

The append acknowledgement policy must be explicit: an in-memory acceptance is not durable; a local durable acknowledgement waits for the configured filesystem flush policy; a future quorum acknowledgement waits for the configured replication quorum. Users choose the latency/durability tradeoff per stream. Test power loss at each write/flush/manifest transition before calling a mode durable.

Database consistency

YAZA's PostgreSQL API should later provide a typed transactional outbox integration: commit application state and an outbox record in one database transaction, then relay that record to Pulse Log with an idempotent producer key. This prevents the common failure where a database commit succeeds but an in-memory publish is lost. It does not guarantee an external consumer's side effect exactly once.

Where both output records and consumer offsets live in the same Pulse cluster, a later bounded broker transaction may atomically commit output records and source offsets. For effects outside that transaction boundary, consumers still need idempotency or an outbox/inbox pattern. Pulse must not market universal exactly-once processing.

9. Security and isolation

The design requires TLS at the selected host/provider, strict handshake parsing, exact Origin policy for browser sessions, authentication before joining protected rooms, authorization for each command and subscription, and authorization rechecks after identity or policy changes. CORS configuration is not authentication.

It also requires message, connection, room, fanout, rate, replay, and queue bounds; per-tenant isolation; protocol-version checks; schema validation before handler dispatch; bounded decompression; and safe, non-reflective error responses. Credentials should not be placed in URLs. Access logs must omit message bodies, tokens, and query strings unless an explicitly reviewed redaction policy permits selected fields.

Datagrams are untrusted, lossy input and should be allowed only for event types declared transient. The runtime must not use datagrams for authorization decisions, durable commands, or financial state changes. New codecs and handshake parsers need fuzzing, sanitizer builds, malformed-frame regression tests, and a versioned threat model before production claims.

10. Scale-out and persistence

The first release should be single-process with explicit connection, room, topic, partition, disk, and consumer limits. Later scale-out should define partition ownership, bounded consumer rebalancing, broker membership, leader epochs/fencing, inter-node fanout, replication acknowledgement, disk repair, and split-brain behavior before adding multi-node operation.

The recommended first distributed design is a YAZA-authored implementation of a reviewed consensus protocol such as Raft for broker metadata and per-partition leader epochs, with durable append replication and quorum commit rules specified separately. It must have a simulator/model test suite for partitions, delayed/reordered messages, crashes, restarts, and stale leaders before production use. Realtime fanout and transient session state should not be forced into a database transaction for every message. PostgreSQL remains an application database and outbox source, not the broker's per-message hot path.

Cloudflare Workers and similar edge hosts may eventually run Pulse clients and a stateless session adapter. Durable Pulse Log requires a host with supported persistent storage and coordination; an ephemeral worker runtime alone cannot be treated as a Kafka-class durable broker. Cloudflare-specific durable primitives can be an optional host adapter, with their own limits clearly surfaced, rather than hidden behind a claim of identical storage guarantees.

11. Ecosystem advantages to prove

The intended advantage is end-to-end sharing, not a claim that one transport beats all competitors on every network:

• One compiler-checked event schema shared by API handlers, browser code, and mobile clients.
• No hand-maintained duplicate DTOs or string-based event dispatch.
• Reuse of YAZA shape, variant, Option, Result, diagnostics, API auth hooks, metrics, and scheduler policies.
• Transport capability reporting, one command for local development, and one set of test fixtures across server and clients.
• One typed event contract can publish to a browser/mobile live session and to one or more durable service streams without hand-written DTO or schema-registry duplication.
• A small single-node broker should be useful for local development; production deployments can choose explicit retention, partitioning, consumer groups, and later replication as workload grows.
• YAZA UI components can subscribe to typed events and schedule state updates within frame budgets; mobile clients can coalesce transient updates to limit battery and render work.
• One bounded observability model for queue delay, handler time, publish outcomes, reconnects, replay gaps, and transport errors.

These are design goals. They become advantages only after the runtime, compiler, generated clients, and measurements exist.

12. Implementation stages and exit gates

1. Contract and semantics: specify event schemas, IDs, compatibility rules, error outcomes, authorization hooks, live/durable profiles, and sample compiler diagnostics. Exit when malformed/incompatible schemas are rejected and golden contract tests pass.
2. Bounded in-process live engine: connect the cooperative scheduler to generated application handlers; implement sessions, rooms, cancellation, per-session budgets, and metrics without network transports. Exit with fairness, overflow, cancellation, and soak tests.
3. Single-node durable broker: implement topic/partition append, offsets, consumer groups, acknowledgements, replay, retention, quotas, segment recovery, and idempotent producer sequence handling. Exit with crash-at-every-write-boundary, disk-full, duplicate-ack, replay, slow-consumer, and corruption tests.
4. WebSocket and SSE adapters: add strict handshake and frame handling plus SSE framing/reconnect IDs over HTTP/1.1. Bridge live Pulse topics to SSE and bidirectional sessions to WebSocket. Exit with browser-level compatibility tests, protocol fuzzing, slow-consumer tests, and ASan/UBSan evidence.
5. Generated clients and local tooling: generate browser and native client bindings from one contract; provide yaza pulse dev, inspect, test, and replay commands only after executable support exists. Exit with cross-language golden vectors and end-to-end reconnect/consumer recovery tests.
6. WebTransport/HTTP/3: integrate a reviewed maintained QUIC/TLS provider, pin the HTTP/3 binding revision, implement streams and optional datagrams, and maintain WebSocket fallback. Exit after interoperability and adversarial network tests; retain experimental status until the standard/provider combination is stable.
7. Replication and mobile: build the reviewed consensus/replication path, then add Android first according to RFC 0008 and iOS second. Exit only after quorum-loss, split-brain, recovery, network-change, and mobile resource tests.

Every stage must update the runtime capability matrix. A proposal, syntax example, or generated file is not evidence that the feature is implemented.

13. Benchmark plan

Before claiming “faster,” publish repeatable results for identical application behavior, hardware, payloads, TLS, connection lifetime, and load generator. Compare Pulse Live with SignalR on WebSocket, raw WebSocket, and SSE for its one-way use case; compare WebTransport only when both stacks use equivalent protocol capabilities. Compare Pulse Log separately with Kafka, NATS JetStream, and Redis Streams using the same persistence, replication, retention, producer-ack, and consumer-ack guarantees. Record:

• Throughput and p50/p95/p99 end-to-end latency, including queue delay.
• CPU per delivered message, memory per active connection, total RSS, and maximum bounded fanout.
• Bytes on wire per event, reconnect and replay tail latency, dropped/coalesced/rejected messages, and fairness under noisy-neighbor load.
• Cold start, binary and client bundle size, TLS handshake cost, and mobile energy/render impact.
• Behavior with delayed readers, packet loss, network change, server restart, replay-window exhaustion, and saturated queues.
• Broker append/read throughput, local and quorum acknowledgement latency, disk bytes per record, consumer lag, group rebalance time, partition skew, replay throughput, recovery time, and behavior under disk-full and replica loss.

Use at least five independent runs, publish raw measurements and scripts, disclose versions/configuration, and report variance. Do not compare YAZA on WebTransport against a competitor on WebSocket as if transport were held constant. Performance acceptance thresholds should be set from the first reliable baseline rather than invented before measurement.

14. Open decisions

This proposal recommends the following defaults for the first RFC:

1. Keep WebSocket and SSE as the first live adapters because the current runtime direction is HTTP/1.1; design the transport-neutral contract so WebTransport can be added later.
2. Treat WebTransport as the preferred future high-capability adapter when supported, but version-gate it while the IETF HTTP/3 binding remains a draft.
3. Offer transient pub/sub and durable partitioned logs as distinct profiles. Make bounded replay and at-least-once semantics opt-in; keep live messaging as the lightweight default.
4. Ship JSON interoperability first. Design a compact YAZA binary codec as a separately versioned optional profile, with conformance vectors and independent review before enabling it by default.
5. Build the first broker as a single-node, disk-backed system before a replicated cluster; do not promise multi-node fault tolerance until quorum behavior and recovery are tested.
6. Use a typed PostgreSQL outbox adapter for application state changes that must be published reliably, rather than pretending a broker can share a transaction with an external database.

References

• ASP.NET Core SignalR overview — hub model, connection management, groups, supported transports, and fallback.
• Apache Kafka introduction — topics, partition ordering, consumer groups, and replication concepts.
• NATS JetStream — durable streams, consumer acknowledgements, and replay.
• Redis Pub/Sub delivery semantics — at-most-once transient fanout.
• Redis Streams — append-only logs, consumer groups, pending messages, replay, and acknowledgement patterns.
• WebSocket API — browser bidirectional API and its lack of built-in backpressure.
• WebTransport API — current browser API availability, streams, datagrams, and secure-context requirement.
• WebTransport over HTTP/3 IETF draft tracker — draft status and revisions as of 2026-10-24.
• WHATWG Server-Sent Events — one-way event stream, reconnection behavior, and event IDs.
• RFC 6455: The WebSocket Protocol.
• RFC 9297: HTTP Datagrams and the Capsule Protocol.

← Kembali ke semua dokumentasiVersi dan status rilis →