r34 Hugging Face WAL catch-up and batching

Consolidated-release note: this document records an earlier r34 correction. Its retained technical behavior is included in the full release. Use the current native upgrade sequence for deployment and schema-10 recovery rules; do not reinstall a historical patch.

The native controller's durable WAL receiver batches contiguous write-ahead log bytes while retaining synchronous durable acknowledgement and single-writer ownership checks. Restored checkpoints can require receiver catch-up before the application is admitted.

Receiver behavior

The receiver collects contiguous WAL bytes in bounded batches, up to 4 MiB, with a 100-millisecond collection limit. An idle input stream still flushes when that short collection deadline arrives, so a lone small transaction does not depend on another message. Complete-record verification and remote publication occur for the batch rather than each individual network message. Remote verification and publication time is additional; the collection window is not a promise of a completed durable write within 100 milliseconds.

The safety conditions are unchanged: receiver feedback acknowledges only the complete-record boundary confirmed by durable publication. Buffered bytes, the primary's advertised WAL end and an uploaded but unpublished object do not authorize feedback. Gaps, overlaps, source identity or timeline mismatches, lost ownership and storage failures still prevent a successful acknowledgement. Cancellation does not turn unacknowledged buffered bytes into durable progress.

Controller startup still requires one receiver in streaming and sync state whose flush position covers the confirmed durable acknowledgement. A connected receiver in catchup or potential state does not pass readiness.

Bounded startup setting

PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS controls the separate receiver startup wait. Its default is 600 seconds, and accepted values are integers from 30 through 3600. On HF, this is an optional controller Space variable; leaving it unset uses the default. It does not change PF_HF_PG_START_TIMEOUT_SECONDS, the PostgreSQL process startup budget, or a simulation's execution limit.

The receiver wait remains bounded. Increasing this limit gives healthy catch-up more time; it does not bypass failed publication, permit asynchronous operation or waive the readiness checks. Invalid values are rejected.

Existing deployment

Follow section 11 of the native deployment guide for an already running service. It covers pausing, creating and verifying a private HF Bucket copy, and replacing code while preserving existing stable keys, Bucket configuration and state. This maintenance needs no new S3 configuration, database recreation or first-time bootstrap. Compare the current generated runtime context with the retained qualified one; rebuild and qualify it when the context or image provenance differs. Pin the resulting immutable digest.

Use a fresh extraction of the full release; its receiver implementation is already included. Generate new private jobs and controller space contexts together. Upload the private payload, record its exact commit and new content hash, qualify those identities, then update the controller pins and upload the matching controller context. A previously generated private-code hash must not be reused after these source changes.

After restarting, wait for receiver catch-up and normal application admission. Check restored accounts and projects, a small real simulation, durable writes and the existing cancellation and authorization acceptance checks. If startup still fails, inspect the same attempt before repeatedly restarting it or replacing configuration.

Acceptance scope

Software fixtures cover batching limits, idle flushes, contiguous ordering, durable feedback, failure/cancellation handling and startup setting validation. Their commands and results remain bound to their recorded source identities. Mock storage latency and request-count tests do not measure live HF Bucket throughput or establish PostgreSQL recovery. Consult the current release record for newly executed checks; complete deployment recovery and live acceptance separately against the exact deployed code.