Write Pipeline

A write does not appear in the Oak segment store at once. It moves through seven stages: ingress, the unverified queue, verification, the adaptive packing buffer, release, the backpressure gate, and Aeron commit. Each stage has its own queue, limits and counters. Operators tuning throughput need the stage boundaries; anyone diagnosing a latency spike needs to know which stage owns which counter.

The consensus commit step itself is documented in Consensus Model; this page covers everything around it. Walk the pipeline once, then watch GET /v1/proposals/queue/stats while you submit test traffic.

Glossary: release-flow is the model (verified work is released by capacity); "epoch" survives only in field names such as currentEpoch and proposalsByEpoch.

End-to-End Pipeline

Stage-by-Stage

Field names below are keys of GET /v1/proposals/queue/stats on the validator, unless marked otherwise.

1. Ingress and Signature

• Client submits POST /v1/propose-write or POST /v1/propose-delete to the leader. A follower answers 307 with the leader's URL.
• The leader applies request guards and checks the signature. Mock mode checks the signature format only; Sepolia mode also recovers the signer.
• Accepted requests get 202 with "state": "PENDING". Follow one operation with GET /v1/ops/operations/{id}.
• Fields: totalProposals, writeProposals, deleteProposals.

2. Unverified Queue

• Accepted proposals wait here for the verifier.
• Fields: unverifiedQueueSize (also mempoolSize), pendingCount (also mempoolPendingCount), mempoolAvgAgeMs, mempoolOldestMs.

3. Verification and Payment

• Verifier agents check the payment proof, signature and authorization, then persist the result. In Sepolia mode the payment proof is the ProposalSettledV5 event for the request's ethereumTxHash; mock mode simulates payment.
• A proposal with no proof yet, or an unconfirmed one, is re-queued. A proposal past its confirmation timeout is rejected.
• Fields: verifierAttemptCount, verifierSuccessCount, verifierRequeueNoProofCount, verifierRequeueUnconfirmedCount, verifierRejectedCount, verifierErrorCount, verifierQueueWaitAvgMs, verifierQueueWaitMaxMs, verifierAvgTotalMs, verifierLastTotalMs, verifiedCount, totalVerifiedCount, rejectedCount, totalRejectedCount.

4. Adaptive Packing Buffer

• Verified proposals move into the adaptive packing buffer, grouped by wallet for storage locality.
• Fields: verifiedPackingBufferCount, adaptiveVerifiedPackingBufferCount, adaptivePackingWalletCount, adaptivePackingQueuedProposalCountTotal, adaptivePackingDrainedProposalCountTotal, adaptivePackingBufferStats.

5. Release

• The release finalizer drains ready work from the buffer into bounded batches, as the release governor allows (see Release Model). Under pressure, batches go to a separate overflow buffer.
• Chunk size comes from oak.proposal.finalization.chunk.size.
• Fields: releaseReadyProposalCount, releaseReadyBatchCount and batchQueueSize (the same queue), adaptivePackingCreatedBatchCountTotal, backpressureOverflowProposalCount, backpressureOverflowBatchCount, adaptiveReleaseGovernorState, adaptiveReleaseAction, adaptiveReleaseReasonCodes.

6. Backpressure Gate

• The sender takes up to oak.proposal.batch.max batches per cycle and caps in-flight messages at oak.consensus.max.pending.messages.
• If acknowledgements do not arrive within oak.consensus.backpressure.timeout.ms, the sender re-queues the batch.
• This protects validator stability under heavy write load.
• Fields: backpressureActive, backpressurePendingCount, backpressurePendingRawCount, backpressureMaxPending, backpressureTimeoutCount, backpressureReconciliationCount, backpressurePendingOldestMs, backpressurePendingStalledMs.

7. Aeron Commit and Apply

• Batches are offered through Aeron ingress and ordered in the replicated log. Every validator applies them to its own store.
• A write is COMMITTED once a majority of validators have applied it and flushed it to disk.
• Fields: totalFinalizedCount, processedCount, totalProposalsSent, persistencePendingChanges, persistenceFlushAvgMs, persistenceFlushLastMs.

The total*Count fields cover the current counter window (counterWindowStartMs, counterRotationIntervalMs); the *Lifetime variants cover the process lifetime. Queue stats are per validator, and operation records live only on the validator that accepted the write.

Release Model

After payment and signature checks pass, Oak does not send every write straight into Aeron. Verified proposals enter the adaptive packing buffer, where the runtime:

• groups nearby writes
• organizes them for storage locality
• forms bounded micro-batches
• releases those batches according to live system capacity

Release timing depends on verification status, queue pressure, and Aeron health. The payment contract has no tiers and does not set it.

The release governor reads the verified-to-finalized gap, backpressure pending count and age, and the buffer and release-ready counts. It reports a state (HEALTHY, PRESSURED or OVERLOADED), an action (DIRECT, BUFFERED or THROTTLED) and reason codes.

Use these runtime surfaces to inspect live behavior:

• GET /v1/proposals/release-flow (release-flow.v1)
• GET /v1/blockchain/config

Ethereum confirmation windows (requiredConfirmations) still matter for payment verification in Sepolia mode. They do not set write latency.

Edge Overview Fields

The local edge adapter from oak-chain-edge-worker serves GET http://127.0.0.1:8787/ops/v1/overview. It exists only when start-cluster.sh found that repository and started the adapter. Its fields are derived from validator data:

• queue.pending / queue.queuePending: the larger of pendingCount and batchQueueSize.
• queue.backpressurePending: backpressurePendingCount.
• queue.oldestPendingAgeMs: the validator does not emit this field, so it reads 0. Use mempoolOldestMs and backpressurePendingOldestMs instead.
• durability.pendingAcks: persistencePendingChanges, the proposal-queue changes not yet flushed to disk.
• durability.ackTimeouts: maxRetryCount, the highest retry count seen on one proposal.
• replication.maxLagMs: the queried validator's replicationLag. That value counts Aeron log positions, so the Ms suffix is misleading. replication.maxLagNodeId is not emitted by the validator.

Interpretation:

• High pendingCount or batchQueueSize with low backpressurePendingCount usually indicates release or finalization limits.
• High backpressurePendingCount means ingress is outpacing the send and acknowledge loop.
• If verifier wait and total times are low but the queue grows, the verifier is not the bottleneck. If verifier wait grows sharply, intake is outrunning verifier scheduling.
• persistencePendingChanges rising then falling under burst load is normal. Rising and flat means persistence is stalled.

Operating States

Use these five labels on dashboards and in runbooks. A dashboard derives them from the fields above; the validator does not emit them.

• IDLE: queue near zero, no backpressure debt, nothing pending persistence.
• FILLING: net queue slope positive; ingress > finalize.
• DRAINING: net queue slope negative; finalize > ingress.
• CONSTRAINED: queue high and flat or rising while backpressure stays elevated.
• SETTLED: queue drained and totalVerifiedCount == totalFinalizedCount.

Core Derived Metrics

Recommended derived metrics (computed in a dashboard or worker):

• Finalize Rate (ops/s): delta of totalFinalizedCount over window
• Ingest Rate (ops/s): delta of accepted proposals over window
• Net Queue Slope (ops/s): ingest_rate - finalize_rate
• Queue Pressure: label from slope and absolute queue size
• Gap Ratio (%): (totalVerifiedCount - totalFinalizedCount) / max(totalVerifiedCount,1) * 100

Notes:

• Gap Ratio can be high at startup or over short windows; always pair it with slope and queue size.
• Avoid single-point judgments; require at least a 2-5 minute trend.

Fast Triage Commands

# Cluster overview (needs the local edge adapter on :8787)
curl -s http://127.0.0.1:8787/ops/v1/overview | jq '.data | {
  leader, queue, durability, replication
}'

# Leader queue stats (use the leader's port)
curl -s http://127.0.0.1:8090/v1/proposals/queue/stats | jq '{
  totalProposals,totalVerifiedCount,totalFinalizedCount,
  verifiedCount,processedCount,batchQueueSize,
  backpressurePendingCount,backpressurePendingRawCount,
  persistencePendingChanges,persistenceFlushAvgMs,persistenceFlushLastMs,
  verifierQueueWaitAvgMs,verifierQueueWaitMaxMs,
  adaptiveReleaseGovernorState,adaptiveReleaseAction
}'

# Release flow
curl -s http://127.0.0.1:8090/v1/proposals/release-flow | jq

Practical Operator Checklist

Before load tests:

1. Confirm the queue starts drained (pendingCount=0, batchQueueSize=0, backpressureActive=false).
2. Confirm queue tuning at startup: the QUEUE_TUNING_SOURCE log line lists the values in use.

During load:

1. Watch backpressurePendingRawCount, backpressureTimeoutCount and re-queue log lines.
2. Compare verifiedCount with processedCount and totalFinalizedCount.
3. Watch adaptiveReleaseGovernorState and its reason codes.

After load, require all of:

1. batchQueueSize = 0
2. backpressurePendingCount = 0
3. persistencePendingChanges = 0
4. verifiedCount = 0
5. totalVerifiedCount == totalFinalizedCount

If all pass, the run has settled. Capture a queue stats snapshot as run evidence. See the 2026-02-07 report for a run that did not settle.