Segment Store Garbage Collection

Oak's segment store uses generational garbage collection to reclaim disk space. Content writes create immutable segments, but edits and deletes leave "garbage" (unreachable segments). GC compacts live data into a new generation and deletes old TAR files.

This is the "cost" of the append-only DAG architecture. Understanding it is important for operating Oak Chain at scale.

Status

Segment GC and binary GC run in every validator by default, and neither is paid. A bounded live run on one host (two enterprise clusters of three validators each, mock mode) passed: each member of the compacting cluster reclaimed 3 to 4 MB of deleted content and its store shrank to about 120 kB; three deleted binaries were released and deleted on every member while a kept binary stayed; and the other cluster read the remaining content through its mount on every attempt, with no missing read, no 503 and no 500. Not yet run: several hosts, chain-backed mode, or sustained load.

How Oak Chain Runs GC

GC is not paid and not voted on

Writes and deletes are the paid operations: in chain-backed mode each is settled on the payment contract. Garbage collection is ordinary maintenance. Each validator runs Oak's own revision GC on its own segment store, on a schedule, with no payment, no vote and no HTTP trigger.

Validators of one cluster hold the same content but not the same bytes: segment ids are random and segment boundaries depend on each member's flushes. GC does not change content, so members do not need to agree on it. Each member compacts its own store.

Two kinds of GC run:

• Segment GC reclaims the segment store: records left behind by edits, deletes and older revisions.
• Binary GC reclaims IPFS space for binaries that no content references any more. Unpinning a binary affects every member, so this part goes through the replicated log.

Segment GC

What Runs

Oak's own tailGC() and fullGC(): estimation, compaction, then cleanup of segments older than the retained generations. A tail compaction rewrites only what changed since the last compaction; a full compaction rewrites all live content. Oak's guards stay on: a run is skipped when estimation finds too little growth, when free memory is low, or too soon after the last successful run.

When It Runs

• Runs happen in wall-clock slots on a dedicated thread, one at a time. With the defaults, every slot is a daily tail compaction and every seventh is a weekly full compaction instead.
• Members are staggered: member i of n runs at an offset of i × interval / n. Three members run at 00:00, 08:00 and 16:00 UTC, so at most one member compacts at a time and the cluster keeps its quorum.
• Slots are anchored to the clock, not to start-up, so restarts keep the stagger. The first run is the first slot at least 15 minutes after start-up, and no run starts before genesis is committed.

Writers Are Not Stalled

Replicated writes apply on Aeron's service thread. Oak can finish a compaction by blocking all writers for up to its force timeout (60 seconds by default). The validator sets the force timeout to 0, which Oak treats as "never force": a compaction that cannot catch up with concurrent writes gives up, cleanup still runs, and the next slot tries again. Writes keep landing during compaction.

Remote Readers

Another enterprise's cluster can mount this cluster's namespaces and read them over HTTP. Its view is a head of this cluster's store. Every compaction starts a new generation, and cleanup removes segments older than the retained generations, so a mounted head stays readable for about the reader window: the shorter compaction interval × (retained generations − 1), one day with the defaults.

• GET /v1/federation publishes the window in its gc block (retainedGenerations, tailIntervalMs, fullIntervalMs, readerWindowMs).
• Mounts refresh their view on access, far more often than the window.
• A read that still reaches a removed segment answers 503 mount_stale with Retry-After: 1 and refreshes the mount, so the retry reads current content. An owner that cannot be reached answers 503 mount_unavailable and counts against the endpoint's circuit breaker, so the mount fails over. Neither case answers 500 or empty content.

Generational Model

How Generations Work

1. Generation N is the current (active) generation
2. Generation N-1 is retained as a safety buffer
3. Generation N-2 is deleted during cleanup

Why Retain 2 Generations?

Readers may hold references to segments in the previous generation. Keeping 2 generations means a reader whose view is less than about one compaction interval old can still read it. A reader with an older view can hit a "segment not found" error, which is why remote mounts refresh their view and answer 503 mount_stale instead (see Remote Readers).

This is a trade-off:

• More retained generations = more disk space
• Fewer retained generations = risk of read failures

Oak retains 2 generations by default; the cleanup log reports it (with 2 retained generations).

TAR File Cleanup

The Cleanup Process

1. Scan each TAR file's segment index
2. Check if segments are referenced by current generation
3. Mark empty TAR files for deletion
4. Rewrite partial TAR files (optional, to reclaim more space)
5. Reaper thread deletes marked files

TAR File Naming

data00000a.tar  ← Generation 1
data00001a.tar  ← Generation 1
data00000b.tar  ← Generation 2
data00001b.tar  ← Generation 2

The letter suffix indicates the generation. During cleanup, entire generations of TAR files are removed.

Binary GC

Binaries live in IPFS, outside the segment store. Binary GC releases binaries that no content references and removes them from IPFS. It is on by default.

1. Stamp. Every replicated write that stores or references a validator-hosted binary records the log position and cluster time on its entry in the replicated binary index (/:oak-chain-binaries).
2. Mark (leader). Once per interval the leader notes its applied log position, collects the binary references of its segment store (every generation it still keeps), and proposes a BINARY_RELEASE for index entries that nothing references and that were last referenced longer ago than the grace period.
3. Release (every member). Applying the release removes each listed entry that was last referenced before the mark; an entry referenced since is skipped. Every member reaches the same result from the same log.
4. Delete (every member). After a short delay each validator checks the index again and deletes its own copy of each released binary. With IPFS Cluster, the cluster pin is removed only when ipfs.cluster.unpin-on-delete is set, which is safe only when the IPFS Cluster serves this enterprise cluster alone.

A deleted binary's bytes are released only after segment GC has dropped every generation that still references it: typically two full compactions. Kubo reclaims the blocks with its own GC (ipfs daemon --enable-gc or Datastore.GCPeriod). A CID an author supplied without the validator storing it is never released.

Settings

Monitoring GC

• GET /v1/gc/status returns {"segment": …, "binary": …}; the same object is the gc block of /health.
• segment: the settings, this member's index and offset, the head's generation, Oak's current phase, and the last tail and full run (outcome, reclaimed bytes, store size before and after) and the next ones.
• binary: the settings, the next mark, the last mark (references, candidates, proposed), the last release (released, skipped) and delete counts.
• Oak records every cleanup in gc.log in the store directory. Representative log lines: TarMK GC #5: cleanup completed, Segment GC full completed: reclaimed … bytes, Binary release marked at …: released …, skipped ….

The Economics of GC

• Authors pay a flat fee per write and per delete in chain-backed mode (simulated in mock mode).
• Validators carry storage and GC as operating costs. GC has no fee, no debt and no vote.
• Pricing is under review, in particular for large binaries, whose cost grows with size.

The prices are contract constants (see Smart Contract); validators do not set them.