Running a Validator

Run and observe an Oak Chain validator cluster. Current evidence covers local and test environments; there is no permissionless public validator network or guaranteed fee-sharing policy.

Who This Is For

• Ecosystem players wanting to earn from validator economics
• Infrastructure operators with experience running distributed systems
• Organizations wanting to host their own content provenance layer

Deployment strategy: local cluster validation -> a future verified testnet settlement flow. Mainnet remains unavailable.

Examples below assume the current local process workflow from oak-chain-infra. If you run validators under Kubernetes or another service manager, translate the stop/start/log commands to that runtime.

Requirements

Hardware

Software

• JDK 17 or 21 (JDK 22+ is outside current build policy)
• IPFS (Kubo): the validator does not start without a reachable IPFS API
• Docker (optional)
• Access to Ethereum RPC (Infura, Alchemy, or own node), for Sepolia mode only

Quick Start (Build from Source)

Docker images are not yet published. Build from source:

# Clone the repository
git clone https://github.com/somarc/jackrabbit-oak.git
cd jackrabbit-oak

# Build the validator JAR
mvn -B -ntp -pl oak-segment-consensus -am package -DskipTests

# Start IPFS in another terminal first: ipfs init (once), then ipfs daemon

# Run one validator (a one-member cluster that elects itself leader)
java --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED \
  -Dconsensus.enabled=true \
  -Dhttp.bind.host=127.0.0.1 \
  -Daeron.dir="$PWD/run/aeron" \
  -jar oak-segment-consensus/target/oak-segment-consensus.jar \
  --port 8090 --store "$PWD/run/validator-0"

Without -Dconsensus.enabled=true, Aeron does not start and writes return 503. Logs go to run/logs/validator.log.

Current public local automation lives in the infrastructure repo. Use modes/mock/validators/lifecycle/start-cluster.sh for mock mode or modes/sepolia/validators/lifecycle/start-cluster.sh for Sepolia instead of archived Docker Compose flows. The Quickstart walks through a 3-validator mock cluster.

Configuration

The validator JAR has no OSGi runtime. Configure it with command-line options, -D system properties and a few environment variables. A running validator reports its effective settings at GET /v1/config/osgi.

3x2 Validation Rig

For local 3x2 or other multi-cluster proofs, keep cluster identity, HTTP ports, Aeron ports, runtime roots, and MediaDriver directories separate. This is a local validation rig, not production density guidance.

The knobs below are environment variables of the oak-chain-infra scripts (start-cluster.sh, start-3x2.sh). The scripts turn them into validator options such as --port, --store and -Daeron.cluster.basePort; the validator does not read them.

Cross-cluster mounts are lazy, read-only views outside Aeron. The local Aeron cluster owns the writable repository for its shard.

Environment Variables

Environment variables the validator itself reads:

Cluster Membership

Cluster membership is set with -D system properties on each validator:

--port 8090                          # HTTP port; port + 1 is used by the Oak standby server
--store /var/oak-chain/validator-0   # segment store directory
-Dconsensus.enabled=true             # required to accept writes
-Daeron.cluster.nodeId=0             # this validator's index in aeron.cluster.hostnames
-Daeron.cluster.hostnames=h0,h1,h2   # one host per member; the count sets the cluster size on a fresh start
-Daeron.cluster.basePort=9000        # Aeron uses UDP ports basePort + 100 x nodeId + 1 to 7
-Daeron.dir=/var/oak-chain/aeron     # media driver directory prefix; give each validator its own
-Dconsensus.self.url=http://h0:8090  # this validator's HTTP URL
-Dconsensus.peers=http://h1:8090,http://h2:8090   # the other validators' HTTP URLs

Every member must be configured with the same set of URLs; genesis records it. The JVM also needs --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED.

Example: 3-Node Cluster

Only a local three-validator cluster on one host has been tested. The multi-host layout below shows the settings; it has not been validated.

Node 0 (validator-0.example.com):

java --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED \
  -Dconsensus.enabled=true \
  -Daeron.cluster.nodeId=0 \
  -Daeron.cluster.hostnames=validator-0.example.com,validator-1.example.com,validator-2.example.com \
  -Daeron.cluster.basePort=9000 \
  -Daeron.dir=/var/oak-chain/aeron \
  -Dconsensus.self.url=http://validator-0.example.com:8090 \
  -Dconsensus.peers=http://validator-1.example.com:8090,http://validator-2.example.com:8090 \
  -jar oak-segment-consensus.jar \
  --port 8090 --store /var/oak-chain/validator-0

Node 1 (validator-1.example.com):

# Same hostnames and basePort as above, with:
-Daeron.cluster.nodeId=1
-Dconsensus.self.url=http://validator-1.example.com:8090
-Dconsensus.peers=http://validator-0.example.com:8090,http://validator-2.example.com:8090

Node 2 (validator-2.example.com):

# Same hostnames and basePort as above, with:
-Daeron.cluster.nodeId=2
-Dconsensus.self.url=http://validator-2.example.com:8090
-Dconsensus.peers=http://validator-0.example.com:8090,http://validator-1.example.com:8090

Modes

Sepolia Mode

For experiments with chain-backed payments on the Sepolia testnet. Chain-backed payments are not yet validated (see the validator README, Maturity).

OAK_BLOCKCHAIN_MODE=sepolia
OAK_BLOCKCHAIN_RPC_URL=https://sepolia.infura.io/v3/your-infura-key

Mainnet Mode

Unavailable: the validator refuses mainnet in this version. With OAK_BLOCKCHAIN_MODE=mainnet it stops at startup. No production V5 contract, public validator topology, or operational SLO is recorded.

Reality check: Use Sepolia to validate payment and contract behavior. That validation has not been done yet.

Ports

Open the Aeron UDP ports between the validators of one cluster. For multi-cluster proofs, repeat the same pattern with separated base ports and runtime roots so the clusters do not share writable Aeron state.

Storage

Data Directory

Layout created by the oak-chain-infra mock scripts:

$CLUSTER_RUNTIME_ROOT/        # default ~/oak-chain
├── validator-0/              # node 0 segment store (--store) and its Aeron media directory
├── validator-1/
├── validator-2/
├── logs/                     # validator-N.log, ipfs-daemon.log
└── pids/

Backup

# Stop locally managed validators first
cd /path/to/oak-chain-infra/modes/mock/validators/lifecycle
./stop-cluster.sh

# Back up the validator stores
tar -czf backup-$(date +%Y%m%d).tar.gz "${CLUSTER_RUNTIME_ROOT:-$HOME/oak-chain}"/validator-*

# Restart
./start-cluster.sh

Restart and restore from a backup are not yet validated.

Monitoring

Health Check

curl -s http://localhost:8090/health \
  | jq '{status, clusterHealthy, reachableCount, totalMembers, quorumSize, currentRole}'

{
  "status": "UP",
  "clusterHealthy": true,
  "reachableCount": 3,
  "totalMembers": 3,
  "quorumSize": 2,
  "currentRole": "LEADER"
}

An unhealthy cluster returns HTTP 503 with "status": "UNHEALTHY" and an unhealthyReason. The full response also carries success, timestamp, store, blobStoreType, blobStoreActive and sharding. /health/local, /health/cluster and /health/deep give narrower views. Health responses are observations; they do not prove that validators have converged.

Prometheus Metrics

Metrics available at http://localhost:8090/metrics (Oak metrics plus standard JVM metrics):

# Consensus
oak_consensus_is_leader                           # 1 = leader, 0 = follower
oak_consensus_leader_epoch                        # increments on leader change
oak_consensus_time_since_last_heartbeat_seconds
oak_validators_reachable

# Storage
oak_segments_stored_total                         # TAR segment-store files on this validator
oak_segments_disk_usage_bytes                     # logical bytes in TAR segment files

# Runtime
oak_active_connections
oak_mediadriver_has_crashed
oak_mediadriver_crash_count
oak_mediadriver_force_bootstrap

# IPFS write policy
oak_api_ipfs_policy_rejections_total
oak_api_ipfs_enterprise_cid_accepted_total

With OAK_VALIDATOR_AUTH_TOKEN set, /metrics requires the token.

For queue, verifier and release signals, see Write Pipeline.

Grafana Dashboard

Published Grafana dashboards have not yet been re-homed into oak-chain-infra. For now, use the script-based monitoring surfaces in:

• modes/mock/validators/monitoring/
• modes/sepolia/validators/monitoring/

See the infrastructure repo for the current operational layout.

Troubleshooting

Validator Won't Start

# Check the validator log file (default root: ~/oak-chain)
tail -f "${CLUSTER_RUNTIME_ROOT:-$HOME/oak-chain}/logs/validator-0.log"

# Common issues:
# - Port already in use
# - IPFS API not reachable
# - Two validators sharing one aeron.dir

Can't Connect to Cluster

# Check a peer's HTTP API (validators also reach each other over HTTP)
curl -s http://validator-1.example.com:8090/health

# Aeron traffic is UDP: allow basePort + 100 x nodeId + 1 to 7 between validators
# (9001-9007, 9101-9107, 9201-9207 with the default base port 9000)

Leader Election Stuck

# Check all nodes are running
# Verify aeron.cluster.hostnames, aeron.cluster.basePort and the
#   consensus.self.url / consensus.peers set are identical on all nodes
# Check for network partitions

Economics

ValidatorPaymentV5 defines one flat settlement price for writes and one for deletes. It has no Standard, Express, or Priority products and no tier-based finality promise. Prices and gas figures are on Smart Contract.

Settled fees accrue to one configured cluster treasury. The contract does not split fees among validators or guarantee operator revenue; any treasury distribution is an off-chain operator policy.

Cluster Treasury

One ValidatorPaymentV5 instance is deployed per Aeron cluster. The constructor binds it to a single clusterTreasury address that accumulates all settled fees. withdrawClusterFees() sends the accumulated balance to the treasury and can be called only by the treasury address itself.

Accumulated fees leave the contract only through that withdrawal. recoverExcessETH is limited to the balance above the accumulated fees, so it cannot drain cluster fees. The contract owner can rotate the treasury with setClusterTreasury(address); the next withdrawal goes to the new address.

Operator Costs

• Hardware: see Requirements above, per node.
• Storage growth: the Oak segment store, plus IPFS for binaries.
• Network: 100 Mbps or more for replication.
• Ethereum RPC: JSON-RPC access for contract event watching (Sepolia mode).

For when verified writes are released to consensus, see Write Pipeline: Release Model.

Next Steps

• Architecture - How the system works
• Consensus Model - Aeron Raft details
• Write Pipeline - Stages, queue signals, release model and state labels
• Segment GC - Consensus-coordinated garbage collection
• Smart Contract - Flat settlement price and treasury contract
• API Reference - Full public API surface (including Consensus & Proposals endpoints)

Operator Reports

• Validator Load Test Report (2026-02-04)
• Validator Load Test Report (2026-02-07): Overnight Finalization Gap