Go toolkit and CLI for reconciliation

RCDS

Based on the Allerton 2019 RCDS paper, this Go project explores content-dependent shingling, set reconciliation, and exact file sync workflows for distributed data.

Quick Start Read Paper

3 algorithms RCDS, IBLT, and full sync

2 modes Set reconciliation and file pull

Verified Unit, integration, e2e, and vet

Quick start

Run a real sync locally.

Start a one-shot server with a local set, then connect a client with a different set. Both sides finish with the same reconciled values and byte counters.

make build
./bin/rcds server --algorithm full \
  --items server-only,shared \
  --output server.out

./bin/rcds client --algorithm full \
  --items client-only,shared \
  --output client.out

Capabilities

Built for research code that needs to behave like software.

Set reconciliation

Use `rcds`, `iblt`, or `full` from the same CLI and GenSync-compatible API.

Chunked file pull

Reuses chunks the client already has and verifies the final SHA-256 checksum.

TCP transport

Length-prefixed payloads, bounded reads, full writes, and one-shot server workflows.

Tested paths

Unit, integration, and e2e tests exercise the CLI, algorithms, and file workflow.

How it flows

Metadata first. Payloads only when needed.

RCDS_GO separates the reconciliation decision from payload movement. The set algorithms discover differences; the file path exchanges manifests and missing chunks before writing the verified result.

local data

digest

diff

transfer

verify

Documentation

Pick the entry point that matches your job.

CLI Reference Commands, flags, set examples, and file examples. Paper Background How the Allerton 2019 RCDS paper maps to this Go implementation. Architecture Package layout, algorithm adapters, protocol flow, and design notes. Deployment Local binaries, containers, Kubernetes manifests, and production cautions.

Research lineage

Based on the RCDS paper from Allerton 2019.

Song and Trachtenberg's paper reduces string reconciliation to set reconciliation by recursively partitioning strings into content-dependent shingles. This repository makes that lineage explicit while separating research-grade RCDS metadata from the production readiness of the current CLI and transport.