Storage model

The bucket is the durability boundary. Everything on your disk — the visible notes directory, the private Git repository, the pack cache — can be thrown away and rebuilt from the bucket. Nothing on disk is authoritative, and the local caches never decide accepted state.

What one notebook looks like in the bucket

One configured prefix holds exactly one notebook:

<prefix>/current
<prefix>/packs/checkpoints/<through-generation>-<checkpoint-id>.pack
<prefix>/packs/increments/<generation>-<publication-id>.pack

There is no bare repository and no .git directory in the bucket, and the server never uses a bucket listing to decide which state is current. The prefix itself is empty or a slash-separated relative key prefix with no leading or trailing slash, no empty segment, no backslash, and no . or .. segment; an invalid prefix is refused by configuration.

Pack keys carry their target or through-generation and a unique publication or checkpoint ID. Both IDs are UUIDv7 values in the canonical lowercase 8-4-4-4-12 text form, validated for version 7 and the RFC 4122 variant. Ordering is defined by the generation fields, never by UUID ordering. One writer owns each key, and accepted pack bytes are never changed.

Every pack upload also writes slivingdoc-sha256, slivingdoc-size, slivingdoc-kind (increment or checkpoint), and slivingdoc-generation as S3 user metadata. That metadata helps diagnose and resume uploads, but the manifest descriptor is what is authoritative. A small pack can go up in one request; a large checkpoint pack can use multipart upload. A retry first inspects the object already at that unique key and accepts the existing bytes only when the recorded SHA-256 and size both match.

The current manifest

current is the only authoritative state index. It names one active checkpoint and every accepted increment after it:

{
  "version": 1,
  "generation": 8121,
  "head": "<git-object-id>",
  "checkpoint": {
    "id": "<checkpoint-id>",
    "publication": "<publication-id-of-head>",
    "throughGeneration": 8119,
    "head": "<git-object-id>",
    "key": "packs/checkpoints/8119-<checkpoint-id>.pack",
    "sha256": "<pack-sha256>",
    "size": 123456
  },
  "increments": [
    {
      "generation": 8120,
      "publication": "<unique-publication-id>",
      "parent": "<git-object-id>",
      "head": "<git-object-id>",
      "key": "packs/increments/8120-<publication-id>.pack",
      "sha256": "<pack-sha256>",
      "size": 4096
    }
  ],
  "retained": []
}

A version 1 manifest carries a required integer version of 1, one monotonic unsigned 64-bit generation, the accepted head object ID, one active checkpoint descriptor, an ordered incremental tail, a checksum and size for every pack, and the retained generations described below.

Generations are strict. An absent current is the implicit remote-empty state at generation 0; the first successful conditional creation writes generation 1; every successful replacement, including a checkpoint replacement, increments the observed generation by exactly one. Generation values never reset, and an overflow rejects the operation.

The manifest is read strictly as well. An unknown field, a duplicate field name, a missing required field, and an explicit JSON null are all rejected at every object level, with duplicate names detected before any value is decoded. A future incompatible schema uses a different version, and a version 1 reader rejects it before it touches a single pack.

Validation runs before any pack is accessed, and it checks the whole chain, not just the shape: the checkpoint cutoff is not beyond generation, active increment generations increase and are above the cutoff, the first increment’s parent equals the checkpoint head, each later increment’s parent equals the preceding head, the top-level head equals the last increment head (or the checkpoint head when the tail is empty), each key matches its namespace pattern and contains no backslash, empty segment, . or .. segment, and every object key and checkpoint ID is unique across the manifest. Each retained generation validates as an independent chain under the same rules.

Conditional replacement

The first manifest write uses If-None-Match: *. Every later write uses the exact ETag returned by the preceding read, in If-Match. HTTP 412 maps to the semantic precondition error — expected contention, to be merged and retried. A timeout or connection error after the request bytes were sent is an ambiguous result, not a precondition failure, and is resolved by reading current and looking for the proposal’s publication ID.

Note: An S3 ETag is only a concurrency token for current. It is not a content checksum, and multipart ETags are not reliable digests. Content is protected by the SHA-256 in the manifest descriptor.

This is the whole of the concurrency control. There is no LOCK.lock object, no lease, no renewal loop, no stale-lock recovery, and no clock-based expiry. See A lost race for the writer’s view of it.

Immutable packs and pack integrity

A pack is an immutable Git pack file. An increment holds the objects of one accepted publication and may depend only on the indexed packs before it. A checkpoint is closed: it reconstructs a complete notebook state without older packs or external base objects, and uses no pack delta whose base object exists only in an older pack. Importing a checkpoint pack alone into an empty repository must make its commit and complete file tree readable.

V1 repositories use Git’s SHA-1 object format, so every object ID is exactly 40 lowercase hexadecimal characters; no other algorithm or representation is accepted. On top of that, each manifest descriptor carries a SHA-256 checksum — exactly 64 lowercase hexadecimal characters — and the byte size of the complete pack. Both are validated before a downloaded pack is imported, and a mismatch is a storage-integrity error: the server refuses the import and does not advance or materialise corrupt state.

The downloaded pack-byte cache is keyed by that SHA-256, never by an untrusted S3 key, and a cache hit still has to match the expected size and pass a fresh SHA-256 check. That is what makes the optional shared pack cache safe to share between agents: no entry is trusted for its location or its writer.

Checkpoints and stable-prefix compaction

After the active tail reaches --checkpoint-packs (256 by default) increments, one checkpoint effort is scheduled for that observed manifest. Retained tails do not count toward the threshold. The worker takes the oldest increments up to the threshold as its stable prefix, and the generation of the last selected increment becomes the cutoff.

Compaction does not block normal writers. When the checkpoint pack is ready, the worker reads the latest manifest, and if that manifest still contains its compacted prefix, replaces only that prefix and keeps every later increment. The proposal generation is the latest generation plus one; the new checkpoint’s through-generation is the cutoff, and its head and publication ID are those of the final compacted increment. The final manifest replacement is the ordinary ETag compare-and-swap, so a lost race costs one more small manifest rewrite, never another checkpoint build.

The worker may reuse its pack only while the latest active descriptor chain still contains that exact prefix. A normal commit preserves it; a competing checkpoint at or beyond the same cutoff removes it, and then the proposal is discarded. Competing checkpoint workers are therefore safe: at most one manifest replacement wins, and the loser’s pack is simply an unreferenced object. A checkpoint failure never changes the result of an already accepted commit.

Retention and cleanup

The manifest keeps the active checkpoint generation and, by default, one previous generation; --retained-checkpoints configures the count, where 0 retains only the active generation. Each retained entry holds its checkpoint descriptor, its complete ordered increment tail through the cutoff that replaced it, and its accepted head — enough to reconstruct the exact state the newer checkpoint replaced, so a stale reader can restart.

Physical objects are removed on a best-effort basis, only after a later checkpoint’s manifest write has succeeded:

C0 becomes C1  -> retain C0 and C1 storage
C1 becomes C2  -> retain C1 and C2, then clean C0 storage

The active and retained manifest data are the garbage-collection roots. Cleanup lists only the packs/checkpoints/ and packs/increments/ namespaces, parses the generation out of each valid protocol key, ignores malformed keys, and considers only keys at or before the cutoff of the checkpoint that triggered it. Before each delete batch it rereads and validates current and rebuilds the live key set; it sends at most 1,000 keys per delete request, follows every listing continuation token, and records a per-key delete error to retry on a later cleanup. The current key is never a deletion candidate.

Cleanup failure does not fail a commit or a checkpoint. If checkpoints do not succeed, cleanup does not run and old proposals simply remain stored. An orphan proposal after the current cutoff is left alone until a later cutoff makes its generation eligible. And a reader never depends on a download timeout for safety: a stale reader that finds a removed pack rereads current and restarts.

What is history and what is not

slivingdoc promises current-state durability. It does not promise historical recovery.

• A checkpoint preserves the exact file state at one accepted head. It omits older commit ancestors, records that head as a shallow boundary, and validation permits only that declared gap.
• Commit messages are retained in recent internal Git data only.
• There are no branch, tag, ref, revision, checkout, or rollback APIs, and no public Git remote.

Warning: Bucket versioning, replication, object lock, lifecycle rules, and external backups are deployment recovery policies. They complement slivingdoc and are not prerequisites of the synchronisation algorithm — choose them for your own recovery requirements. See Guarantees and limits.

Next

• S3 requirements — the permissions and the conditional-write guarantees a store must provide.
• Guarantees and limits — the failure table and what remains the operator’s job.
• Configuration — --checkpoint-packs, --retained-checkpoints, and the rest.