Remote Service Protocol

This document specifies version 1 of the Graft remote service protocol. It is an interoperability contract between Graft clients and remote services; it does not prescribe a programming language, hosting platform, database, or object store.

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, and MAY are to be interpreted as described by BCP 14.

Repository URLs

The canonical, Git-style repository URL is:

https://host/<namespace>/<repository>

Clients MUST also accept the explicit transport form:

graft+https://host/<namespace>/<repository>

graft+https selects this protocol but uses HTTPS on the wire. Clients MAY accept graft+http for local development or trusted networks. Production services SHOULD use HTTPS.

The repository path is the protocol base URL. It is opaque to the client and MAY contain more than two segments when a service needs a different tenancy model. User-facing URLs SHOULD keep the familiar <namespace>/<repository> shape and MUST NOT expose an internal /api/... route or protocol version.

The optional token_env query parameter is client-local configuration:

https://host/acme/archive?token_env=GRAFT_ARCHIVE_TOKEN

The client MUST remove token_env before making requests. Tokens MUST NOT be placed directly in a repository URL. Version 1 repository URLs MUST NOT contain userinfo, fragments, or any other query parameter.

To derive the HTTP base URL, a client removes the graft+ prefix, removes the client-local query, and removes one trailing slash. For example:

graft+https://host/acme/archive?token_env=GRAFT_ARCHIVE_TOKEN
-> https://host/acme/archive

Protocol Version

Every protocol request MUST contain:

Graft-Protocol: 1

Every response, including an error response, MUST echo the supported version in the same header. A service that cannot serve the requested version SHOULD return 426 Upgrade Required and advertise its version in Graft-Protocol. The version belongs in negotiation metadata, not in the repository URL.

Authentication

A service MAY require authorization. Version 1 clients support bearer tokens:

Authorization: Bearer <token>

The default client environment variable is GRAFT_REMOTE_TOKEN; token_env selects another variable. A missing or invalid credential SHOULD return 401 with an appropriate WWW-Authenticate header. Authorization policy, token issuance, tenancy, and ACL storage are service concerns outside this protocol.

Repository Descriptor

GET {base} returns a descriptor for diagnostics and capability discovery:

{
  "protocol": "graft-remote",
  "version": 1,
  "capabilities": [
    "range",
    "list",
    "put-if-absent",
    "upload-bundle",
    "receive-pack",
    "receive-bundle",
    "multipart-object",
    "cas",
    "cad"
  ],
  "limits": {
    "max_request_bytes": 67108864,
    "multipart_part_bytes": 16777216
  }
}

The descriptor MAY include additional fields. Clients MUST ignore unknown fields.

Object Keys

Operations address opaque, repository-relative object keys. Known version 1 keys include:

HEAD
refs/heads/<branch>
objects/<fanout>/<object-id>
objects/pack/<pack-id>.pack
objects/pack/<pack-id>.idx
store/files/<fanout>/<object-id>
logs/<log-id>/commits/<lsn>
segments/<segment-id>

HEAD and refs/** are transactional metadata. Content below objects/**, store/**, logs/**, and segments/** is immutable after creation. locks/** is reserved and MUST NOT be exposed as repository data.

Each key segment is UTF-8 and is percent-encoded independently. / separates segments and MUST NOT be percent-encoded as data. Empty segments, ., .., backslashes, NUL, control characters, and invalid percent encodings MUST be rejected. A service MAY impose documented key and body limits and return 413 or 414 when they are exceeded.

Operations

All paths below are relative to {base}.

Request	Purpose	Success
HEAD /raw/<key>	Test existence and obtain size metadata.	200 OK
GET /raw/<key>	Read raw bytes.	200 OK
PUT /raw/<key>	Replace transactional metadata.	204 No Content
DELETE /raw/<key>	Delete transactional metadata.	204 No Content
PUT /raw-if-not-exists/<key>	Create an object only when absent.	204 No Content
POST /upload-bundle/<ref-key>	Stream a ref snapshot and its immutable storage.	200 OK
POST /receive-pack/<ref-key>	Publish one object pack and atomically update a ref.	204 No Content
POST /receive-bundle/<ref-key>	Publish immutable objects, a pack, and a ref.	204 No Content
POST /multipart-start/<key>	Start or resume a multipart immutable upload.	200 OK
PUT /multipart-part/<key>	Upload one numbered part.	204 No Content
POST /multipart-complete/<key>	Assemble all uploaded parts at the immutable key.	204 No Content
DELETE /multipart-abort/<key>	Abort an incomplete multipart upload.	204 No Content
POST /cas/<key>	Atomically compare and replace metadata.	204 No Content
POST /cad/<key>	Atomically compare and delete metadata.	204 No Content
GET /list?prefix=<prefix>	Recursively list matching keys.	200 OK

Request bodies for PUT and POST /cas are the new raw bytes and SHOULD use application/octet-stream. Successful mutation responses have no body. A successful HEAD response MUST include the byte length that a corresponding full GET would return in Content-Length.

HEAD /raw/<key> remains mandatory for version 1 services. For compatibility with a legacy gateway that explicitly returns 405 Method Not Allowed or 501 Not Implemented, a client MAY retry the existence probe with GET /raw/<key> and Range: bytes=0-0. 200 or 206 proves that the object exists, 404 means it is absent, and 416 proves an existing empty object only when it also includes Content-Range: bytes */0. The client MUST release a successful probe response without consuming the full object when a gateway ignores Range, and MUST NOT use this fallback for authentication failures, other status codes, or transport errors. The fallback response itself MUST still carry a valid Graft-Protocol header; the initial 405 or 501 might be generated by a gateway before the protocol service and therefore need not carry one.

Services MUST support raw-if-not-exists for immutable objects. A service SHOULD reject unconditional overwrite or deletion of immutable objects. Graft publishes data in this order: immutable objects first, pack index after its pack, and the ref CAS last. A failed ref update can therefore leave unreachable objects but cannot expose an incomplete commit.

Deleting an already absent key through DELETE /raw SHOULD succeed so the operation remains idempotent.

Upload Bundle

upload-bundle is an optional version 1 capability for Git-style clone transport. A client selects a branch and sends one authenticated request:

POST /upload-bundle/refs/heads/main
Content-Length: 0

The service reads the requested ref, lists the repository’s immutable keys, then reads the ref again. If it changed, the service returns 409; the client may retry. A stable snapshot returns Content-Type: application/vnd.graft.upload-bundle and the x-graft-bundle-manifest-bytes header. The response starts with that many bytes of UTF-8 JSON:

{
  "version": 1,
  "reference": {
    "path": "refs/heads/main",
    "value_hex": "<lowercase hexadecimal ref bytes>"
  },
  "objects": 3
}

Exactly objects binary frames follow. Each frame is a 4-byte unsigned path length, an 8-byte unsigned object length, the UTF-8 path, then the object body. Integers use network byte order. Paths MUST be strictly ordered, unique, immutable repository keys. The body MUST end immediately after the final frame. The service streams each immutable backend object without buffering the whole bundle.

Version 1 bundles all immutable keys because the protocol service treats their contents as opaque. The client validates every frame, creates a temporary local remote, and resolves the selected commit graph and checkout from that local snapshot. This gives clone one bulk data response; future protocol versions may add reachability negotiation. Clients MUST fall back to the version 1 raw/list operations when upload-bundle returns 404 or 405.

Receive Pack

receive-pack is an optional version 1 capability that collapses pack upload, index upload, and the final ref CAS into one authenticated request. The request is:

POST /receive-pack/refs/heads/main
Content-Length: <pack-bytes + index-bytes>
x-graft-pack-id: <64 lowercase hexadecimal characters>
x-graft-pack-bytes: <decimal byte length>
x-graft-index-bytes: <decimal byte length>
x-graft-ref-replacement-hex: <lowercase hexadecimal ref bytes>
x-graft-expected-present: true
x-graft-expected-hex: <lowercase hexadecimal expected ref bytes>

The body is the exact pack bytes immediately followed by the exact index bytes. Content-Length MUST equal the sum of the two declared lengths. The service MUST stream and create the pack at objects/pack/<pack-id>.pack, then stream and create the index at objects/pack/<pack-id>.idx, and only then perform the ref CAS. Existing immutable pack or index objects are treated as an idempotent success. A malformed or truncated body MUST NOT update the ref.

The expected-value rules are identical to /cas. A mismatch returns 409 and MAY leave unreachable immutable objects. Services advertise support with the receive-pack descriptor capability. Clients MUST preserve the individual raw-if-not-exists plus /cas sequence as a fallback for services that return 404 or 405 for this optional operation.

Receive Bundle

receive-bundle is an optional version 1 capability that extends receive-pack with immutable objects that the pack references, such as SQLite segments, SQLite storage commits, and external file payloads. It uses all receive-pack headers plus:

x-graft-bundle-manifest-bytes: <decimal manifest byte length>

The body contains the UTF-8 JSON manifest, each declared object in manifest order, the pack, and the index. A manifest has this shape:

{
  "version": 1,
  "objects": [
    { "path": "segments/example", "bytes": 4096, "allow_existing": true },
    {
      "path": "logs/example/commits/0000000000000001",
      "bytes": 128,
      "allow_existing": false
    }
  ]
}

Every path MUST be an immutable repository key and MUST occur at most once. Content-Length MUST equal the manifest length, all object lengths, pack length, and index length. The service MUST create objects in manifest order, then the pack, then the index, and perform the ref CAS last. A malformed, truncated, or trailing body MUST NOT update the ref.

allow_existing: true treats an existing object as an idempotent success. false returns 412 so the client can use the individual v1 path to read and verify the collision before retrying publication. Clients MUST fall back to individual immutable writes followed by receive-pack or /cas when receive-bundle returns 404 or 405.

Multipart Immutable Objects

multipart-object is an optional version 1 capability for immutable objects that exceed an HTTP gateway’s request-body limit. It changes only the transfer: the completed object remains at the original key and retains the same logical content identifier.

The descriptor MUST include limits.multipart_part_bytes when this capability is advertised. A service SHOULD also publish limits.max_request_bytes so a client can skip an aggregate receive-bundle or receive-pack request that cannot reach the application. A client starts or resumes an upload with:

POST /multipart-start/segments/<segment-id>
Content-Length: 0
x-graft-object-bytes: <total decimal bytes>

The response identifies the durable session and the parts already stored:

{
  "upload_id": "opaque-upload-id",
  "total_bytes": 111249535,
  "part_bytes": 16777216,
  "uploaded_parts": [{ "part_number": 1, "bytes": 16777216 }]
}

Clients MUST send all missing parts in ascending order. Part numbers start at one. Every non-final part has exactly the advertised part size; the final part contains the remainder.

PUT /multipart-part/segments/<segment-id>
Content-Length: <part bytes>
x-graft-upload-id: <opaque-upload-id>
x-graft-part-number: <positive decimal number>

Re-uploading a part number replaces that part in the same session, which makes a lost response safe to retry. Repeating multipart-start for the same key and length returns the same session and completed-part list. After all parts are present, POST /multipart-complete/<key> with x-graft-upload-id and an empty body atomically exposes the complete immutable object. DELETE /multipart-abort/<key> with the same header releases an incomplete session.

The start and complete operations return 412 when the immutable target already exists. Clients apply the same collision policy used by raw-if-not-exists. Multipart completion never publishes a ref; the client still performs receive-pack or CAS only after all immutable objects exist.

Compare Operations

POST /cas and POST /cad carry the expected value in two headers:

x-graft-expected-present: true
x-graft-expected-hex: 6162630a

• x-graft-expected-present: false means that the key is expected to be absent; x-graft-expected-hex MUST then be present and empty.
• x-graft-expected-present: true means that the current bytes MUST exactly equal the lowercase hexadecimal value. An empty hexadecimal value represents a present, zero-length object and differs from absence.
• /cas replaces the matched value with the request body.
• /cad deletes the matched value and has no request body.

The comparison and mutation MUST be a single atomic operation with respect to all clients of the repository. CAS and CAD are REQUIRED for HEAD and refs/**; a service MAY reject them for immutable keys.

Range Reads

GET /raw/<key> MUST support one standard byte range:

Range: bytes=1024-2047

A valid partial response returns 206 Partial Content, Content-Range, Content-Length, and Accept-Ranges: bytes. An invalid, multiple, or unsatisfiable range returns 416 Range Not Satisfiable. When the object exists, that response MUST include Content-Range: bytes */<size>. Services SHOULD stream large request and response bodies instead of buffering them.

Listing

prefix is encoded as one query component, so a prefix such as refs/heads/ is sent as refs%2Fheads%2F. A response page is JSON:

{
  "paths": ["refs/heads/feature/search", "refs/heads/main"],
  "next_cursor": "opaque-service-value"
}

Version 1 listing is recursive. A service MAY paginate a response to keep its work and memory bounded. When next_cursor is present, the client MUST request the next page by sending it unchanged as one encoded query component:

GET /list?prefix=refs%2Fheads%2F&cursor=opaque-service-value

The cursor is opaque to clients. A client MUST NOT parse or construct one. A service MUST omit next_cursor only after it has returned the final page. The optional positive limit query parameter is a page-size hint; a service MAY clamp it to a smaller documented maximum or reject an out-of-range value with 400. A service that returns the entire result in one page remains conforming, and a response without next_cursor remains compatible with early clients.

Across a complete traversal, the service MUST return every matching key exactly once, using decoded repository-relative paths. Each page and the concatenated traversal MUST be sorted by ascending bytewise lexical order. Cursors MUST make forward progress and SHOULD expire only after a documented interval.

Status Codes

Status	Meaning
200	Complete read, descriptor, HEAD, or list response.
204	Successful mutation.
206	Successful byte-range read.
400	Malformed path, query, header, or request.
401 / 403	Missing credentials or insufficient authorization.
404	Repository or object is missing.
405	The operation is not allowed for that key.
409	CAS or CAD expected bytes did not match.
412	raw-if-not-exists found an existing object.
413 / 414	Service body or key limit exceeded.
416	Invalid or unsatisfiable byte range.
423	Optional transient lock contention in a locking implementation.
426	Unsupported Graft-Protocol version.
429	Service rate limit; clients may retry with backoff.
500 / 503	Server failure or unavailable service configuration.

409 and 412 are intentionally different: clients interpret 409 as a concurrent ref change and 412 as a create-only collision.

Errors SHOULD use application/problem+json. Clients MUST rely on the status code and MUST NOT require a particular error-body schema.

Consistency and Durability

A conforming service MUST provide:

• durable read-after-write storage for successful mutations;
• create-only atomicity for raw-if-not-exists;
• linearizable CAS and CAD for each transactional key;
• repository isolation, so one repository cannot read or mutate another;
• byte-preserving reads and writes;
• in the absence of concurrent mutations, a complete list traversal in which every write matching the prefix that completed before the first page is visible.

The service MAY store transactional metadata and immutable bytes in different systems. These requirements describe observable behavior, not implementation.

Compatibility

Early documentation used URLs such as:

graft+https://host/api/graft/v1/repos/acme/archive

Services MAY retain that path as a compatibility alias, but new configuration SHOULD use https://host/acme/archive (or the explicit graft+https form). Clients MUST treat the configured repository base as opaque and MUST NOT insert the legacy /api/graft/v1/repos prefix themselves.