core.encoding

regression-only

Upstream defects block public API paths. Regression controls record the failing shapes and any working subset.

Layer 4.7 — Data encoding and serialization

Pure-Verum encoders and decoders for every wire format production code typically reaches for. The module covers textual encodings (JSON, Base64/32/58, hex, PEM), binary wire formats (CBOR, MessagePack, DER, varint), and the interoperability-critical "canonical" / "pointer" sub-formats layered on top.

Submodules

Submodule	Purpose	Reference
encoding.json	JSON parsing into JsonValue and serialization	RFC 8259
encoding.jcs	JSON Canonicalization Scheme (signing-deterministic)	RFC 8785
encoding.json_pointer	Path syntax for JSON sub-value lookup	RFC 6901
encoding.cbor	Concise Binary Object Representation	RFC 8949
encoding.msgpack	MessagePack binary encoder/decoder	spec.md
encoding.base64	Base64 (+ URL-safe variant)	RFC 4648 §4 / §5
encoding.base32	Base32 (case-insensitive, trailing-bits validation)	RFC 4648 §6
encoding.base58	Bitcoin-style Base58 + Base58Check	Satoshi
encoding.hex	Lowercase hex encoder + decoder	RFC 4648 §8
encoding.pem	PEM textual envelope for DER blobs	RFC 7468
encoding.varint	SQLite-style 1–9 byte varints	sqlite.org
encoding.der	ASN.1 Distinguished Encoding Rules	X.690

Two sibling varint encodings live in the stdlib next to the wire formats that need them and are not duplicated here: LEB128 lives at core.protobuf.wire (Protocol Buffers style, little-endian) and QUIC varint lives at core.net.http3.frame (1/2/4/8-byte length-prefixed, RFC 9000 §16).

json

mount core.encoding.json.*;

public type JsonValue is
      JsonNull
    | JsonBool(Bool)
    | JsonInt(Int)
    | JsonFloat(Float)
    | JsonString(Text)
    | JsonArray(List<JsonValue>)
    | JsonObject(Map<Text, JsonValue>);

public type JsonMap is Map<Text, JsonValue>;

public fn parse(source: &Text) -> Result<JsonValue, JsonError>;
public fn parse_strict(source: &Text, max_bytes: Int) -> Result<JsonValue, JsonError>;
public fn parse_value(source: &Text) -> Result<JsonValue, JsonError>;
public fn decode_bytes(source: &[Byte]) -> Result<JsonValue, JsonError>;
public fn stringify(value: &JsonValue) -> Text;
public fn stringify_pretty(value: &JsonValue) -> Text;

parse reads a complete document and rejects trailing content, comments, trailing commas and single-quoted strings. Repeated object keys retain the last value. decode is an alias for parse; encode is an alias for stringify.

parse_strict adds a caller-supplied byte limit and rejects repeated decoded keys with DuplicateKey, including spellings such as "name" and "\u006eame" in the same object. It applies that rule to nested objects; the same key in separate objects is allowed. The limit counts the entire UTF-8 source, including whitespace and escapes, before constructing JSON strings or containers. An oversized document returns DocumentTooLarge; a negative limit returns InvalidLimit. A zero limit admits no nonempty document.

Both full-document parsers retain the parser's nesting, string and collection limits. parse_value reads one value and allows trailing content, but returns neither the remaining input nor an offset. It is not an incremental event parser.

Parsing builds an owned value tree. Strings use Text storage; arrays and objects use List and Map storage. The parser has no zero-allocation guarantee. Its result is always JsonValue: a record annotation or @derive(Deserialize) does not change the parser's return type. See the JSON cookbook for explicit field validation.

decode_bytes constructs text with Text.from_utf8_lossy before calling parse. When invalid UTF-8 must be rejected, validate the bytes with Text.from_utf8 and handle that error before calling parse_strict.

stringify builds compact output in a Text; stringify_pretty uses fixed two-space indentation. Neither accepts a writer sink or formatting options. Object members follow map iteration order, without canonical key sorting. Check floating-point values for finiteness before serialization; the serializer returns Text, without a validation-error result.

JsonError exposes four public diagnostic fields: kind: JsonErrorKind, line: Int, column: Int and message: Text. Callers can construct and update diagnostics; the record alone does not establish that a parser produced it. Errors returned by the parser use one-based positions, with columns counted in bytes. Parser errors describe JSON syntax and limits; application field validation needs its own error policy. The public signatures and parser behavior are defined in core/encoding/json.vr.

base64

mount core.encoding.base64.*;

public fn encode(input: &[Byte]) -> Text;              // RFC 4648 §4
public fn encode_url(input: &[Byte]) -> Text;          // §5 URL-safe alphabet
public fn decode(input: &Text) -> Result<List<Byte>, Base64Error>;
public fn decode_url(input: &Text) -> Result<List<Byte>, Base64Error>;

Default encoder emits = padding. URL-safe variant uses -_ instead of +/ and omits padding (web-friendly).

Canonicality

Both decoders enforce the RFC 4648 §3.5 trailing-bits rule: unused low bits of the final character must be zero. Both also reject non-canonical padding shapes — multiple distinct inputs decoding to the same bytes is a malleability surface for JWT / capability-token equality checks, so the decoders treat non-canonical padding as Base64Error.InvalidPadding:

n % 4	Standard decode	URL-safe decode_url
0	accepted (full block)	accepted
1	InvalidLength	InvalidLength (dangling tail)
2	requires == (AABB==); rejects AA	accepts AA (no pad) or AA==; rejects AA= (mid-shape)
3	requires = (AAA=); rejects AAA	accepts AAA (no pad) or AAA=; rejects AAA==

URL-safe input that mixes shapes (e.g. AA, AA=, AA== all decoding to the same byte) is now rejected with InvalidPadding whenever any padding is present — pre-fix, the URL-safe decoder accepted all three.

hex

mount core.encoding.hex.*;

public fn encode(input: &[Byte]) -> Text;                 // lowercase
public fn decode(input: &Text) -> Result<List<Byte>, HexError>;

Case-insensitive decoder, lowercase encoder (matches RFC 4648 §8).

varint — SQLite-style

mount core.encoding.varint.*;

public fn sqlite_encoded_len(value: Int64) -> Int;
public fn sqlite_encode_into(out: &mut List<Byte>, value: Int64) -> Int;
public fn sqlite_encode(value: Int64) -> List<Byte>;
public fn sqlite_decode(buf: &[Byte], start: Int) -> Result<(Int64, Int), VarintError>;
public fn sqlite_decode_first(buf: &[Byte]) -> Result<(Int64, Int), VarintError>;
public fn sqlite_skip(buf: &[Byte], start: Int) -> Result<Int, VarintError>;

Big-endian 1–9 byte form used throughout the SQLite file format (see sqlite.org/fileformat2.html under "Varint"). Bytes 1–8 carry 7 data bits + continuation bit in the MSB; byte 9 (if present) holds the remaining 8 bits with no continuation. The full signed i64 range is covered — negative values use the 9-byte form because their top bit is always set, unlike LEB128 which zig-zags first.

Errors (VarintErrorKind):

Kind	When
Truncated	Input buffer ends mid-varint or before byte 9
Overflow	Reserved for future extensions; no i64 input triggers it

jcs — JSON Canonicalization Scheme (RFC 8785)

mount core.encoding.jcs.{canonicalize_value, canonicalize_str};

public fn canonicalize_value(v: &JsonValue) -> Result<Text, JcsError>;
public fn canonicalize_str(input: &Text) -> Result<Text, JcsError>;

The deterministic JSON serialisation signing workflows require — byte-identical output regardless of map-key insertion order so signer and verifier produce the same input.

Rule	§	Effect
Keys sorted by UTF-16 code unit	3.2.3	{"b":1,"a":2} → {"a":2,"b":1}
No whitespace	3.2.4	compact form
Integer-valued floats collapse	3.2.2.1	1.0 → 1
Minimal string escapes	3.2.1	only ", \, U+0000..U+001F

Used by JWS-on-JSON, W3C Verifiable Credentials, DIDs, Matrix room events, signed build manifests.

let canonical = jcs.canonicalize_str(&raw_json)?;
let digest = sha256(canonical.as_bytes());

Astral-plane characters (U+10000+) are decoded into UTF-16 surrogate pairs before comparison so the sort order matches the JSON.stringify(sorted) form used by other JCS implementations. NaN / ±Inf values return UnsupportedValue — JSON cannot represent them.

json_pointer — JSON Pointer (RFC 6901)

mount core.encoding.json_pointer.{
    JsonPointer, parse, format_json_pointer, resolve,
};

public fn parse(s: &Text) -> Result<JsonPointer, JsonPointerError>;
public fn format_json_pointer(p: &JsonPointer) -> Text;
public fn resolve(p: &JsonPointer, doc: &JsonValue) -> Maybe<JsonValue>;

Path syntax for referencing a specific sub-value inside a JSON document. The building block behind JSON Patch (RFC 6902), OpenAPI $ref, JSON Schema traversal, and CRDT operational transforms.

Token	Example	Meaning
""	root	whole document
"/foo"	document["foo"]	property by name
"/foo/0"	document["foo"][0]	array index
"/a~1b"	document["a/b"]	/ escaped
"/m~0n"	document["m~n"]	~ escaped

Builder API for programmatic construction:

let p = JsonPointer.root()
    .push(Text.from("users"))
    .push(Text.from("0"))
    .push(Text.from("name"));
let s = json_pointer.format_json_pointer(&p);   // "/users/0/name"

cbor — Concise Binary Object Representation (RFC 8949)

mount core.encoding.cbor.{CborValue, encode, decode, encode_canonical};

public fn encode(v: &CborValue) -> List<Byte>;
public fn decode(bytes: &[Byte]) -> Result<CborValue, CborError>;
public fn encode_canonical(v: &CborValue) -> List<Byte>;   // §4.2

Compact self-describing binary format used by COSE (RFC 9052), CWT (RFC 8392), WebAuthn attestation, IoT / CoAP payloads, and Verum's own stdlib disk-cache header.

Type coverage: unsigned/negative ints, byte strings, text strings, arrays, maps, tagged values, bool/null/undefined, half/single/double-precision floats. Decoder accepts definite and indefinite-length forms; encoder emits only definite (per §4.1 recommendation). f16/f32 payloads widen to f64 via bit- exact manual conversion — no float-cast intrinsic dependency.

encode_canonical sorts map keys by encoded byte-lex order for deterministic signing (the form required by COSE).

msgpack — MessagePack

mount core.encoding.msgpack.{MsgPackValue, encode, decode};

Binary alternative to JSON used by Redis RESP3, Pinterest MySQL adapter, PyPy, Erlang / Ruby / Python msgpack libraries. Full type coverage: nil / bool / int (u64 / i64 range with smallest- container encoding) / float32 / float64 / str / bin / array / map / ext. Encoder auto-selects the most compact wire form for each value; decoder handles every fixed + 8/16/32 length variant.

Nesting guarded at MAX_MSGPACK_NESTING = 128; pathological inputs return NestingTooDeep rather than overflowing the stack. Trailing bytes after a complete value return TrailingBytes — no silent partial-parse.

base32 — RFC 4648 §6 / §7

mount core.encoding.base32.{
    encode, decode, decode_no_pad,                    // §6 standard alphabet
    encode_hex, decode_hex, decode_hex_no_pad,        // §7 ExtendedHex alphabet
};

Standard alphabet A-Z + 2-7, case-insensitive on decode, = padded to a multiple of 8 chars. ExtendedHex alphabet 0-9 + A-V (used by NSEC3 hashed-owner names — preserves sort order with the underlying bytes).

Canonicality

Trailing-bits validation : unused low bits of the last quintet MUST be zero (non-zero → TrailingBits). On top of that, the decoders reject non-canonical pad counts. RFC 4648 §6 fixes pad ∈ {0, 1, 3, 4, 6} (encoding 5 / 4 / 3 / 2 / 1 bytes); other counts would let an attacker forge multiple distinct encodings of the same TOTP secret / onion address by toggling the unused real chars. The decoders surface those as Base32Error.InvalidLength(pad):

pad	Valid	Bytes encoded
0	yes	5 (full block)
1	yes	4
2	no	non-canonical
3	yes	3
4	yes	2
5	no	non-canonical
6	yes	1
7, 8	no	non-canonical

decode_no_pad / decode_hex_no_pad accept QR-code-style inputs that drop padding — used for TOTP secret sharing (Google Authenticator). The synthesised pad always lands in the canonical set, so the same gate applies.

base58 — Bitcoin-style

mount core.encoding.base58.{
    encode, decode, encode_check, decode_check, BASE58_ALPHABET,
};

Alphabet 123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz (omits 0, O, I, l to avoid human-confusion during copy- paste). Zero-prefix bytes map to leading 1s (the classic Bitcoin rule). encode_check / decode_check append a 4-byte double-SHA256 suffix — a one-character flip has probability 2^-32 of passing.

Used by Bitcoin legacy addresses (P2PKH / P2SH), Solana public keys, Stellar StrKey, IPFS v0 CIDs, Monero subaddresses.

pem — RFC 7468

mount core.encoding.pem.{
    PemBlock, encode_block, encode_bundle, decode_one, decode_all,
};

Textual envelope for DER blobs. Label-agnostic — callers dispatch on block.label to pick the parser ("CERTIFICATE", "PRIVATE KEY", "RSA PRIVATE KEY", "CERTIFICATE REQUEST", "X509 CRL", etc.).

let text = pem.encode_block("CERTIFICATE", der.as_slice());
let blocks = pem.decode_all(&text)?;     // concatenated-bundle aware

Line-wraps at 64 base64 chars per §2. Preamble text before the first -----BEGIN is ignored per §2 (the openssl x509 -text headers pass through). Label-mismatch on begin/end returns MismatchedLabels — the RFC-required rejection.

Example — JSON round-trip

mount core.encoding.json.{parse, stringify, JsonValue, JsonError};

fn round_trip() -> Result<Text, JsonError> {
    let raw = "{\"id\": 42, \"tags\": [\"a\", \"b\"]}";
    let value = parse(&raw)?;
    Result.Ok(stringify(&value))
}

Example — bearer token encoding

mount core.encoding.base64.encode_url;
mount core.hash.crypto.sha256.Sha256;

fn bearer_from_secret(secret: &[Byte]) -> Text {
    let digest = Sha256.digest(secret);
    encode_url(&digest)
}