Repository navigation
Harden schema verification across the reference SDKs - #50
Merged
Merged
Conversation
SP-3, SP-4, SP-5: the four canonicalizers were not byte-compatible. Go HTML-escaped <, > and & (and always escaped U+2028/U+2029), JavaScript dropped __proto__ keys by assigning through the prototype setter and sorted keys by UTF-16 code unit, Python emitted NaN/Infinity and 1e-07 style exponents, and integers above 2^53 silently collided in JS/Go. Every SDK now implements the same hand-written serializer: keys sorted by code point, minimal string escaping with literal <>& and U+2028/9, ES6 Number::toString for non-integral values, integral values as plain decimals, and rejection of NaN/Infinity, lone surrogates, duplicate keys, invalid UTF-8 and any number whose magnitude exceeds 2^53-1. Each SDK gains a strict parser (parse_schema_json / parseSchemaJson / ParseSchemaJSON / parse_schema_json) that enforces the rejection rules at parse time, and the Rust canonicalize functions now return Result. tests/cross-language/canonicalization/ holds 33 vectors (19 canonical with expected bytes and SHA-256, 14 reject cases) asserted by all four suites. Also formats the Rust tree and silences the async-trait double_must_use lint so clippy -D warnings passes on current toolchains.
SP-2: Go passed the schema digest straight to ecdsa.Sign while Python, JavaScript and Rust all sign ECDSA(SHA-256(digest)), so no Go-signed schema or skill verified anywhere else and vice versa. SignHash and VerifySignature now apply SHA-256 to their input first, matching the other three; the bundle path stops pre-hashing so its wire format is unchanged (the shared signed_bundle fixture still verifies). SP-10: Python accepted any EC curve, JavaScript accepted any OpenSSL key including RSA, Go never checked the curve. Key loading, signing, verification and fingerprinting now reject anything but secp256r1. SP-19: Go ignored trailing bytes after the DER signature; it now rejects them. go.mod pins go 1.22 with a toolchain directive. Adds tests/cross-language/signatures/schema-signature.json, a fixed ECDSA vector (signature, tampered schema, trailing-byte signature) asserted by all four suites, test-only key material under tests/cross-language/keys/ including P-384 and RSA public keys that every SDK must reject, and a Go unit test pinning the hash convention.
SP-11: three SDKs fetched keys over http:// when the domain string carried that prefix, every fetcher followed redirects (including scheme downgrades), none capped the response body, Rust had no timeout, and no SDK validated the domain before using it as a URL, pin key or file name. Each SDK now has a shared RFC 1123 domain validator (domain module) and a hardened transport helper: https only, redirects refused, 64 KiB body cap, connect/read timeouts, and the JavaScript abort timer now covers body consumption. construct_well_known_url validates and normalizes the domain and refuses http://. SP-16: LocalFileResolver built paths from the raw domain string, so "../x" or an absolute path selected an arbitrary JSON file. Paths are now derived from the validated domain and must stay under the configured directory. SP-9: revocation fetch failures were swallowed as "not revoked" in the Python, JavaScript and Go resolver and legacy paths and in Rust's ChainResolver. fetch_revocation_document now raises/returns an error, ChainResolver propagates it, validate_key_not_revoked fails closed and also consults revocation_endpoint, the Go workflow/pinning/CLI paths no longer "proceed with caution", and Go's KeyPinning gains a SetRevocationChecker hook so integrators can plug in offline sources. The Go --auto-pin flag now pins without --interactive as documented.
SP-1: the skill signature covered only the 32-byte root hash, so
expires_at, schema_version, previous_hash, canonicalization, domain,
skill_name, signer_kid, signed_at and file_manifest were all unsigned
and freely rewritable (expiry stripping, lineage splicing). New
signatures are written as schemapin_version "1.4" and sign the
canonical (schemapin-v1) signature document with signature set to "";
verifiers recompute the root hash, require skill_hash to match it
(SP-14, now hex(root) in Rust too) and verify over the canonical
document. Pre-1.4 "1.3" documents still verify through the legacy
root-hash path with a legacy_signature_format warning, and
verify_chain refuses to anchor on them.
SP-12/SP-13: skill canonicalization v2 hashes every entry as
SHA-256("f" || be64(len(path)) || path || be64(len(content)) ||
content) and the root over length-prefixed path/digest pairs, so the
path/content boundary is no longer malleable; symlinks and special
files abort canonicalization instead of being silently skipped; file
count and total size are capped. The legacy algorithm is kept only for
verifying 1.3 documents.
SP-8: verify_skill_with_resolver binds the caller's domain to the
signature's domain before any I/O, and the offline verifiers accept an
expected domain (DOMAIN_MISMATCH / DOMAIN_INVALID); the signature's own
domain field is display-only and flagged with domain_unbound when used.
SP-17: expiration is fail-closed (SIGNATURE_EXPIRED) with a 5-minute
skew window and strict RFC 3339 parsing in every SDK; allow_expired
restores the degraded-not-failed behavior. Pins are written only after
a successful verification (KeyPinStore gains check/pin/seed), the DNS
TXT cross-check runs before pinning, and a DNS-aware schema verifier is
added. Go rejects an explicitly empty canonicalization field (SP-19).
tests/cross-language/skill/ adds a v1.4 signed fixture and a legacy 1.3
fixture with expected manifests and skill_hash, asserted by all four
suites including per-field tamper checks.
… key SP-6: verify_trust_bundle TOFU-pinned the bundle authority by the kid string inside the bundle, so any self-signed bundle under a fresh kid passed on first use and could load attacker keys for any domain. The authority must now be pre-seeded (seed_bundle_authority / seed_bundle_authority_from_pem) or verification fails with BUNDLE_AUTHORITY_UNTRUSTED; first-use trust is an explicit opt-in (allow_first_use) and the structured result reports first_use so hosts can gate on it. The authority is pinned only after the signature verifies. Design issue 4: merge_trust_bundles compared timestamp strings lexicographically and let any authority win. Merges now take an optional authority precedence list and compare parsed RFC 3339 instants; unparseable timestamps rank oldest. SP-19: the bundle signing input is built from the known field set in Python and JavaScript too (optional strings omitted, revoked_keys always present), matching the Go/Rust struct serialization, so unknown fields no longer change the bytes in two SDKs only; duplicate domain entries are rejected at sign, parse and resolver load; JS parses expires_at with the strict RFC 3339 parser. SP-18: the fixture authority private key (go/schemapin_private.pem) and the demo private keys are removed from the repository and ignored going forward; tests/cross-language/signed_bundle.json is re-signed under the clearly test-named kid schemapin-test-fixture-authority with the key in tests/cross-language/keys/, and every SDK's fixture test seeds that authority instead of relying on first use. Python keygen and the demo scripts write private keys with mode 0600.
SP-15: the legacy KeyPinning stores (Python SQLite, JavaScript JSON, Go Bolt) were keyed by tool_id alone, so whoever got pinned first under a bare tool id owned it for every domain and the workflow's domain argument only affected revocation lookups. Pins are now keyed by tool_id@domain in all three stores, existing stores are migrated in place on open, lookups take the domain and never return a pin made for another domain (the domain-less lookups are kept as deprecated shims that refuse to guess when a tool id is pinned for several domains), and the SchemaVerificationWorkflow in all three languages validates the domain and pins only after the signature verifies. The JavaScript store additionally fails hard on an unreadable or malformed file instead of silently resetting to "first use", writes atomically (temp file + rename) with mode 0600, and its interactive flows now run the fail-closed revocation check (with an injectable revocationChecker for offline use); the Python store is created 0600. Design issue 9: the legacy store and workflow are marked deprecated in favor of KeyPinStore + verify_schema_with_resolver.
SP-7: the server accepted unauthenticated key upload and revocation, combined CORS allow_origins ["*"] with allow_credentials, wiped the revocation list on every upload, ran with debug/reload on 0.0.0.0 and never read its own rate-limit or admin_api settings. Every mutation now requires a bearer token (security.admin_token or SCHEMAPIN_ADMIN_TOKEN) and features.admin_api, which defaults to off; uploads merge (the previous key is moved to revoked_keys, existing revocations are preserved, a revoked key cannot be re-uploaded) and only P-256 keys are accepted; credentials are never sent with a wildcard origin; a per-client rate limit is enforced; defaults bind to 127.0.0.1 with debug and reload off. Design issue 11: /.well-known/schemapin.json now selects the developer from the request Host (RFC 8615) with no first-developer fallback, so a client resolving one domain is never handed another developer's key; the per-domain path is documented as a non-standard rewrite target. The app is built by create_app(config) so it can be tested; server/tests/test_server.py covers auth, merge-on-upload, host routing, CORS, key policy and rate limiting.
Design issue 3: spec section 18.4 asks verifiers to keep a latest_known_hash per tool, but no SDK shipped one, so verify_chain could only compare two documents the caller already held and a publisher (or anyone with the key) could fork from an old version or reuse a schema_version. LineageStore (Python, JavaScript, Go, Rust) records the latest accepted skill_hash and schema_version per tool_id@domain, requires every new document to name that hash as its previous_hash, requires schema_version to increase (numeric major.minor.patch comparison), refuses legacy 1.3 documents whose hashes are unsigned, and round-trips through JSON for persistence.
cargo audit flagged crossbeam-epoch 0.9.18 (RUSTSEC-2026-0204), h2 0.4.13 (RUSTSEC-2026-0258), rustls 0.23.36 (RUSTSEC-2026-0285) and the yanked chacha20 0.10.0 in the optional fetch/dns dependency trees; they are updated to patched releases and the audit is clean. govulncheck reported go1.26.0 standard-library issues reachable from the discovery code; go.mod now pins toolchain go1.26.9 so builds use a patched runtime.
No fuzz or property tests existed for the parsers that face untrusted input. Python gains hypothesis tests, JavaScript a seeded-generator property suite, Go native fuzz targets (run as seeds in go test) for the strict JSON parser, TXT parser, domain validator, PEM loader, .schemapin.sig and trust-bundle parsing, and Rust gains proptest properties plus libFuzzer targets under rust/fuzz/. Every property checks that canonicalization is idempotent and that parsers fail only with their documented error types. The Rust property test found a real divergence: serde_json's default float parsing is best-effort and rounds some 17-significant-digit inputs to a different double than Python, JavaScript and Go, which would change the canonical bytes. The float_roundtrip feature is now enabled and a corpus vector (34-float-17-digits) pins the case for all four SDKs.
SP-20 and the spec gaps from the review. The specification now defines schemapin-v1 canonicalization normatively (section 4: code-point key order, minimal escaping, ES6 numbers with the 2^53-1 limit, rejection rules, the signing construction), adds section 4b for skill directory canonicalization and the signed signature document, makes pinning happen after verification (sections 7 and 15), states the transport and fail-closed revocation requirements (section 9), rewrites expiration as fail-closed with a risk statement (section 16), covers the DNS cross-check for schemas (section 17), specifies the LineageStore (section 18.4) and the bundle authority allow-list, signing field set and merge precedence with a risk statement (section 21). SECURITY.md lists the supported versions and the fail-closed properties; the "Rust only" banners are gone; docs/skill-signing.md examples now match the real APIs and describe the v1.4 format; auto-pin is documented as an opt-in rather than the standard call; trust-bundle and version-binding docs describe the allow-list and LineageStore; the integration demo no longer disables revocation checks or uses plaintext discovery; keygen only produces ECDSA P-256 keys; CHANGELOG gains an Unreleased section listing every fix by ID.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Schema verification now uses a defined cross-language canonicalization contract, P-256 validation, hardened discovery and revocation handling, authority-bound trust bundles, and domain-qualified pin stores. Skill signatures cover the complete signature document, and verifier-side lineage tracking binds schema versions. The reference server, CLI, documentation and shared conformance fixtures describe and exercise the security-contract changes.
Python requires 3.10+ and Rust requires 1.89+. Packaging and CI enforce the supported minimums, with patched Python/server dependencies, build tooling and JavaScript development dependencies. Python 3.10 uses the compatible UTC timezone API.
Validation: current-head CI passed Rust 1.89/stable with all features and Python 3.10/3.12 including reference-server routes and dependency audits. Local validation also passed Rust tests, all-target Clippy and formatting, 344 Python tests, 277 JavaScript tests and lint (36 existing warnings), Go tests/vet and eight server route tests. Dependency audits passed; Go reported no reachable vulnerabilities. Built Python wheel metadata and the 18 affected pin-store tests were checked locally. No release or tag is included.