Conformance

Claiming conformance

What an implementation must do to claim conformance, as SPEC §9 states it today:

An implementation claiming conformance with schema set 1.2 (or any 1.x set, under the additive rule of §10) MUST:

  1. Validate every manifest and lease against the published schemas, and refuse what does not validate.
  2. Implement the narrowing algebra of §5 exactly, including refusal of an empty grant, and pass every vector in conformance/narrowing-vectors.json by supplying narrow on its adapter.
  3. Record every issue, narrowing and refusal.
  4. Produce canonical bytes per §7 such that its declared_hash for a given declaration matches the value another conforming implementation produces, pass every vector in conformance/canonical-vectors.json and conformance/jcs/ (§7.1), and, when it verifies signatures, pass conformance/signature-vectors.json by supplying hash and verify on its adapter: the fixtures verify under the published test key and stop verifying when one byte of the signature or of the granted manifest changes. An implementation that only reads artefacts and never issues, signs or hashes one is exempt from this clause and MUST say so when it claims conformance, because the runner reports it as unchecked rather than as passed.
  5. Pass the published conformance corpus: accept every fixture under fixtures/valid/ and reject every fixture under fixtures/invalid/ at the place its .expect.json names. Each invalid fixture carries a .reason file stating what the rejection is for and an .expect.json naming the instance path the rejection must be reported at; an implementation that rejects a fixture somewhere else has rejected it for the wrong reason, which is not conformance. The runner checks the path when the adapter reports its errors, and says so when it cannot.
  6. State which signature algorithm it uses.
  7. Apply the lease rules of §6 and §6.1 (L1, L2, and L3 when the journal is available) and pass every fixture under fixtures/invalid-by-rule/ by supplying rules(lease, context) on its adapter: each such fixture is schema-valid and MUST be refused at the rule its .expect.json names. An implementation that supplies no rules is reported as unchecked, not as passing.

An implementation SHOULD publish the result of running the corpus, so that the claim is evidence rather than assertion.

The corpus is the conformance test. A schema alone is not: two implementations can both validate against the same schema and still disagree about what they refuse, which is the disagreement that matters.

Schema agreement is not interoperability. Two implementations can accept and reject exactly the same artefacts and still produce different bytes for the same declaration, and therefore be unable to verify a single one of each other's signatures. §7.1 is the half of the corpus that catches this, and it is the half an implementation is most likely to skip, because everything looks correct until somebody else's hash arrives.

The corpus tests schema validity, not issuance. These are different questions and an implementer MUST NOT conflate them. A fixture under fixtures/valid/ is a well-formed manifest; it is not a manifest that must be granted a lease. agent-lease-manifest.minimal.json is the clearest case: it validates, and an issuer MUST refuse it, because it declares kind: "native" with runtime_agent: "unknown" (§4.2) and because narrowing leaves allow.tools empty (§5.2). Both statements are true at once.

Conformance therefore has three halves, which is one more than the phrase allows and exactly the point: validate as the schemas say, canonicalise as §7 says, and issue or refuse as §4 and §5 say. The corpus checks all three. conformance/narrowing-vectors.json carries declared-and-parent pairs with the granted manifest or the refusal codes narrowing MUST produce, covering every row of the §5 table, every containment row of §5.1 and every refusal of §5.2; an implementation runs them by supplying narrow(declared, parent) on its adapter, and one that does not is reported as unchecked rather than as passing. The vectors were generated from the reference implementation and committed, so the reference implementation is the oracle and the file is the contract: a second implementation that disagrees with a vector has found either its own defect or the reference's, and either is a finding the contract wants.

Implementation reports

Recorded portability

From the README, as written:

In CI, on every pull request and every push to main, npm test runs on Node 20 and 22 and the additive guard runs on Node 22 (against the base branch of a pull request, or the commit a push started from), beside a job that compiles every pattern under RE2. The point of a corpus is that a third party can check the claim, so the check has to be runnable by someone who has never seen the product.

Since 1.2 the claim that "a reader in any language needs nothing else" is checked rather than made. conformance/suite/draft7/ carries the corpus in the official JSON-Schema-Test-Suite format (one file per schema, every fixture a test; npm test fails when it is stale), and CI runs it through Bowtie against six validators in six languages — go-jsonschema, rust-jsonschema, python-jsonschema, java-json-schema, dotnet-jsonschema-net and js-ajv — failing on any disagreement. A second job runs Sourcemeta's jsonschema metaschema and lint over the schemas, with six style rules excluded by name and for a reason each in the workflow. The latest result is the contract workflow's run on main: https://github.com/Cognitive-Delivery/contract/actions/workflows/contract.yml.

From the recorded checks under fixtures/evidence/: