Test suite

The test_suite directory is the executable part of the specification. Every implementation runs it, and an implementation that passes it behaves like all the others. The cases are plain JSON files, so any language can read them.

Layout

test_suite/
├── positive/            valid data: check the output
│   └── 01-required/
│       ├── rules.json
│       ├── input.json
│       └── output.json
├── negative/            invalid data: check the errors
│   └── 01-required/
│       ├── rules.json
│       ├── input.json
│       └── errors.json
├── aliases_positive/    like positive, plus aliases.json
└── aliases_negative/    like negative, plus aliases.json

Each case is a directory named NN-rule_name:

File	Content
rules.json	The validation rules
input.json	The data to validate
output.json	positive cases: the expected output of the validator
errors.json	negative cases: the expected errors
aliases.json	alias cases: a list of aliases to register before validation

Running the cases

For every case:

1. Create a validator with rules.json.
2. For alias cases, register every alias from aliases.json.
3. Validate input.json.
4. Positive: validation must succeed, and the output must be equal to output.json. Negative: validation must fail, and the errors must be equal to errors.json.

Compare the values deeply, ignoring the order of object keys. Remember that JSON has a limited set of types, see Types coercing.

In pseudo code:

for (const dir of listDirs('test_suite/positive')) {
    const validator = new Validator(readJson(`${dir}/rules.json`))
    const output = validator.validate(readJson(`${dir}/input.json`))
    assert.deepEqual(output, readJson(`${dir}/output.json`))
}

for (const dir of listDirs('test_suite/negative')) {
    const validator = new Validator(readJson(`${dir}/rules.json`))
    assert.equal(validator.validate(readJson(`${dir}/input.json`)), false)
    assert.deepEqual(validator.getErrors(), readJson(`${dir}/errors.json`))
}

Every rule page of this site shows the test suite cases of that rule, and you can edit and run them in the browser. Look for the "Test suite cases" block below each rule.

Conformance results

The conformance/ directory contains a small test suite runner for each implementation. CI runs them against the latest version of every implementation, and the results are published on the implementations page.

Adding cases

Found an edge case that implementations handle differently? Add it to the test suite:

1. Add the case to an existing directory, or create a new NN-rule_name directory with the next free number.
2. Check that the JavaScript implementation passes it. The live examples on this site make that easy.
3. Open a pull request and describe the behavior the case defines.