Test suite

The test_suite directory is the executable part of the specification. Every implementation runs it, and an implementation that passes it behaves like all the others. The cases are plain JSON files, so any language can read them.

test_suite has the cases of LIVR 2.0. The rules added in LIVR 2.5 have their own suite, test_suite_2.5, see The LIVR 2.5 suite.

Layout

test_suite/
├── positive/            valid data: check the output
│   └── 01-required/
│       ├── rules.json
│       ├── input.json
│       └── output.json
├── negative/            invalid data: check the errors
│   └── 01-required/
│       ├── rules.json
│       ├── input.json
│       └── errors.json
├── aliases_positive/    like positive, plus aliases.json
└── aliases_negative/    like negative, plus aliases.json

Each case is a directory named NN-rule_name:

File	Content
rules.json	The validation rules
input.json	The data to validate
output.json	positive cases: the expected output of the validator
errors.json	negative cases: the expected errors
aliases.json	alias cases: a list of aliases to register before validation

Getting the cases

Copy the test_suite directory into your implementation's repository and update it when the specification changes. To replace your copy with the current one, run this in the directory that contains it:

rm -rf test_suite
curl -fsSL https://github.com/koorchik/LIVR/archive/refs/heads/master.tar.gz | tar -xz --strip-components=1 LIVR-master/test_suite

Running the cases

For every case:

1. Create a validator with rules.json.
2. For alias cases, register every alias from aliases.json.
3. Validate input.json.
4. Positive: validation must succeed, and the output must be equal to output.json. Negative: validation must fail, and the errors must be equal to errors.json.

Compare the values deeply, ignoring the order of object keys. Remember that JSON has a limited set of types, see Types coercing.

In pseudo code:

for (const dir of listDirs('test_suite/positive')) {
    const validator = new Validator(readJson(`${dir}/rules.json`))
    const output = validator.validate(readJson(`${dir}/input.json`))
    assert.deepEqual(output, readJson(`${dir}/output.json`))
}

for (const dir of listDirs('test_suite/negative')) {
    const validator = new Validator(readJson(`${dir}/rules.json`))
    assert.equal(validator.validate(readJson(`${dir}/input.json`)), false)
    assert.deepEqual(validator.getErrors(), readJson(`${dir}/errors.json`))
}

The LIVR 2.5 suite

An implementation of LIVR 2.5 runs test_suite_2.5 too. It is a separate directory, because implementations copy test_suite into their repositories and some of their runners read every directory in it. It has the same layout and files as test_suite, and a few additions:

• schema_errors/ holds schemas that must not build: rules.json, and aliases.json when the error is in an alias. A case passes when creating the validator (with the aliases registered) fails, however the implementation reports it.
• A case may have an options.json, { "now": "2026-10-18T12:00:00Z" }: the current time for today, yesterday and tomorrow. Fix the clock the way the tests of your implementation do; a runner that cannot fix it skips such cases.

To get it, run the command above with test_suite_2.5 in place of test_suite. See the changelog for the details.

Every rule page of this site shows the test suite cases of that rule, and you can edit and run them in the browser. Look for the "Test suite cases" block below each rule.

Conformance results

The conformance/ directory contains a small test suite runner for each implementation. CI runs them against the latest version of every implementation, and the results are published on the implementations page. The score counts the cases of test_suite; the result of the LIVR 2.5 suite is shown separately and is not part of the score.

When an implementation starts failing cases of test_suite, because of a regression in a new release or because the test suite got new cases, CI opens an issue that mentions its maintainer. The issue closes by itself once all cases pass.

Adding cases

Found an edge case that implementations handle differently? Add it to the test suite:

1. Add the case to an existing directory, or create a new NN-rule_name directory with the next free number.
2. Check that the JavaScript implementation passes it. The live examples on this site make that easy.
3. Open a pull request and describe the behavior the case defines.