|
| 1 | +# JMESPath Compliance Tests |
| 2 | + |
| 3 | +This repo contains a suite of JMESPath compliance tests. JMESPath |
| 4 | +Community implementations can use these tests in order to verify their |
| 5 | +implementation adheres to the JMESPath spec. |
| 6 | + |
| 7 | +## Compatibility |
| 8 | + |
| 9 | +JMESPath Community is designed to be fully backwards compatible with [JMESPath.org](https://jmespath.org). |
| 10 | + |
| 11 | +However, in some rare circumstances, some differences may be observed. This paragraph lists the known differences: |
| 12 | + |
| 13 | +|Category|Compliance|Result|JMESPath.org Result|Description |
| 14 | +|---|---|---|---|--- |
| 15 | +|[literal.json](https://github.com/jmespath/jmespath.test/blob/53abcc37901891cf4308fcd910eab287416c4609/tests/literal.json#L193-L197)|`` '\\' ``| `` "\" `` | `` "\\" `` | JMESPath Community `raw-string` supports escaping both `` ' `` (single quote) and `` \ `` (backslash) characters, whereas JMESPath.org can only escape single quotes |
| 16 | +|[pipe.json](https://github.com/jmespath-community/jmespath.test/blob/304b287a9537673227c2e300a34ff8e4757579c5/tests/pipe.json#L131-L136)| `` `null`\|[@] ``| `` [null] `` | `` null `` | JMESPath Community lets a `null` left-hand side of a `pipe-expression` propagate to its right-hand side, whereas JMESPath.og shortcuts and does not evaluate the right-hand side if the left-hand side result is `null`. |
| 17 | + |
| 18 | +## Test Organization |
| 19 | + |
| 20 | +The `test/` directory contains JSON files containing the JMESPath |
| 21 | +testcase. Each JSON file represents a JMESPath feature. Each JSON file |
| 22 | +is a JSON list containing one or more tests suites: |
| 23 | + |
| 24 | + [ |
| 25 | + <test suite 1>, |
| 26 | + <test suite 2>, |
| 27 | + ] |
| 28 | + |
| 29 | +Each test suite is a JSON object that has the following keys: |
| 30 | + |
| 31 | +- `given` - The input data from which the JMESPath expression is |
| 32 | + evaluated. |
| 33 | +- `cases` - A list of test cases. |
| 34 | +- `comment` - An optional field containing a description of the test |
| 35 | + suite. |
| 36 | + |
| 37 | +Each JMESPath test case can have the following keys: |
| 38 | + |
| 39 | +- `expression` - The JMESPath expression being tested. |
| 40 | +- `result` - The expected result from evaluating the JMESPath |
| 41 | + expression against the `given` input. |
| 42 | +- `error` - The type of error that should be raised as a result of |
| 43 | + evaluating the JMESPath expression. The valid values for an error |
| 44 | + are: |
| 45 | + - `syntax` - Syntax error from an invalid JMESPath expression. |
| 46 | + - `invalid-arity` - Wrong number of arguments passed to a |
| 47 | + function. |
| 48 | + - `invalid-type` - Invalid argument type for a function. |
| 49 | + - `invalid-value` - Semantically incorrect value (used in slice |
| 50 | + tests) |
| 51 | + - `unknown-function` - Attempting to invoke an unknown function. |
| 52 | + - `not-a-number` - While evaluating arithmetic expressions. |
| 53 | +- `bench` - If the case is a benchmark, `bench` contains the type of |
| 54 | + benchmark. Available `bench` types are as follows: |
| 55 | + - `parse` - Benchmark only the parsing of an expression. |
| 56 | + - `interpret` - Benchmark only the interpreting of an expression. |
| 57 | + - `full` - Benchmark both parsing and interpreting an expression. |
| 58 | +- `comment` - An optional comment containing a description of the |
| 59 | + specific test case. |
| 60 | + |
| 61 | +For each test case, either `result`, `error`, or `bench` must be |
| 62 | +specified. Only one of these keys can be present in a single test case. |
| 63 | + |
| 64 | +The error type (if the `error` key is present) indicates the type of |
| 65 | +error that an implementation should raise, but it does not indicate |
| 66 | +**when** this error should be raised. For example, a value of |
| 67 | +`"error": "syntax"` does not require that the syntax error be raised |
| 68 | +when the expression is compiled. If an implementation does not have a |
| 69 | +separate compilation step this won\'t even be possible. Similar for type |
| 70 | +errors, implementations are free to check for type errors during |
| 71 | +compilation or at run time (when the parsed expression is evaluated). As |
| 72 | +long as an implementation can detect that this error occurred at any |
| 73 | +point during the evaluation of a JMESPath expression, this is considered |
| 74 | +sufficient. |
| 75 | + |
| 76 | +Below are a few examples: |
| 77 | + |
| 78 | + [{ |
| 79 | + "given": |
| 80 | + {"foo": {"bar": {"baz": "correct"}}}, |
| 81 | + "cases": [ |
| 82 | + { |
| 83 | + "expression": "foo", |
| 84 | + "result": {"bar": {"baz": "correct"}} |
| 85 | + }, |
| 86 | + { |
| 87 | + "expression": "foo.1", |
| 88 | + "error": "syntax" |
| 89 | + }, |
| 90 | + ] |
| 91 | + }] |
| 92 | + |
| 93 | +This above JSON document specifies 1 test suite that contains 2 test |
| 94 | +cases. The two test cases are: |
| 95 | + |
| 96 | +- Given the input `{"foo": {"bar": {"baz": "correct"}}}`, the |
| 97 | + expression `foo` should have a result of |
| 98 | + `{"bar": {"baz": "correct"}}`. |
| 99 | +- Given the input `{"foo": {"bar": {"baz": "correct"}}}`, the |
| 100 | + expression `foo.1` should generate a syntax error. |
| 101 | + |
| 102 | +# Utility Tools |
| 103 | + |
| 104 | +Most languages have test frameworks that are capable of reading the JSON |
| 105 | +test descriptions and generating testcases. However, a `jp-compliance` |
| 106 | +tool is provided to help with any implementation that does not have an |
| 107 | +available test framework to generate test cases. The `jp-compliance` |
| 108 | +tool takes the name of a jmespath executable and will evaluate all the |
| 109 | +compliance tests using this provided executable. This way all that\'s |
| 110 | +needed to verify your JMESPath implementation is for you to write a |
| 111 | +basic executable. This executable must have the following interface: |
| 112 | + |
| 113 | +- Accept the input JSON data on stdin. |
| 114 | +- Accept the jmespath expression as an argument. |
| 115 | +- Print the jmespath result as JSON on stdout. |
| 116 | +- If an error occurred, it must write the error name to sys.stderr. |
| 117 | + This check is case insensitive. The error types in the compliance |
| 118 | + tests are hyphenated, but each individual component may appear in |
| 119 | + stderr (again case insensitive). |
| 120 | + |
| 121 | +Here are a few examples of error messages that would pass |
| 122 | +`jp-compliance`: |
| 123 | + |
| 124 | +- Error type: `unknown-function` |
| 125 | +- Valid error messages: |
| 126 | + - `unknown-function: somefunction()` |
| 127 | + - `error: unknown function 'somefunction()` |
| 128 | + - `Unknown function: somefunction()` |
| 129 | +- Error type: `syntax` |
| 130 | +- Valid error messages: |
| 131 | + - `syntax: Unknown token '$'` |
| 132 | + - `syntax-error: Unknown token '$'` |
| 133 | + - `Syntax error: Unknown token '$'` |
| 134 | + - `An error occurred: Syntax error, unknown token '$'` |
| 135 | + |
| 136 | +> This will be substantially slower than using a test framework. Using |
| 137 | +> `jp-compliance` each test case is evaluated by executing a new process. |
| 138 | +
|
| 139 | +You can run the `bin/jp-compliance --help` for more information and for |
| 140 | +examples on how to use this tool. |
0 commit comments