Pardon Tech - Template Schemas

In the intro documentation, we cover templates and schemas from the user’s perspective. Here we will cover the schema system from the perspective of the maintainer.

Schemas represent the composition of templates, merged in one at a time, while templates are more of an intermediate representation in the scheme of things.

Templates are represented by a mixture of data and functions that return information about the template. Either way, the template value is ultimately interpreted by the schema merging it in, but in most cases (JSON/string data) the data is “expanded” using somewhat generic rules for values.

E.g., strings (text), numbers, booleans, and null values become scalars, arrays and objects become schemas for arrays and objects, and so forth.

How do our requests and responses (text) become interpreted as a JSON object which can be broken down by this system? We have jsonEncoding templates which become schemas which can merge with text externally, but will parse it and push the parsed data through to an internal schema.

As an example, let’s use the notiation “A B” to indicate merging the template A into the schema B.

We’ll use v(x) to indicate a scalar (simple value) schema that represents x. E.g., v("{{x}}") is a scalar value for the pattern "{{x}}", and v(8) is a scalar value for the number 8.

Merging the value 7 into this scalar produces a template value that is the combination of both:

8

v("{{x}}")

v("{{x}}", 8)

The resulting schema represents both the number 8 and the variable x, (implying that the value of x should be bound to the number 8). If the resulting schema were e.g., v("a", "b") this merge fails (the schema should add a diagnostic to the context and return undefined).

A schema representing an object({...}) would delegate the merge operation to each of its fields by key.

As an example, let’s merge 3 fields into an object with two fields.

{
  a: 1,
  b: 2,
  c: 3
}

object({
  a: v("{{a}}"),
  b: v("{{b}}")
})

1

"{{a}}"

2

"{{b}}"

3

...?...

object({
  a: v("{{a}}", 1),
  b: v("{{b}}", 2),
  c: v(3)
})

The resulting object merges the existing fields and adds new fields for the remaining value(s).

In the code, schemas are currently represented as functions that return objects with methods, with three methods which all schemas support: scope, match, and render.

Types of Schema Nodes

The important template node types are

• scalars (strings, numbers, booleans and null)
• objects and arrays
• mix / mux change the context mode.
• reference captures any value/node in a schema.
• encodings base64/json/etc… transforms data between formats.
• keyed / keyed.mv represent arrays as objects so they can be merged with more data.

Scalars are actually pretty involved, because of patterns and the interaction of defining and/or declaring values used via scripts.

Arrays have many options, because the rules to define and merge arrays are context-dependent. Sometimes an array of length=1 is an array of length 1, and sometimes it’s a template for all the elements of the array. (depending on the context mode) Keyed schemas adapt arrays to objects.

Another important core schema type is reference that binds a value to any schema node, and stub nodes exist as undefined nodes that can become anything through matching.

Note

Schemas should be immutable. Match operations return new schemas, any non-immutable state should go into the context runtime scope.

Schema Operations

Schemas operations flow a context object through them, transforming that context contextually. Contexts have a key path, and also a scope. The combination identifies the

For example, in the schema produced from this template,

{
  x: { a: "{{a}}" },
  y: { a: "{{a2}}" },
  c: "{{c}}"
}

the values a and a2 (and c) are all the same scope.

In contrast, an array template like [{ a: "{{x}}" }] creates a scope for each item, so that merging (matching) an array into a array schema works roughly as follows.

[
  { a: 1 },
  { a: 2 }
]

array(
  array={
    a: v("{{x}}")
  }
)

array(
  array=[
    { a: v("{{x}}", 1) },
    { a: v("{{x}}", 2) }
  ]
)

The definition of schema is a function returning an object with various operations.

The three core operations all schemas support are

• merge - matches or merges the schema against a template or value, producing another schema (or undefined on a failed match.)
• render - renders a schema back into plain data.
• scope - scopes out the schema node… registering declarations and definitions into the template runtime scope.

Additional methods are schema-defined, such as the object schema type having an additional method

• object - returns the object of schemas for the fields. etc…

The three main merge modes are

• mix - which integrates schema templates,
• mux - which integrates value templates, and
• match - which applies the schema to data (and only data).

A primary difference between mix and mux is how arrays are handled, when mixing in a single-length array, the value is treated as a schema for all the items in the array, but when muxing in a single length array, the value is treated as a template for an array of one element.

The match mode is applied to data, not templates, so when matching, even values like "{{t}}" should be treated a literal data (not a pattern), also single-length arrays are not treated as item templates, etc…

The render operation is used to create a value. It can be a full render or a preview render (useful for previewing or debugging the structure of a schema without evaluating any of the scripts). Some other render modes are optimized for other edge cases.

The scope operation is called both pre-merge and pre-render and is responsible for setting up the context’s scope structure. The scope structure is how Pardon figures out the correct order to resolve or evaluate dependent values regardless of where they are in request.

The merge operation is synchronous and intended to be relatively fast. In contrast the render operation is asynchronous as it potentially depends on async script evaluation.

Schema Encodings

Schema nodes can also represent various encodings of data.

For instance… we can have a schema like

base64(json(object({
  hello: v("{{hello}}")
})))

which accepts some base64 input, translates it to text, and then json takes that text, parses it as JSON, and matches that into the $decoded collector.

So this template would extract the decoded value into a javascript object.

If we pass "eyJoZWxsbyI6IndvcmxkIn0K" into this schema, first the base64 would parse into text.

"eyJoZWxsbyI6IndvcmxkIn0K"

base64(json(object({
  hello: v("{{hello}}")
})))

this text would be delegated to the json format schema node.

'{"hello":"world"}'

json(object({
  hello: v("{{hello}}")
}))

Next the json object itself matches with the object schema, and so on through to the hello field.

{
  hello: "world"
}

object({
  hello: v("{{hello}}")
})

"world"

v("{{hello}}")

v("{{hello}}", "world")

object({
  hello: v("{{hello}}", "world")
})

This will get rewrapped with the format so the overall behavior of matching "eyJoZWxsbyI6IndvcmxkIn0K" into our schema is magic.

"eyJ..0K"

base64(json(object({
  hello: v("{{hello}}")
})))

base64(json(object({
  hello: v("{{hello}}", "world")
})))

Schema encodings are one of the schema types that generally need to supporting merging with themselves, so base64(x) and base64(y) will merge as base64(x merge y), etc…