Pardon Dataflow

This tutorial covers more details and ways data flows in and out of pardon templates.

The material assumes familiarity with concepts introduced in collections (and templates).

Pardon Collections An introduction to Pardon collections

Data matching

In the collections tutorial we introduced product get, update, and delete endpoints that required a {{product}} identifier. We will update the create products call to tell Pardon (and future consumers of this collection), where these identifiers come from.

The example/products/create endpoint returns newly created products. Let’s define a response template with "id": "{{products}}" to bind the value in the response.

example/products/create.https

>>>
POST https://example.com/products
Content-Type: application/json

{
  "name": "{{name}}"
}

<<<
200 OK

{
  "id": "{{product}}"
}

Response templates help pardon help you help your scripts interpret the request.

Note

Request and mixin https files can have multiple request and response templates: The first matching request and the first subsequent matching response in the endpoint and each mixin are what pardon uses.

To see this in action, please can create three products in our system.

Two new features are available in this tutorial:

1. making requests
2. a data view of the values resolved from the request or response

Tip

As we run these new exercises, you can track the products via the counter in the lower right of the screen.

You can reset the system with the reset button there too.

Exercises

add pencils

Add pencils to the product system.

Post the current request with this button.

Confirm the product count went up by one. See that the id is identified by "product" in the response as well.

add pens

Add pens to the product system.

Replace the input/ask with this and use the same button to reset the control back to request mode.

POST https://example.com/products

{
  "name": "pens",
  "price": 3
}

Add markers

Add markers to the product system.

POST https://example.com/products

{
  "name": "markers",
  "price": 5
}

Loading Pardon Playground...

Data Conflicts

Values can be resolved by template matching on the input request, or provided by inputs and evaluated in expressions. We have also shown how values can override the input in the case of config alternatives (like env=stage and env=prod overriding the endpoint origin).

Before we introduce structured data and scopes, we just tneed to see that values can conflict if they are specified to two values.

Try changing the value of name in either the values or the input request here. See how pardon becomes confused () if it can’t decided what the value is.

Loading Pardon Playground...

Data Scopes

Not all requests have such simple key=value data. Now that we have products, we’ll want to let customers order them, and an order generally has a list.

example/orders/create.https

>>>
POST https://example.com/orders

{
  cart: [{
    "id": "{{product}}",
    "quantity": number("{{quantity=1}}")
  }]
}

Pardon interprets a single element array as a template for each item, and then evaluates each item in its own scope. Let’s see how this works here

Exercises

Add an item

Add a cart item

Add another item to the list. Notice how the default quantity of 1 is supplied for both.

POST https://example.com/orders

{
  "cart": [
    { "product": "P1001" },
    { "product": "P1002" }
  ]
}

Globals

Override the default quantity

We can override the default quantity via values.

quantity=7

Locals

Override a specific item quantity

We can override the quantity for a specific item in the template

POST https://example.com/orders

{
  "cart": [
    { "product": "P1001", "quantity": 3 },
    { "product": "P1002" }
  ]
}

Scoped Compute

We can use a values in the local scope (or parent scopes). Try setting the quantity to a computation (last digit for the product id)

POST https://example.com/orders

{
  "cart": [
    { "product": "P1001", "quantity": 3 },
    {
      "product": "P1002",
      "quantity": "{{= Number(product.slice(-1))}}"
    }
  ]
}

and then try modifying that digit, while keeping an eye on the output.

Loading Pardon Playground...

Structured Values

You might have noticed that we turned off the values output view for the previous exercise,… that’s because nothing shows up there for scoped values.

To fix this, we can specify export names for interfacing values with our templates.

This means replacing {{product}} with {{items.product}} and {{quantity}} with {{items.quantity}} to specify items as the export name for the scoped values.

Exercises

Bind exports

Bind scoped data to export values

Use the editor () to bind export values to items, notice the items array appears in the data output.

example/orders/create.https

>>>
POST https://example.com/orders

{
  cart: [{
    "product": "{{items.product}}",
    "quantity": number("{{items.quantity=1}}")
  }]
}

Pass exports

Pass exports as input

Pardon supports passing the values entirely in terms of the items[] data. This can be useful when translating data in one format to another.

First we delete the cart from the request

POST https://example.com/orders

{
  "cart": [
    { "product": "P1001", "quantity": 7 },
    { "product": "P1002", "quantity": 42 }
  ]
}

And then we pass these values. Now the request, for better or worse, can be specified without any template matching on the body.

items=[
  { product: "P1001", quantity: 7 },
  { product: "P1002", quantity: 42 }
]

Loading Pardon Playground...

Arrays, Mix/Mux and Keyed-scopes

One of the major structural challenges with Pardon is understanding lists/arrays, because there’s not always lists, are they?

In the previous example/orders example, the order of the items is less important than the association of the quantity with the product, i.e., logically we would probably want to merge quantity into the element based on the id field.

[
  { id: "P1001" },
  { id: "P1002" }
]

[{
  id: "P1002",
  quantity: 7
}]

[
  { id: "P1001" },
  { id: "P1002", quantity: 7 }
]

Some less desirable alternative behaviors are

• failing to match because the ids differ in the first element, or
• failing to match because the array lengths differ, or
• appending a third element.

(We will see how to achieve the desired behavior later.)

Pardon achieves some of these behaviors through different modes when interpreting templates. The mode we have been using for collection templates has been mix, which is for specifying a schema-like-structure, and the mode we use for the request input template is mux, which is for templated data.

The primary difference between mix and mux modes is how they handle arrays.

mix mode arrays

In mix mode, a non-singular array is treated as a tuple. This could be useful when the position of the items is important, as well as the number, such as { "point": ["{{x}}", "{{y}}", "{{z}}"] }. In this case x, y and z are not scoped as array items.

A singular array, on the other hand, is treated as a template for all items, and template values referenced in it are scoped to the item (they will read from the outer scope but not resolve into it).

mux mode arrays

A mux mode array is that length, whatever it is. If the template is being applied on top of an existing array field, then the behavior is dependent on the base. If the base specified a tuple, the array must match the length of the tuple and the fields will merge pair-wise. If the base is a singular array (a template for all items), then the template will be merged with each item.

We saw this merging with {{quantity = 1}} applying to all items in our ordering example.

keyed arrays

keyed is not a mode, it is an additional layer of structure that treats the array as a map based on a resolved key. (And it is our key to achivieving the desired behavior we referenced earlier)

To make an array keyed we specify a template for how to resolve the key along with the array. e.g., for the example/products/list endpoint we could specify a keyed interpretation.

keyed({ id: "{{key}}" }, [{
  id: "{{product}}",
  name: "{{name}}",
  price: "{{price}}"
}])

This makes pardon internally treat this array as a map

{
  "P1001": { "id": "P1001", "name": "pencils", "price": 2 },
  "P1002": { "id": "P1002", "name": "pens", "price": 3 }
}

so now another array like [{ "id": "P1003", "name": "markers" }] is clearly a new element based on the id.

This mapping also affects export scopes.

Exercises

match response

create a response template

First let’s create a response template that exports the items as an array.

example/products/list.https

>>>
GET https://example.com/products

<<<
200 OK

[{
  "id": "{{items.product}}",
  "name": "{{items.name}}",
  "price": "{{items.price}}"
}]

Copy this to the request template and then execute the request to see how the response data () changes

Note

you’ll need to have some products created first

keyed response

use keyed on the response to get a map

Now let’s remove the {{items.product}} value and move that to an object key.

example/products/list.https

>>>
GET https://example.com/products

<<<
200 OK

keyed({ "id": "{{key}}" }, [{
  "name": "{{items.name}}",
  "price": "{{items.price}}"
}])

Copy this to the request template and the response data (if it’s still shown) it will update immediately.

super-keyed response

use @value and multiple outputs

Explore changing items.name to names.@value and items.price to prices.@value, this declares data exports of direct values in maps (rather than a map of objects).

See how the data output () changes.

(is this wise in this case? ?)

example/products/list.https

>>>
GET https://example.com/products

<<<
200 OK

keyed({ "id": "{{key}}" }, [{
  "name": "{{names.@value}}",
  "price": "{{price.@value}}"
}])

Explore here how the data output changes.

Loading Pardon Playground...

Multi-value keyed arrays

Unfortunately (for me), one more variant of keyed arrays is commonplace. This is the where there’s more than one element for the same key. The internal representation is a map to a list, but the way Pardon handles this structure is a bit different.

You might think this is a little crazy, but query parameters and headers are both commonly represented as structures.

By default query parameters (technically called the search portion of a request URL) is handled as a simple map, but we can turn on multi-map functionality.

Let’s do that here,… just to get a feel for things.

Exercises

two params

add two query params on the same key

Let’s update the request with two query params, with the same key

GET https://example.com/ping?hello=world&hello=earth

notice that only the last one is produced.

search: multi

allow multiple query params for the same

Update the endpoint with search: multi to enable multiple query param handling.

example/ping.https

search: multi
>>>
GET https://example.com/ping

Try this again

GET https://example.com/ping?hello=world&hello=earth

and now both parameters are produced.

search: multi - merging

include parameters in the tempalte

Include one or more query params in the base template

example/ping.https

search: multi
>>>
GET https://example.com/ping?hello=world

And try this again

GET https://example.com/ping?hello=world&hello=earth

Also try another value, like

GET https://example.com/ping?hello=venus&hello=earth

Or you can use a template and see how that merges (it merges with the first one).

example/ping.https

search: multi
>>>
GET https://example.com/ping?hello={{?greeting}}

Loading Pardon Playground...

Next Steps

Alright! That’s _an _introduction on dataflow. We see there’s many ways to organize collections and conform request and response processing to match the data structures that suit them.

The next section will be on integrating scripts, and then we’ll cover testcases, followed by project layering / organization and sharing.

Pardon Scripting How to write scripts