Pardon Collections

This tutorial builds on the concepts explained in Pardon Templates,

Pardon Templates A quick introduction to Pardon's templates

Services and Endpoints

A Pardon collection is generally one or more services anchored by a service.yaml file and with each endpoint represented by an https (HTTP Schema) file.

Schema files are a sequence of http request and response templates, and optionally some additional inline configuration.

A basic collection defining a service and the actions for a products REST API might look like this.

Directorycollection
- Directoryexample the example service
  service=“example”
  - service.yaml common configuration for all the endpoints
  - ping.https a basic endpoint
    endpoint=example/ping
    action=ping
  - Directoryproducts we can group related endpoints
    create.https create a product
    action=“create”
    endpoint=“example/products/create”
    get.https get a product
    action=“get”
    endpoint=“example/products/get”
    update.https update a product
    delete.https delete a product
    list.https list products

When developing a collection, one might start with a service.yaml and confirm the setup with something like a ping.https endpoint.

service.yaml
ping.https

The service.yaml file contains common specifications like config, defaults, and mixins (which we’ll get to later in this tutorial).

config:
  origin:
    env:
      prod: https://example.com
      stage: https://stage.example.com

And our ping.https file specifies how to make the ping call to the service.

>>>
GET https://example.com/ping

So far, so good: we should be able to build requests for both stage and production origins.

Input values

The input request is the “most direct” mechanism for getting data into Pardon, but not everything can be solved without another layer of indirection!

Values can supply data to templates, and can even be used to rewrite templates where config alternatives are involved.

In these exercises a new input field for values is available to explore this mechanism (the input).

Exercises

Values input
Values override

Try setting a value in the values input.

config choices can be overridden with values. This allows any request to be easily repurposed to a different environment (in particular, replaying tests).

env=stage

Override the the request input

Set env=prod to override https://stage.example.com.

(try the other way as well).

env=prod

GET https://stage.example.com/ping

Loading Pardon Playground...

When using the pardon commandline utility, values can be specified with key=value syntax anywhere in the arguments.

% pardon https://example.com/ping env=stage --http
curl "https://stage.example.com/ping"

We can also omit any http or URL data and select an endpoint directly

% pardon endpoint=example/ping env=stage --http
GET https://stage.example.com/ping

A RESTful resource

The products lifecycle would also defined with https files:

For product creation, this template only defines the name field (which makes it mandatory),

>>>
POST https://example.com/products
Content-Type: application/json

{
  "name": "{{name}}"
}

Defining the response template instructs pardon to extract the product value from the response, assuming it matches. Equally importantly, it provides a little documentation about the shape of the response for humans. We will cover dataflow in a later section, as it’s more of an advanced topic for scripting.

The REST of the product resources endpoints are specified minimally here, just enough to distinguish them from each other.

Requests info for a single product.

>>>
GET https://example.com/products/{{product}}

Updates info for a single product.

>>>
PUT https://example.com/products/{{product}}
Content-Type: application/json

{}

Deletes a single product.

>>>
DELETE https://example.com/products/{{product}}

Lists products.

>>>
GET https://example.com/products

With all this set up we can finally explore how pardon works with a collection of requests.

Exercises

Set a value via values.

Notice how we can provide the {{name}} value here without having to specify a request body at all!

Many simple requests can be parameterized to take advantage of this abbreviated input mechanism.

env=stage name=thneed

Add a parameterized query param

In the spirit of progressive enhancement, we can add a parameterized value in our input,

GET https://stage.example.com/products?type={{type}}

and then define it.

env=prod type=pencils

How can we make supplying the type optional?

Select the endpoint directly with parameterization.

The service, action, and endpoint values can be used to select (or at least dramatically narrow down) which endpoint templates Pardon even attempts to evaluate.

First, please delete the entire input. We don’t need it!

POST https://example.com/products

Then select the endpoint directly.

endpoint=example/products/list

you can try different endpoints, and/or env=stage if you like, for the get/update/delete endpoints, you’ll need to specify a product=... value as well.

This only works for GET method endpoints.

Non-GET endpoints need the method specified as well, this adds a little friction to prevent accidental POST requests.

endpoint=example/products/create method=POST name=thneed

Loading Pardon Playground...

Config

The config mapping enables templates to assert possible combinations of certain values.

A simple mapping of

config:
  key: value

specifies one option of { "key": "value" }, and

config:
  key1: value1
  key2: value2

specifies one option of { "key1": "value1", "key2": "value2" }. This is not functionally any better than normal template matching, but it has its uses since the data is outside the rendering of the template.

Revisting the config we introduced earlier

config:
  origin:
    env:
      prod: https://example.com
      stage: https://stage.example.com

You can read this as

origin = (when env is prod) https://example.com, and
origin = (when env is stage) https://stage.example.com

and this allows for two variants of the request

{ env: "prod", origin: "https://example.com" }, and
{ env: "stage", origin: "https://stage.example.com" }

Specifying env=prod implies the origin value and vice-versa, and env becomes a value that can be used in scripts.

Patterns, with expressions, can be included, e,g., this config

config:
  origin:
    env:
      prod: https://example.com
      stage: https://stage.example.com
      local: http://localhost:{{port=8000}}

also allows env=local and provides a default port.

Mixins

Oh no. In our rush to add features, we forgot to include authorization headers to call these endpoints: the security team took care of this in the gateway on stage, and now all of our endpoints need to be updated, and tested.

We want to tell Pardon that by default all calls to this service needs authentication, rather than updating every endpoint.

We can do this with a mixin at the service level.

config:
  origin:
    env:
      prod: https://example.com
      stage: https://stage.example.com
mixin:
  - ./auth.mix.https

We’re putting it next to our service in the file tree. But any relative path inside the service directory is fine.

Directorycollection
- Directoryexample
  - service.yaml
  - auth.mix.https
  - ping.https
  - Directoryproducts/
    create.https
    …

The contents look (kind of) like any other https endpoint.

>>>
ANY //
Authorization: {{ @auth = `${env}-auth-token` }}

The special request method ANY matches all request methods, and the special value // matches any URL. (A “mixin” is mixed-in only if it is “compatible” with the request. So this mixin can apply to any service call.)

This adds an authorization header to all requests.

HTTP requests unfortunately contain multiple types of information: information that selects an endpoint, information that we want to send to the endpoint, and information that we need to send to tell the server who we are. This last category of values are often both security sensitve and (relatively) useless for human comprehension, so we tend to redact them.

Template values decorated with @ are considered secrets, and Pardon knows to keep them out of durable locations, and just generally hide them by default.

Commonly, we would compute the secret using an imported function.

import:
  ./helper.ts:
    - makeAccessToken
>>>
ANY //
Authorization: {{ @auth = makeAccessToken(env) }}

where makeAccessToken is an async function exported from a javascript (or typescript) file.

Let’s try it out in stage and confirm all our endpoints are working and have authorization applied… (The @ in @auth marks it a secret. You can unlock the secret for viewing by tapping the lock button in this exercise.).

Loading Pardon Playground...

Mixin-Match

TL;DR: mixins progressively enhance the input request, but only if they match.

They do not affect whether the endpoint template matches the input request.

Awesome! We have applied a security measure applied to our requests. And we provided confirmation of all our requests working on stage, so ops rolled these changes out to production.

Unfortunately, now alerts are firing because the (oops) only-running-in-production-liveness-probe hitting /ping also requires an auth token now.

We missed this case in staging because when we tested our updated collection we sent auth tokens to every endpoint.

Well, it’s not actually an outage, and the fix isn’t even on our end (ops is in control of removing the Authorization header check from the /ping endpoint). But we do want to adjust our collection to not send an unnecessary auth token for ping calls.

One option is removing the mixin from service.yaml and including it selectively from every authenticated endpoint.

config:
  origin:
    env:
      prod: https://example.com
      stage: https://stage.example.com
mixin:
  - ./auth.mix.https

mixin:
  - ../auth.mix.https
>>>
POST https://example.com/products

mixin:
  - ../auth.mix.https
>>>
GET https://example.com/products/{{product}}

mixin:
  - ../auth.mix.https
>>>
PUT https://example.com/products/{{product}}
Content-Type: application/json

{}

mixin:
  - ../auth.mix.https
>>>
DELETE https://example.com/products/{{product}}

mixin:
  - ../auth.mix.https
>>>
GET https://example.com/products

but that touches many files. As the collection grows, this kind of maintenance would grow as well.

It is cleaner (in this case, at least) to specify an opt-out for the few exceptional endpoints that don’t need auth, rather than opting-in. To have the mixin supply an opt-out behavior, we can add a config section.

config:
  authorization: token
>>>
ANY //
Authorization: {{ @auth = `${env}-auth-token` }}

Just like origin in the main request, this authorization value needs to match (or be not specified) in the input for this mixin to be applicable.

We can then default authorization as none in our ping endpoint (choosing any value other than token), which disables the mixin selectively.

defaults:
  authorization: none
>>>
GET https://example.com/ping

This implements the same change in behavior with two changes instead of N+1, and provides some additional controls we explore here:

Exercises

Notice the authorization header is gone, because it’s disabled for ping by default now.

GET https://example.com/ping

Send authorization to ping anyway?

Override the default authorization to force an authenticated ping request!

authorization=token

GET https://example.com/ping

Send an unauthenticated request to products.

Override the default authorization to force an unauthenticated products request.

authorization=none

GET https://example.com/products

Loading Pardon Playground...

default.https

A service can have a default request that is tried if no specific request matches. The default.https request extends the service.yaml specification.

Since multiple services can have defaults, we probably want to use defaults only for stage/prod origins, and not localhost ones (since then they would conflict with each other).

config arrays
elided values ...

To specify a subset of environments, you can use a list in the config mapping.

config:
  env:
    - stage
    - prod
mixin:
  - ./auth.mix.https
>>>
ANY //

Alternatively you can use ellipses (...) to specify no value for origin. This means exactly the same thing.

config:
  origin:
    env:
      stage: ...
      prod: ...
mixin:
    - ./auth.mix.https
>>>
ANY //

When merging config mappings, all possibilities of the last mapping will be retained, along with all options of compatible values for each of them, if present. So in this case the config acts as a filter, only applying “default.https” to requests to https://example.com/ and https://stage.example.com/ that don’t match a more specific template.

Collection Layering

To match team structures, collections are defined in layers:

A base layer might define the endpoints, but credentials can be defined in another. Different teams have different access to credentials to the same services, so Pardon supports separation of concerns and access requirements by allowing layers of functionality to be applied.

Requests, mixins, and service configs can all be layered.

A root pardon config defines all the layers of collections to use. We’ll need to revisit this in the section on scripting.

Next Steps

Next we will cover more dataflow mechanisms.

Pardon Dataflow How data flows through templates.

Then layering and scripting collections.