Converting The Interim Swyftx Data Structure Into A Valid OpenAPI Format

In the previous post for this series, we scraped the Swyftx docs website using Puppeteer to create an interim, offline data structure.

The next steps that we want to take is to move that data structure into a valid OpenAPI JSON structure.

In the words of OpenAPI themselves: The OpenAPI Initiative (OAI) was created by a consortium of forward-looking industry experts who recognize the immense value of standardizing on how APIs are described.

This standardization has a myriad of benefits, but I will be aiming to convert into this format so that I have a standard schema logged somewhere for Swyftx based on what I have scraped as well a "source of truth" to generate the TypeScript API flow later.

Source code on this project be found here.

Prerequisites

Previous posts in the "Creating a trading bot on Swyftx" series.

Getting started

At this stage of the post, our repo should already contain some vital information that is required.

If you are not up to date, please check back in with the previous posts in this series.

Otherwise, I'll give a quick update to where we are at. We currently have a file data.json at the root of our repo that has the following shape:

{
  "endpoints": [
    {
      // url path (not inclusive of the base url)
      "url": "/auth/refresh/",
      "request": {
        "requestExampleValue": {
          // example request body
        }
      },
      // array of params as objects
      "parameters": [
        {
          "paramKeyValue": "assetId",
          "paramRequirementValue": "Required",
          "paramDescriptionValue": "Asset ID. See Get Market Assets id for all asset ids"
        }
        // ... more params
      ],
      "responses": [
        {
          "responseStatusValue": "200",
          "responseExampleValue": {
            "accessToken": "eyJhbGciOiJSUzI1N...",
            "scope": "app.account.read ..."
          }
        },
        {
          "responseStatusValue": "500",
          "responseExampleValue": {
            "error": {
              "error": "StillLoading",
              "message": "Please try again or contact support."
            }
          }
        }
        // ... more responses
      ]
    }
    // ... more endpoints
  ]
}

We are going to take the data from that file and convert it to be valid when checked against to OpenAPI 3.0.3 specification via a script.

To do so, let's create a scripts folder to hold our script, as well as a helpers.js file. We will also install Jest to the project as there is a complicated helper function we need to write that will be easier to test with Jest. We will also install Lodash to the project as we will be using it to help with some of the data manipulation.

$ mkdir scripts
# Create a file to hold our script to do the conversion
$ touch scripts/convert-data-to-openapi-format.js
# Create a helper file for our script
$ touch helpers.js
# Install Jest to the dev deps
$ npm i lodash
$ npm i -D jest

I've noted that the repo itself is a bit messy in terms of file structure, but we can rectify that in a later post.

We now have everything in place to write out of script.

Writing out our script

The script itself in scripts/convert-data-to-openapi-format.js will again be another one-and-done style script (similar to our scraper).

I will follow a similar format of writing out the script using the main function and invoking it immediately.

In our script, we need to do the following things:

Import the data.json file that we wrote out previous.
Create a mutable object that will adhere to the OpenAPI specification.
Iterate over the endpoints array in the data.json file and create the relative entries into our specification.

Step (3) is the most complicated part of the script. As opposed to going to deep into it, I will just post the code and follow it up with some comments about choices made.

const _ = require("lodash");
const fs = require("fs");

const data = require("../data.json");
const { generateSchema } = require("../helpers");

async function main() {
  const openApi = {
    openapi: "3.0.3",
    info: {
      title: "Swyftx API",
      description: "Swyftx generated API from official Apiary website",
      version: "0.1.0",
    },
    paths: {},
    servers: [
      {
        url: "https://api.swyftx.com.au",
        description: "Production API URL",
      },
      {
        url: "https://api.demo.swyftx.com.au",
        description: "Demo API URL",
      },
    ],
    components: {
      securitySchemes: {
        bearerAuth: {
          type: "http",
          scheme: "bearer",
          bearerFormat: "JWT",
        },
      },
    },
    security: { bearerAuth: [] },
  };

  data.endpoints.map((endpoint) => {
    const summary = _.startCase(endpoint.path);
    const responses = {};

    // For each endpoint, we need to iterate through the possible responses.
    // Those possible responses could be 200, 500, 400, etc. This is also
    // available to use in from our scraped data in `data.json`.
    for (const response of endpoint.responses) {
      // console.log(endpoint);
      if (
        response.responseStatusValue === "204" ||
        !response.responseExampleValue
      ) {
        responses[response.responseStatusValue] = {
          description: response.responseStatusValue,
        };
      } else {
        responses[response.responseStatusValue] = {
          description: response.responseStatusValue,
          content: {
            "application/json": {
              schema: generateSchema(response.responseExampleValue),
            },
          },
        };
      }
    }

    // This itself could be cleaned up and abstracted but the gist of things is to set
    // params to `undefined` if there are no params (our default is an empty array in the data.json file).
    // If params do exist with a length > 0, then we have an inline map to return the expected
    // OpenAPI specification for `parameters` for that particular endpoint.
    const parameters =
      endpoint.parameters.length === 0
        ? undefined
        : endpoint.parameters.map((parameter) => {
            let [urlPath] = endpoint.url.split("?");

            const inValue = urlPath.includes(parameter.paramKeyValue)
              ? "path"
              : "query";

            const paramObj = {
              in: inValue,
              name: parameter.paramKeyValue,
              schema: {
                type: "string",
              },
              description: parameter.paramDescriptionValue,
            };

            if (inValue === "path") {
              paramObj.required = true;
              // parameter.paramRequirementValue === "Required";
            }

            return paramObj;
          });

    // If the request is POST or PUT, we need to add a body to the request.
    // We expect the content to always be `application/json`, so we can afford
    // to be a bit hacky.
    const requestBody = endpoint.request.requestExampleValue
      ? {
          content: {
            "application/json": {
              schema: generateSchema(endpoint.request.requestExampleValue),
            },
          },
        }
      : undefined;

    // Set the endpoint url. We need to do a quick check on the params.
    // If params exist, we need to add the params to the url.
    // We do this by interpolating the string correctly
    // to have params within curly braces ie. /path/to/{param}
    let endpointPath = endpoint.url;
    if (parameters) {
      let [urlPath, queryParams] = endpoint.url.split("?");

      for (const param of endpoint.parameters) {
        urlPath = urlPath.replace(
          param.paramKeyValue,
          `{${param.paramKeyValue}}`
        );
      }

      endpointPath = queryParams ? `${urlPath}?${queryParams}` : urlPath;
    }

    // Add in the summary and responses properties
    openApi.paths[endpointPath] = {
      [_.lowerCase(endpoint.method)]: {
        summary,
        responses,
      },
    };

    // Conditionally add the parameters property
    if (parameters) {
      openApi.paths[endpointPath][_.lowerCase(endpoint.method)].parameters =
        parameters;
    }

    // Conditionally add a requestBody property
    if (requestBody) {
      openApi.paths[endpointPath][_.lowerCase(endpoint.method)].requestBody =
        requestBody;
    }
  });

  fs.writeFileSync("./openapi.json", JSON.stringify(openApi, null, 2), "utf-8");
}

main();

As for the comments on the code above:

We import a generateSchema from the helpers.js file. We haven't written this yet, but I already noted that we will need to do some recursion and be smart about it based on the examples scraped from the website. This will be attacked in the next section.
I used some Lodash helpers just to handle some case changes around the place.
There are some complicated dynamic keys for the object being added here (like the endpointPath key). This is in line with the specification. Scroll to the bottom to see the end result example, but basically I am following along with OpenAPI spec.
In general, the OpenAPI spec can use references to components that are defined elsewhere. This is a good way to keep the spec clean and easy to read. I am opting to manually write all the definitions out manually (and repeated in the case of errors) for the sake of time. This is an improvement that could be made to the script.

At the end of the script, we have a openapi.json file that will become a valid OpenAPI spec. Before we can run it however, we need to write our generateSchema function.

Generating the schema from the scraped data

Our most difficult function in this example is a function that relies on recursion. What we want to do is take an example response and convert is into a format that describes the "type" of each key-value pair within the tree deeply.

To explain this via example, we want to take the following:

const input = {
  address: [
    {
      id: 1,
      code: "AUD",
      address_details: {
        address: "18PpTWTShSsHuJ5ze23gU2rmsQbAVVMZia",
        destination_tag: "654987312",
        payment_id: "654987312",
        memo: "memo text",
        biller_code: "321654",
        bsb: "654-312",
        payid: "email@address.example",
        reference: "654987312519",
        message: "b&2H_8s!...",
      },
      time: 1517316338030,
      name: "My BTC Address",
      type: "deposit",
    },
  ],
};

...and have it come out as the following:

const output = {
  type: "object",
  properties: {
    address: {
      type: "array",
      items: {
        type: "object",
        properties: {
          id: { type: "number" },
          code: { type: "string" },
          address_details: {
            type: "object",
            properties: {
              address: { type: "string" },
              destination_tag: { type: "string" },
              payment_id: { type: "string" },
              memo: { type: "string" },
              biller_code: { type: "string" },
              bsb: { type: "string" },
              payid: { type: "string" },
              reference: { type: "string" },
              message: { type: "string" },
            },
          },
          time: { type: "number" },
          name: { type: "string" },
          type: { type: "string" },
        },
      },
    },
  },
};

As you can see, it's not a simple one-for-one replacement, and so we need to grok the type as we iterate down into the object and sub-objects.

We can do this using some good old fashioned Test-Driven Development.

Create a new folder __tests__ and add the file __tests__/helpers.test.js. Inside of it, we can add the test based on the example we had above:

const { generateSchema } = require("../helpers");

describe("helper functions", () => {
  test("generateSchema returns a valid OpenAPI object", () => {
    const input = {
      address: [
        {
          id: 1,
          code: "AUD",
          address_details: {
            address: "18PpTWTShSsHuJ5ze23gU2rmsQbAVVMZia",
            destination_tag: "654987312",
            payment_id: "654987312",
            memo: "memo text",
            biller_code: "321654",
            bsb: "654-312",
            payid: "email@address.example",
            reference: "654987312519",
            message: "b&2H_8s!...",
          },
          time: 1517316338030,
          name: "My BTC Address",
          type: "deposit",
        },
      ],
    };

    const output = {
      type: "object",
      properties: {
        address: {
          type: "array",
          items: {
            type: "object",
            properties: {
              id: { type: "number" },
              code: { type: "string" },
              address_details: {
                type: "object",
                properties: {
                  address: { type: "string" },
                  destination_tag: { type: "string" },
                  payment_id: { type: "string" },
                  memo: { type: "string" },
                  biller_code: { type: "string" },
                  bsb: { type: "string" },
                  payid: { type: "string" },
                  reference: { type: "string" },
                  message: { type: "string" },
                },
              },
              time: { type: "number" },
              name: { type: "string" },
              type: { type: "string" },
            },
          },
        },
      },
    };

    const res = generateSchema(input);
    expect(res).toEqual(output);
  });
});

In the test, we are saying "Given the input, if I put that into the function generateSchema, then I expect to get back the equivalent of the output." - this will help us understand if we have written our function to convert out data structure in data.json into something expected by the OpenAPI spec.

The code itself is probably better explained, so inside of helpers.js, add the following:

/**
 * Determine the type of the `entry` arg input. If array, return array, else return the `typeof` result.
 *
 * @param {Object} entry - The variable we want to determine the type of.
 */
function determineType(entry) {
  return Array.isArray(entry) ? "array" : typeof entry;
}

/**
 * We want to be able to take an object and guess the schema
 * based on the input and return valid a openapi schema
 * @param {*} obj
 */
function generateSchema(obj) {
  const response = {};

  const type = determineType(obj);
  response.type = type;
  response.properties = {};

  if (!obj) {
    return undefined;
  }

  for (const [key, value] of Object.entries(obj)) {
    const type = determineType(value);
    response.properties[key] = {
      type,
    };

    if (type === "array") {
      // take first item from array to sample from
      response.properties[key].items = generateSchema(value[0]);
    } else if (type === "object") {
      response.properties[key] = generateSchema(value);
    }
  }

  return response;
}

module.exports = {
  generateSchema,
};

The code can be explained as the following:

Our "private" determineType function is an abstraction to return the type of the input.
generateSchema is an exported function that itself is recursive. If the type of the entry is an array or object, we invoke the recursion, otherwise we simply return the response.

To test out function out, we will need to adjust package.json to run jest from the test script.

Update package.json to look like the following (or at least the "scripts" object):

{
  "name": "swyftx-apiary-to-api",
  "version": "1.0.0",
  "description": "## Helper notes",
  "main": "index.js",
  "scripts": {
    "test": "jest"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "jest": "^27.2.3",
    "lodash": "^4.17.21",
    "puppeteer": "^10.4.0"
  }
}

Now we are ready to test our function. Run npm test:

$ npm test

> swyftx-apiary-to-api@1.0.0 test
> jest

 PASS  __test__/helpers.test.js
  helper functions
    ✓ generateSchema returns a valid OpenAPI object (2 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        0.397 s, estimated 1 s
Ran all test suites.

Success! Our function is running as expected.

Generating the spec

We are finally at the point where we let our script have at it.

Run node scripts/convert-data-to-openapi-format.js in the terminal.

Upon a successful run, we will now see a delightful 5700 lines of JSON in the openapi.json file output.

A short summary of the output:

{
  "openapi": "3.0.3",
  "info": {
    "title": "Swyftx API",
    "description": "Swyftx generated API from official Apiary website",
    "version": "0.1.0"
  },
  "paths": {
    "/auth/refresh/": {
      "post": {
        "summary": "",
        "responses": {
          "200": {
            "description": "200",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "properties": {
                    "accessToken": {
                      "type": "string"
                    },
                    "scope": {
                      "type": "string"
                    }
                  }
                }
              }
            }
          },
          "500": {
            "description": "500",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "properties": {
                    "error": {
                      "type": "object",
                      "properties": {
                        "error": {
                          "type": "string"
                        },
                        "message": {
                          "type": "string"
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        },
        "requestBody": {
          "content": {
            "application/json": {
              "schema": {
                "type": "object",
                "properties": {
                  "apiKey": {
                    "type": "string"
                  }
                }
              }
            }
          }
        }
      }
    }
    // ... omitted: a zillion other paths
  },
  "servers": [
    {
      "url": "https://api.swyftx.com.au",
      "description": "Production API URL"
    },
    {
      "url": "https://api.demo.swyftx.com.au",
      "description": "Demo API URL"
    }
  ],
  "components": {
    "securitySchemes": {
      "bearerAuth": {
        "type": "http",
        "scheme": "bearer",
        "bearerFormat": "JWT"
      }
    }
  },
  "security": {
    "bearerAuth": []
  }
}

Now that is a lot of JSON.

Validating the OpenAPI Spec

A delighted website is available for us to copy-paste our generated JSON in to the validate against the 3.0.3 spec.

Once there, select Validate text and paste the JSON in the text area. Once you select to validate, you'll see we are successful!

Validated spec

Import the OpenAPI Spec into Postman

Something I like to do for the sake of it is to import OpenAPI file into Postman and setup a list of the endpoints. As mentioned in a previous post, there may still be some endpoints that have an edge case and won't work correctly, but Postman is a great way for us to check against that.

This section requires the Postman app. If you don't have it, then you can just follow along.

Inside of postman, select Import and paste in the text.

Once you confirm, it will prompt you for the name of the name of the generated collection.

Confirming the import

Our API spec will give both examples of the responses where provided as well as the capability for us to make a request against the endpoints.

Example in Postman

As part of the spec we generated, there is a section that helps us tell Postman that bearer authentication is required:

{
  "components": {
    "securitySchemes": {
      "bearerAuth": {
        "type": "http",
        "scheme": "bearer",
        "bearerFormat": "JWT"
      }
    }
  },
  "security": {
    "bearerAuth": []
  }
}

If you edit the collection with your Swyftx access token, you can now make requests against the endpoints. Here is an example of getting my user data (with some redacted data):

Production /user endpoint

Note: In a hilarious turn of events, it actually looks like Apiary docs don't always have the updated response. Not sure what is going on there, but you'll notice the difference in responses from the actual API and the example response object. We will persist regardless, but I will have to tinker about and see what to do if we can't rely on the docs as the source of truth. I might email Swyftx.

Summary

In this post, we generated the OpenAPI spec for the Swyftx API based on the data that we scraped from the Apiary website using Puppeteer.

After generating the spec, we validated the spec against the OpenAPI 3.0.3 spec and then imported the spec to Postman to test out the live endpoints.

With our OpenAPI spec generated, we are now in a good place to build out our TypeScript API (sans the fact that the docs and actual response objects are different, ha!). In the next post, we will start on just that as well as clean up a few things around the repository.