Oats — How We Learned to Stop Worrying and Love Types

We created our own library to facilitate calling and creating HTTP APIs while validating types at compile-time and runtime. In this blog post, I'll explain the situation that led us to need such a library and the problems we seek to solve with it. Then I’ll explain how you can use the library in your own projects.

How we got to where we are

Back in the early days of Smartly.io, we needed a service that allowed us to iterate and grow at an incredibly fast pace. We chose to go with a PHP monolith and MongoDB, and those served us quite well for a long time. When we were only a handful of developers working very closely together on a small set of features, things worked great, and it allowed us to reach the heights we were aiming for.

But as we grew, the number of developers and features also grew. What worked for a small group of developers no longer worked for several dozens working on multiple features simultaneously. Bugs started to appear faster than we could fix them, and development slowed down and became frustrating. Most of the bugs we faced were to do with data validation (or lack thereof). We didn’t have a clear understanding of our current data model, and the stored data was a hotchpotch of numerous versions and iterations of our back-end and Facebook data models. All this caused havoc, especially once model changes were necessary, for example, when Facebook changed their API.

It was clear we needed a new and less dynamic architecture, but one that would give us more safety and security when changing existing code and implementing new features. We started to chip away at the old monolith, creating microservices that better encapsulated specific parts of our tool, and using technologies better suited for them. With this physical separation between our features, teams could better organise themselves and take better ownership of their code. All this would allow us to continue growing at speed with fewer barriers and better infrastructure for our current needs.

The ads and creatives microservice

The maintenance of the part of our monolith that handled Facebook ads and creatives was getting increasingly more cumbersome to maintain. Bugs cropped up all the time, and whenever Facebook released an API update, the situation only deteriorated further. We needed a microservice that would act as a sort of interface between us and the Facebook API, performing data transformations back and forth, while also providing robust data validation both ways. It would help us fix—or at least reveal—several bugs in our system in one fell swoop.

We chose to write our new microservice in TypeScript, thanks to its satisfactory type system and because most of our engineers were experienced with NodeJS or JavaScript. We also chose OpenAPI, as a system-agnostic standard to define our data models. It would allow us to reuse the same definition between the old monolith and the new microservice whenever they interacted.

To avoid writing tonnes of boilerplate code and to provide out-of-the-box runtime type checking (which TypeScript doesn’t have), we decided to automate the generation of clients (TypeScript and PHP) and servers (TypeScript) based on the OpenAPI specification we defined. At first, we went with the existing 3rd party libraries, dtsgenerator and oas3-chow-chow, but they proved inadequate due to existing bugs and incompatibilities between them, resulting in us having to write a lot of code to work around those problems. We then decided to create our own library, which would give us total freedom to adapt to our specific use case and solve all the issues we were facing.

In comes Oats

This is how Oats (OpenApi-TypeScript) was born. It can generate clients (services that consume an API) and servers (services that provide an API) that neatly wrap OpenApi specifications into classes or objects. It generates a lot of boilerplate for both, as well as all the required runtime data validation. The client is built around the Axios HTTP library, and the server is built around the Koa library. For the client, Oats generates an object that contains all the routes as promise-returning functions, which validate, at runtime, both the input and output of the underlying HTTP call. For the server, it creates an Endpoints interface and adapter, which ensures all the endpoints expect and return the correct types at runtime.

How to start using Oats

To get started, you need an OpenAPI compliant specification of your server or client. In the example below, you can see one that defines an entity “Item” which has the properties “id” and “name”, and three endpoints, “POST /item”, “DELETE /item/{id}”, and “GET /item/{id}”, for creating, deleting, and fetching an “item” entity respectively.

Creating a server

To create a server, define the destination of the generated files:

This will generate two files, one containing the necessary types, such as the “Item”, request headers, parameters, and so on and another containing the server’s Endpoints interface. Now all that is left is implementing the actual server by creating an object with the Endpoints interface type, which enforces that all the Koa endpoints are implemented correctly, receiving and returning the correct types.

The “Item.make” method is used for runtime validation of our types. It throws errors if required properties are missing, or are the wrong type, or if there are extra unexpected properties (although the latter can be configured to simply strip those properties away instead). We chose to throw validation errors instead of simply logging them as this will increase visibility, and prompt us to fix them faster. Nowadays, we rarely get errors related to invalid data, which means we have a high degree of confidence in what the data we’re dealing with looks like, even though it’s a third-party API.

Creating a client

Creating a client is very similar to creating a server. Again, you need the OpenAPI specification for the server which you’ll consume. Similarly, you have to define the destination of the generated files in your build step. The result will, again, be two files, one with the types, and another with the server’s endpoints and a consumer object, which will contain the methods mapped to the API calls of the server.

After that, performing API calls to the server is as easy as calling a Promise:

And as with the server, runtime validation of the data types returned by the server is performed, ensuring all the fields are correct.

Conclusion

In short, I hope what you get out of this blog post is just how important types can be for the long term growth and success of your projects. Many companies like Dropbox and Facebook have faced bugs and decreased productivity, which they attribute to the use of untyped languages. Dealing with all the boilerplate that comes with them may impede the iteration of your project. Libraries such as Oats enabled us to continue growing as fast as we needed, but in a sustainable manner, and we hope that they might do the same for you.

Getting started should be easy and straightforward if you follow our guide. Having types might actually save you time, as we described in a separate blog post having types enabled us to write fewer tests, which considerably increased how fast we could implement or overhaul existing features.

We’re more than happy to accept your contributions or answer your questions in our public repository, so please send issues and PRs!