Fighting PostgreSQL write amplification with HOT updates
Up to this point, this blog has mostly focused on how we have solved internal challenges including scaling our database infrastructure to handle increasing volume, and managing our workflow in a growing development team.
But as it is APIs that provide access to the Adyen payments platform, in this post I will discuss some recent initiatives to make it easier for developers to use them. Specifically, we will look at how we are leveraging the OpenAPI Specification (OAS) to:
In January this year, the Programmable Web had more than 19,000 APIs in its directory — with over 2,000 being added on average per year from 2014 onwards. New regulations, such as PSD2 in Europe, mean the growth in the number of APIs is set to continue.
An API specification is a document that defines and describes an API (see here for an in-depth definition, and here for an overview of some of the most commonly used specs). It is stored in a machine-readable form such as JSON or a YAML file, and can be converted to human-readable API documentation. As the number of APIs continues to rise, the lack of global API standard risks slowing down innovation and limiting collaboration.
We felt it would be valuable to work with a specification to automatically generate and improve our documentation, as this saves time and improves consistency. Also, it has the potential to enhance the developer experience as it makes it easier to explore and try out an API.
A few years ago, we adopted the Swagger specification (the predecessor of OpenAPI) v1.2 for generating our API reference and used it to create an interactive API explorer. Because of the popularity of this tool, we recently sat down to see how we could take this to the next level with the following broad criteria in mind:
We decided to go with OAS due to significant industry adoption, a wide range of open-source and commercial tools, and an active community contributing to the spec design.
We were impressed by the maturity and flexibility of the version 3.0, released last summer. With this version, the standard has a number of improvements, including:
All these improvements were necessary if we were to correctly describe our existing APIs, and with the constant updates to the OpenAPI standard and tooling around it we saw a good opportunity for us to be able to use it in a long term.
By working with OAS, we opened up the possibility of having the API Reference documentation generated from source code, meaning the documentation is always accurate, up-to-date, and supports versioning. This can be done in two ways: either with the Swagger UI, or with a custom implementation.
For us, it was essential to provide users a custom experience when exploring our APIs. Given the somewhat limited possibilities for customizing the Swagger UI, we opted to create our own implementation with additional requirements to the API Reference. Some of the requirements we had included:
Content. Automatically generated documentation often provides very limited descriptions. We wanted content in our API Reference to be as useful as in any other part of the documentation portal. Luckily, OpenAPI supports CommonMark, which means that you can format and interlink this documentation where necessary.
To benefit from this support, our technical writers work alongside API developers to make sure that our API Reference contains all necessary details and is consistent with all other parts of the documentation portal.
Interactivity and code examples. Another common practice for API Reference portals is to provide a sandbox where a developer can try out their requests to the API. In our portal, we decided to take this a step further, and filled the documentation with predefined examples, which a developer can use to perform certain tasks. These examples, with the ability to submit any custom additional request, make it much easier for a developer to learn about the API without leaving the documentation portal.
Design. We want to provide consistent experience for developers, regardless of whether they are browsing the main documentation portal or exploring our API Reference. With our custom solution, we have full control on the design and can also make sure that all parts of our documentation are properly linked to one other.
Analytics. Good UX requires data, which is not available by default through the Swagger UI. With building our own API Reference portal, we had an opportunity to test any single element we want and to get analytics reflecting developer experience both in API Explorer and on the whole documentation portal in general.
Security. When developers interact with our API, either in their personal sandbox or the documentation portal like API Explorer, we want to make sure that we provide the highest possible security level.
The result of combining all these elements is our recently-launched API Explorer portal. This portal is powered by OAS specifications but provides an information-rich experience in line with the rest of our documentation.
Our API Explorer 🛫
At this point, our API Explorer UI is somewhat tailored to our specific needs. However, we may consider open sourcing it in the future as it matures.
In addition to up-to-date and feature-rich API documentation, we decided to open source our API specifications files. Sharing our specs on GitHub makes them public and accessible to developers, and provides a place to ask questions and make suggestions.
The specifications in this repository are organized into two subfolders:
/2.0 — Spec files in the OpenAPI 2.0 format.
/3.0 — Spec files in the OpenAPI 3.0.0 format.
Each subfolder contains specifications files in both .json and .yaml formats for most of our APIs, per their version numbers.
In addition to standard OAS properties, our specifications contain some custom vendor extensions, which are supported by OAS. We find this extensibility very useful, because it provides the freedom to store additional metadata about your APIs within the same specification files. In our case, we store custom information, which is required to properly display API endpoints in our API Explorer:
This extension provides a list of all endpoint groups for the selected API. For example:
These extensions specify how to group endpoints and sort them within a group. For example:
Another way API specification files can add value to the developer experience is by loading them into REST clients such as Postman or Insomnia. This enables developers to send API requests, explore responses, and create and share their personal collections of endpoints and examples for their favourite APIs.
For example, Postman can import OAS specifications and therefore prepare the environment for exploring your API. To try how it works, you can download Adyen’s OAS 2.0 files from the GitHub repository and then import them into Postman:
Postman then creates a new collection of endpoints, which your API users can expand and share with other members of the community, thus helping you market your API as well.
API specifications such as OAS can greatly improve the learning experience and expand the ways developers can explore your APIs.
The first and most obvious win is auto-generated documentation, which is therefore always accurate and up-to-date. But with OAS you can go even beyond and think about alternative ways for your users to learn about your APIs — such as facilitating the dialogue about your APIs by sharing the specifications and making sure that your APIs are easily accessible through popular API tools.
Are you using API specifications? Did you find interesting ways to share knowledge about your APIs through specification files? Let us know your thoughts and experiences below!
By submitting this form, you acknowledge that you have reviewed the terms of our Privacy Statement and consent to the use of data in accordance therewith.