REST is Best?

Is the architectural style of the 90s web right for modern applications?

The philosopher Wilfred Sellars once described the goal of philosophy as being “to understand how things in the broadest possible sense of the term hang together in the broadest possible sense of the term”. When we talk about REST as an “architectural style” we have a similar if somewhat reduced sort of generality in mind. Within the context of client-server interactions over the web, how do we understand the “things” with which clients and servers are mutually concerned, and how can we make those things “hang together” so that they form both an information system (a coherent collection of things which can be known) and an operational domain (a coherent collection of things which can be done)?

The RESTful answers to these questions are as follows. A “thing” is a resource, presented by a server to clients. When we follow the REST architectural style, we decide on a set of resource-things, objects of mutual concern about which the client and server will exchange messages. The states of these things are not directly held in common by the client and server, as they would be if for example both had access to the same data store. Instead, the server owns and manages the resource, while its clients obtain views of the resource’s state, and request operations that will cause that state to be updated. There is an analogy to be drawn with encapsulation in object-oriented programming: clients of an object’s public interface do not have direct access to the object’s private state. The client and server instead communicate by means of hypermedia representations of resource state, conveyed via the hypertext transfer protocol (HTTP); hence the full name of which REST is the acronym, “REpresentational State Transfer”.

The “hypermedia” part here is interesting. The term was the subject of much excited discussion among media theorists in the 1990s; nowadays, the fact that media may have an address somewhere on the internet, and incorporate links to other media, is a widely understood and accepted feature of the way media in general work. In the REST architectural style it is significant in two ways: firstly because resources themselves are internet-addressable, and secondly because representations of resource state may include links to other resources. This gives rise to an approach to modelling interaction patterns spanning multiple operations known as HATEOAS (“hypermedia as the engine of application state”). I’m not going to talk much more about HATEOAS here, except to mention that workflows and state machines are a natural pairing and HATEOAS gives an interesting perspective on how a client may discover and operate a state machine whose definition and implementation are held on the other side of an abstraction boundary by the server.

The internet-addressability of resources is extremely significant regardless. It means that from the client’s point of view a resource is located in a somewhat virtualised space: between a given client and a given server there may be any number of relays such as firewalls, gateways, caches and proxies, but a given resource is addressable in the same way regardless of where it finally resides, through a Universal Resource Indicator (URI). The machinery which resolves requests addressed to URIs to procedures executed by actual computers is (mostly) invisible to the client, which knows only the domain to which the resource belongs, and the path to the resource within that domain. The domain part is resolved via DNS to an endpoint to which HTTP requests are sent. Resolving the path is an application-specific concern, and may include mechanisms like API gateways picking a service to route to based on a fragment of the path, or an application server picking a method to call based on a pattern matched by the path and the HTTP method used.

This is an important point about REST as an architectural style which is often missed when thinking solely in terms of service APIs and their implementation. If we take a “procedure-first” view of the system, in which the core elements are the procedures exposed by the service and REST is just a way of binding those procedures to paths and HTTP verbs so that they can be called remotely by an HTTP client, then we are looking down the wrong end of the telescope. The client does not POST data to a URI simply because that is the protocol by which it can cause a given procedure to be executed by a service, but because it wants the resource indicated by that URI to be modified in the way described by the body of the request. How the resource is represented internally by the owning system, and how updates to it are processed, are none of the client’s concern. This is a strong abstraction boundary.

A widespread approach in RESTful implementation is to couple entities across this boundary, mapping resources directly to database records and HTTP verbs to CRUD operations on those records. This is sometimes known as the “I can see your database from here” antipattern. This approach is unfortunately encouraged by the similarity between URI paths and filesystem paths, which suggests that resources might be organised hierarchically into collections like folders in a file system. POST to the collection URI /orders to create a new item in the collection; GET /orders to retrieve all the items in the collection; POST to an item by ID at /orders/{item_id} to update it, and so on.

This isn’t an inappropriate way of organising resources in many cases, but it can result in service implementations that are nothing but layers of duplication and data copying between a protocol handling layer (e.g. a Spring “controller” bean binding methods to URI path patterns) and a data access layer updating an RDBMS or key/value store. Although the layering provides some encapsulation and the ability to evolve the service implementation independently of the API, the deep conceptual coupling between the resource model addressed by the client and the structure of the underlying data store can be difficult to undo.

A deeper problem is that modern system architectures are often event-driven and/or asynchronous in ways that don’t align well with this version of REST as CRUD-over-HTTP. But there are other ways of modelling things - objects of mutual concern - as resources which may be a better fit.

Consider a RESTful API to a service which offers command queues for requesting operations asynchronously, a variety of query-optimised views for inspecting the current state of data in the system, and event feeds for observing externally-visible events to which a client might want to react. New commands are POSTed to a resource representing a command queue, receiving an HTTP 201 Created response with a Location header giving a link to a resource representing the pending command. The processing status of the command can then be queried by polling this resource; additionally, updates to the statuses of pending commands might be published to an event feed.

The query-optimised data views are obtained with suitably parameterised GET requests. We can see that the HTTP verb semantics line up quite well here with the CQRS (“command/query responsibility segregation”) approach taken by the target system. There is a slight mismatch concerning idempotency: for reliability reasons the semantics of the command queue should be that command submission is idempotent so that commands can be safely resubmitted if we don’t receive a response to our first submission. POST is defined as non-idempotent and not safely retryable. But that doesn’t mean we can’t handle POSTs idempotently if it’s the right thing to do: if the receiving system detects a duplicate command submission, it can return a 303 See Other response pointing to the already-created command.

The event feeds are more challenging, as HTTP is a request-response protocol driven by the client and doesn’t support server-initiated messaging. We can offer three ways of obtaining updates from a feed. The first is straightforward polling via GET, with a query parameter specifying an offset (for example, “give me all the events since timestamp T”). The second is to model subscriptions to feeds as resources in their own right, created by POSTing a webhook to which new events should be sent, and cancelled by DELETEing the subscription resource. Finally, a websocket connection can be created to enable full two-way communications between the client and server.

This arrangement of resources exposes the processing model of the underlying system somewhat: the client cannot be ignorant of the fact that commands are processed and events emitted asynchronously. Because state updates are carried out through commands, which are first-class objects in the resource model, updates to the state of other entities are not modelled as HTTP verbs targetted at resources representing the entities themselves. Have we forsaken the RESTfulness of the system by modelling it in this way? Not really: we are just making different choices about how the client sees the capabilities offered by the server. Other options are available. We could wrap asynchronous command queues with resources serviced by synchronous endpoints that block until a result is obtained and return it in the HTTP result body. (Whether this is a good idea or not is a fairly context-sensitive matter. Usually not.)

Recall that resources do not have to map directly to system-internal entities. It may be that the implementing system has a single command queue, but our resource model represents this as a collection of separate resources closely linked to the entities (or aggregate roots) that commands are addressed to: /orders/commands or even /orders/{order_id}/commands. It is also worth considering that some commands might affect multiple entities, for example a Terraform-style “interpret this description of the desired state of multiple resources, and update those resources so that they match the description”. What we are POSTing in this case is not an update to the fields of a record, as in a CRUD model, but a representation of some behaviour we would like the service to enact. The point here is that it is reasonable for a resource to represent a task list for a virtual agent which can act on our behalf within the bounds of the target system, or a set of reconciliation items to be checked and settled.

While I hope this is a plausible approach to crafting an API to a modern event-driven system while following a RESTful architectural style, it’s fair to ask whether REST is the approach that would have emerged if systems of this kind were the model, rather than the 1990s version of the World Wide Web (Roy Fielding published his dissertation defining REST in 2000). There does not seem to be the right combination of opportunity and urgency to drive the development from scratch of a new approach that might displace the incumbent. Whether developers are really following a RESTful architectural style or simply tunnelling JSON-encoded RPC requests over HTTP, they are often constrained by security and firewall policies to direct traffic to port 443. (It is not generally considered prudent to offer direct access from the public internet to one’s Kafka broker). But I also think that despite some quirks related to its fundamental signalling having been designed for web browser clients, REST remains an apt style to follow when client and server sit on opposite sides of a strong abstraction boundary.

REST may not always be the best fit for communications from one microservice to another within the same estate. Binary RPC protocols are faster and lighter on the network for direct cross-service signalling, and durable event broadcast across a common broker is an immensely powerful way of propagating state updates via a replicated commit log. But both of these mechanisms are geared towards communicating across bounded contexts within a common estate; they are more problematic at a full system boundary, not only because of the security and firewalling issues mentioned earlier but also because they inhabit the global semantic context (what Domain Driven Design calls the ubiquitous language) of the system. RESTful resource modelling is a way of defining common objects of concern between such contexts: between your organisation and its customers, for example. The level of indirection it provides (especially with HATEOAS in play) may seem cumbersome, but it enables the both the contractual stability and the degree of decoupling you need to offer services to the public web.