The Power of HTTP for REST APIs — PART 2

Hypermedia Links: It’s All Relative

A hypermedia API is one driven by self-descriptive links that point to other, related API endpoints. Often, these links point to other resources that are related, e.g. the owner of a project, or to relevant endpoints based on the context of the consumer. To take advantage of hypermedia links, we build upon the core principles of HTTP by assigning unique URLs to our resources.

Below is a Github example of embedding hypermedia links within a resource representation:

GET https://api.github.com/users/launchany
{
  “login”: “launchany”,
  “id”: 17768866,
 "avatar_url”: “https://avatars3.githubusercontent.com/u/17768866?v=3",
  “gravatar_id”: “”,
 “url”: “https://api.github.com/users/launchany",
 “html_url”: “https://github.com/launchany",
 “followers_url”:  “https://api.github.com/users/launchany/followers",
 “following_url”:
 “https://api.github.com/users/launchany/following{/other_user}",
 “gists_url”:   “https://api.github.com/users/launchany/gists{/gist_id}",

  …
}

The links provided in this example response help the client navigate the API as it needs further details about the user, e.g. the user’s gravitar image. The client doesn’t need to be concerned about where images are hosted and if the image hosting service changes over time — it simply uses the provided URL when displaying the gravitar. Our API clients become more resilient to change and also benefit from no longer needing to hand-craft URLs.

APIs that need to support pagination are commonly designed using offset and limit parameters to request subsequent pages after the first set of results are returned. For data sets that change often, a better approach may be to use a cursor to avoid skipping or duplicating results. Our pagination strategy may need to change and we don’t want to cause our API clients to suddenly break if we change our pagination strategy.

How do we design a pagination approach that works for both of these cases and perhaps others not currently known? We can use hypermedia links once again, but this time to guide API clients on navigating through results, not just referencing external images or other parts of the API.

Let’s look at a hypermedia response example:

{
   “_links”: { 
   “self”: {“href”: “/projects” }, 
  “curies”: [{“name”: “cofrel”, “href”:   “http://api.example.com/hypermedia/rels/{rel}", “templated”: true }],
  “next”: {“href”: “/projects?since=d266f6cd-fddf-41d8–906f355cbecfb2de&maxResults=20” }, 
  “prev”: {“href”: “/projects?since=43be807d-d518–41f3–9206- e43b5a8f0928&maxResults=20” }, 
  “first”: {“href”: “/projects?since=ef24266a-13b3–4730–8a79- ab9647173873&maxResults=20” }, 
  “last”: {“href”: “/projects?since=4e8c74be-0e99–4cb8-a473- 896884be11c8&maxResults=20” }, 
  “cofrel:find”: { “href”: “/orders{?id}”, “templated”: true }, 
 }, 
 “currentlyActive”: 4, 
 “currentlyArchived”: 24
 }

With our navigational hypermedia links, API clients are able to follow the results in any direction through the results. They do not have to be concerned about the pagination style and how to compose the URL properly or if the style has changed. This is extremely powerful and even mimics how we use the web today when we perform a Google or Bing search.

This approach to managing pagination has the added benefit of conveying server-side state to the client as well. If we are on the first page of results, the ‘prev’ link won’t be provided and the UI can reflect that with a disabled ‘prev’ link. Similarly, if we are on the last page of results, the ‘next’ link won’t be provided. This is an application of the HATEOAS constraint.

HATEOAS (“Hypermedia As The Engine Of Application State”) is a constraint within REST that originated in Fielding’s dissertation. The primary advantage of HATEOAS is to avoid sending boolean fields or state-related fields that require the client to interpret them and decide what action(s) can be taken next. Instead, the server determines this ahead of time and conveys what can and cannot be done by the presence or absence of the links provided.

Summary: Hypermedia and HATEOAS are powered by URLs assigned to resources within an API. These links can convey state to clients, informing users what can and cannot be done at a particular time, based on their permissions and the state of the data. It is the the use of HTTP and URLs that allow us to build evolvable applications through hypermedia and HATEOAS.

Client-Side Caching: Improving App Performance

A cache is a local store of data to prevent re-retrieval of the data in the future. Developers familiar with the term have likely used server-side caching use tools such as memcached to keep data in memory and reduce the need to fetch unchanged data from a database to improve application performance.

HTTP cache semantics allow for cacheable responses to be stored locally by clients, moving the cache away from the server-side and closer to the client for better performance and reduced network dependence.

To support this, HTTP makes available several caching options through the Cache-Control response header that defines if the response is cacheable and, if so, for how long. Responses may only be cached if the HTTP method is GET or HEAD and the proper Cache-Control header indicates the content is cacheable.

Let’s re-examine our content negotiation example request:

GET https://api.example.com/projects HTTP/1.0
Accept: application/json;q=0.5,application/xml;q=1.0

Below is an example response that includes a caching directive from the API server:

HTTP/1.0 200 OK
Date: Tue, 16 June 2015 06:57:43 GMT
Content-Type: application/xml
Cache-Control: max-age=240

<project>…</project>

In this example, the max age indicates that the data may be cached for up to 240 seconds (4 min) before the client should consider the data stale.

The Google Developer website has a great article by Ilya Grigorik that details the client-server interactions of HTTP caching.

Summary: HTTP provides a cache-control header that informs clients if a response is cacheable and for how long. Applying this header to our API client code will reduce network traffic and speed up our web and mobile applications. Thoughtful caching design is required to take advantage of these capabilities offered by HTTP, including what resources are cacheable and for how long.

Intermediary Caching: Reducing Network Latency

The HTTP spec defines support for intermediaries, allowing requests to be processed by a chain of connections. These intermediaries are placed between the client and API server as needed, enabling network behavior be added without the APIs knowledge. Examples of intermediary usage include:

• A reverse proxy/gateway used for routing to services inside the firewall and enforcing security/rate limiting.
• A web application firewall to protect against common attack vectors, such as XML parser attacks, SQL injection, etc.
• Generate metrics and analytics of incoming requests to provide teams with insights into what endpoints are used, if any are returning excessive error response codes, etc.
• Caching servers that can return responses back to the client quickly and without involving a backend service, such as those provided by Varnishand nginx+Redis.

The last point listed above, caching servers, is powerful. They sit in front of an API that provides Cache-Control response directives and store the responses on behalf of an API server. This frees API clients from implement caching support while providing transparent app acceleration when placed in front of an API backend.

The further away the caching server is from the client application, the more time it will take to make a round-trip API request. This is where content distribution networks (“CDNs”) help, as they place caching servers all around the world to be closer to client applications. Let’s look at an example:

A mobile application connecting from London to an API server in Australia may transparently connect through a CDN edge node and to the remote API server on the first request. The CDN edge node then caches the API server response and returns the result back to the API client. Subsequent requests will be serviced by the CDN edge node that is closer to the mobile app — until the cache expiration time has been exceeded and the CDN edge node needs to refresh its cache from the API server.

Fastly has published a nice article on API caching that provides a detailed example of how it works, along with some of the backend cache control techniques used to invalidate CDN caches when backend data has changed.

It is important to note that when using caching servers, the client application still must make a network connection to the caching server to refresh data. To avoid the network round trip completely, client-side caching may be used to store the data within the application itself. In combination, we reduce network round trips when they are unnecessary, but benefit from caching servers when we need to refresh our client-side cache.

Summary: HTTP supports intermediaries between an API client and server. They understand the HTTP protocol and can assess incoming requests for cacheability without the need to parse or understand the specific request/response payload.

Conditional Requests: Staying Up-to-Date

Conditional requests is a lesser known but powerful capability offered by HTTP. Conditional requests allow clients to request an updated resource representation only if something has changed. Clients that send a conditional request will either receive a 304 Not Modified if the content has not changed, or 200 OK along with the changed content. There are two options for telling the server about the client’s local cached copy for comparison: eTags and time-based.

The entity tag, or “eTag”, is an opaque value that represents the current resource state. The client may store the eTag after a GET, POST, or PUT request and use the value to check for changes to the representation in the future via a HEAD request. Commonly, the eTag is a hashed value of the state, although this is not a requirement. All that is required is a way for the server generate a unique eTag value that can be used to determine if the state has been modified since it was last retrieved:

200 OK
Location: /projects/12345
Cache-Control: public, max-age=31536000
ETag: “17f0fff99ed5aae4edffdd6496d7131f”

The client may then use the If-None-Match request header to indicate the last eTag received:

GET /projects/12345
If-None-Match: “17f0fff99ed5aae4edffdd6496d7131f”

Alternatively, we can use time-based preconditions with the Last-Modifiedresponse header. The If-Modified-Since request header can then be used to specify the last updated timestamp to compare against the last update timestamp on the server to see if anything has changed:

200 OK
Location: /projects/12345
Cache-Control: public, max-age=31536000
Last-Modified: Mon, 19 Mar 2018 17:45:57 GMT

When the API client sends a GET request, it includes the last modified timestamp as part of the conditional request:

GET /projects/12345
If-Modified-Since: Mon, 19 Mar 2018 17:45:57 GMT

If the resource hasn’t changed since Mon, 19 Mar 2018 17:45:57 GMT, then the client will receive a 304 Not Modified rather than a 200 OK with the latest resource representation.

Summary: Conditional requests reduce the effort required to validate and re-fetch cached resources. eTags are opaque values that represent the current internal state, while last modified timestamps may be used for date-based comparison rather than eTags. We use the appropriate precondition request header to inform the server of the version on the client. The API server then returns the latest representation of the resource, or a 304 Not Modified if nothing has changed since the last fetch.

Concurrency Control: Protecting Resource Integrity

Conditional requests are also used to support concurrency control. By combining eTags or last modified dates with state change methods (e.g. PUT), we can ensure that data is not overwritten accidentally by another API client. This is especially important in the case of a PUT method, where the entire resource representation is replaced by a new representation provided by the client.

When an API client issues a modifying request, they may add a precondition to the request to prevent modification if the eTag has changed (via the If-Match request header) or if the timestamp has not changed (via the If-Unmodified-Since request header). Should the precondition fail, a 412 Precondition Failed response is sent by the server. Servers may also enforce the requirement of a precondition header to enforce concurrency control by responding with a 428 Precondition Required if neither of these request headers were found.

Let’s take an example where two API clients are trying to modify a project. First, each client retrieves the client using a GET request, obtaining the following response:

Location: /projects/12345
Cache-Control: public, max-age=31536000
ETag: “27f0fff99ed5aae4edffdd6496d7131f”

{…}

The first API client then modifies the project by replacing the current representation with a new one:

PUT /projects/1234
If-Match: “27f0fff99ed5aae4edffdd6496d7131f”

{ “name”:”Project 1234", “Description”:”My project” }

The server responds with a 200 OK, since the eTag matches and returns an updated representation with a new eTag:

200 OK
Location: /projects/12345
Cache-Control: public, max-age=31536000
ETag: “57f0fff99ed5aae4edffdd6496d7131f”

{…}

The second API client, which has the same eTag when it originally fetched the project, is trying to modify the project as well:

PUT /projects/1234
If-Match: “27f0fff99ed5aae4edffdd6496d7131f”

{ “name”:”Project ABCDE”, “Description”:”My renamed project” }

Unfortunately, the project has changed so the second API client receives a different response:

412 Precondition Failed

The second API client must now re-fetch the current representation of the resource instance, then inform the user of the changes and allow them to determine if they wish to re-submit the changes made or leave it as-is.

Summary: Concurrency control may be added to an API through HTTP preconditions in the request header. If the eTag/last modified date hasn’t changed, then the request is processed normally. If it has changed, a 412 response code is returned, preventing the client from overwriting data as a result of two separate clients modifying the same resource concurrently.

Wrap-up PART 2

In this two-part series we have only examined a small portion of what is useful for our APIs from the HTTP specification. With just these items in hand, we start to realize that HTTP is a robust protocol that supports a variety of our needs. By applying these techniques, we can build robust APIs that drive complex applications that are both resilient and evolvable.

Special thanks to Darrel Miller for reviewing this article.

The links provided in this example response help the client navigate the API as it needs further details about the user, e.g. the user’s gravitar image. The client doesn’t need to be concerned about where images are hosted and if the image hosting service changes over time — it simply uses the provided URL when displaying the gravitar. Our API clients become more resilient to change and also benefit from no longer needing to hand-craft URLs.

APIs that need to support pagination are commonly designed using offset and limit parameters to request subsequent pages after the first set of results are returned. For data sets that change often, a better approach may be to use a cursor to avoid skipping or duplicating results. Our pagination strategy may need to change and we don’t want to cause our API clients to suddenly break if we change our pagination strategy.

How do we design a pagination approach that works for both of these cases and perhaps others not currently known? We can use hypermedia links once again, but this time to guide API clients on navigating through results, not just referencing external images or other parts of the API.

Let’s look at a hypermedia response example:

With our navigational hypermedia links, API clients are able to follow the results in any direction through the results. They do not have to be concerned about the pagination style and how to compose the URL properly or if the style has changed. This is extremely powerful and even mimics how we use the web today when we perform a Google or Bing search.

This approach to managing pagination has the added benefit of conveying server-side state to the client as well. If we are on the first page of results, the ‘prev’ link won’t be provided and the UI can reflect that with a disabled ‘prev’ link. Similarly, if we are on the last page of results, the ‘next’ link won’t be provided. This is an application of the HATEOAS constraint.

HATEOAS (“Hypermedia As The Engine Of Application State”) is a constraint within REST that originated in Fielding’s dissertation. The primary advantage of HATEOAS is to avoid sending boolean fields or state-related fields that require the client to interpret them and decide what action(s) can be taken next. Instead, the server determines this ahead of time and conveys what can and cannot be done by the presence or absence of the links provided.

Client-Side Caching: Improving App Performance

A cache is a local store of data to prevent re-retrieval of the data in the future. Developers familiar with the term have likely used server-side caching use tools such as memcached to keep data in memory and reduce the need to fetch unchanged data from a database to improve application performance.

HTTP cache semantics allow for cacheable responses to be stored locally by clients, moving the cache away from the server-side and closer to the client for better performance and reduced network dependence.

To support this, HTTP makes available several caching options through the Cache-Control response header that defines if the response is cacheable and, if so, for how long. Responses may only be cached if the HTTP method is GET or HEAD and the proper Cache-Control header indicates the content is cacheable.

Let’s re-examine our content negotiation example request:

Below is an example response that includes a caching directive from the API server:

In this example, the max age indicates that the data may be cached for up to 240 seconds (4 min) before the client should consider the data stale.

The Google Developer website has a great article by Ilya Grigorik that details the client-server interactions of HTTP caching.

Intermediary Caching: Reducing Network Latency

The HTTP spec defines support for intermediaries, allowing requests to be processed by a chain of connections. These intermediaries are placed between the client and API server as needed, enabling network behavior be added without the APIs knowledge. Examples of intermediary usage include:

• A reverse proxy/gateway used for routing to services inside the firewall and enforcing security/rate limiting.
• A web application firewall to protect against common attack vectors, such as XML parser attacks, SQL injection, etc.
• Generate metrics and analytics of incoming requests to provide teams with insights into what endpoints are used, if any are returning excessive error response codes, etc.
• Caching servers that can return responses back to the client quickly and without involving a backend service, such as those provided by Varnishand nginx+Redis.

The last point listed above, caching servers, is powerful. They sit in front of an API that provides Cache-Control response directives and store the responses on behalf of an API server. This frees API clients from implement caching support while providing transparent app acceleration when placed in front of an API backend.

The further away the caching server is from the client application, the more time it will take to make a round-trip API request. This is where content distribution networks (“CDNs”) help, as they place caching servers all around the world to be closer to client applications. Let’s look at an example:

A mobile application connecting from London to an API server in Australia may transparently connect through a CDN edge node and to the remote API server on the first request. The CDN edge node then caches the API server response and returns the result back to the API client. Subsequent requests will be serviced by the CDN edge node that is closer to the mobile app — until the cache expiration time has been exceeded and the CDN edge node needs to refresh its cache from the API server.

Fastly has published a nice article on API caching that provides a detailed example of how it works, along with some of the backend cache control techniques used to invalidate CDN caches when backend data has changed.

It is important to note that when using caching servers, the client application still must make a network connection to the caching server to refresh data. To avoid the network round trip completely, client-side caching may be used to store the data within the application itself. In combination, we reduce network round trips when they are unnecessary, but benefit from caching servers when we need to refresh our client-side cache.

Conditional Requests: Staying Up-to-Date

Conditional requests is a lesser known but powerful capability offered by HTTP. Conditional requests allow clients to request an updated resource representation only if something has changed. Clients that send a conditional request will either receive a 304 Not Modified if the content has not changed, or 200 OK along with the changed content. There are two options for telling the server about the client’s local cached copy for comparison: eTags and time-based.

The entity tag, or “eTag”, is an opaque value that represents the current resource state. The client may store the eTag after a GET, POST, or PUT request and use the value to check for changes to the representation in the future via a HEAD request. Commonly, the eTag is a hashed value of the state, although this is not a requirement. All that is required is a way for the server generate a unique eTag value that can be used to determine if the state has been modified since it was last retrieved:

The client may then use the If-None-Match request header to indicate the last eTag received:

Alternatively, we can use time-based preconditions with the Last-Modifiedresponse header. The If-Modified-Since request header can then be used to specify the last updated timestamp to compare against the last update timestamp on the server to see if anything has changed:

When the API client sends a GET request, it includes the last modified timestamp as part of the conditional request:

If the resource hasn’t changed since Mon, 19 Mar 2018 17:45:57 GMT, then the client will receive a 304 Not Modified rather than a 200 OK with the latest resource representation.

Concurrency Control: Protecting Resource Integrity

Conditional requests are also used to support concurrency control. By combining eTags or last modified dates with state change methods (e.g. PUT), we can ensure that data is not overwritten accidentally by another API client. This is especially important in the case of a PUT method, where the entire resource representation is replaced by a new representation provided by the client.

When an API client issues a modifying request, they may add a precondition to the request to prevent modification if the eTag has changed (via the If-Match request header) or if the timestamp has not changed (via the If-Unmodified-Since request header). Should the precondition fail, a 412 Precondition Failed response is sent by the server. Servers may also enforce the requirement of a precondition header to enforce concurrency control by responding with a 428 Precondition Required if neither of these request headers were found.

Let’s take an example where two API clients are trying to modify a project. First, each client retrieves the client using a GET request, obtaining the following response:

The first API client then modifies the project by replacing the current representation with a new one:

The server responds with a 200 OK, since the eTag matches and returns an updated representation with a new eTag:

The second API client, which has the same eTag when it originally fetched the project, is trying to modify the project as well:

Unfortunately, the project has changed so the second API client receives a different response:

412 Precondition Failed

The second API client must now re-fetch the current representation of the resource instance, then inform the user of the changes and allow them to determine if they wish to re-submit the changes made or leave it as-is.

Wrap-up PART 2

In this two-part series we have only examined a small portion of what is useful for our APIs from the HTTP specification. With just these items in hand, we start to realize that HTTP is a robust protocol that supports a variety of our needs. By applying these techniques, we can build robust APIs that drive complex applications that are both resilient and evolvable.

Special thanks to Darrel Miller for reviewing this article.