Pagination response payload from a RESTful API

I want to support pagination in my RESTful API.

My API method should return a JSON list of product via /products/index. However, there are potentially thousands of products, and I want to page through them, so my request should look something like this:

/products/index?page_number=5&page_size=20

But what does my JSON response need to look like? Would API consumers typically expect pagination meta data in the response? Or is only an array of products necessary? Why?

It looks like Twitter's API includes meta data: https://dev.twitter.com/docs/api/1/get/lists/members (see Example Request).

With meta data:

{
  "page_number": 5,
  "page_size": 20,
  "total_record_count": 521,
  "records": [
    {
      "id": 1,
      "name": "Widget #1"
    },
    {
      "id": 2,
      "name": "Widget #2"
    },
    {
      "id": 3,
      "name": "Widget #3"
    }
  ]
}

Just an array of products (no meta data):

[
  {
    "id": 1,
    "name": "Widget #1"
  },
  {
    "id": 2,
    "name": "Widget #2"
  },
  {
    "id": 3,
    "name": "Widget #3"
  }
]

ReSTful APIs are consumed primarily by other systems, which is why I put paging data in the response headers. However, some API consumers may not have direct access to the response headers, or may be building a UX over your API, so providing a way to retrieve (on demand) the metadata in the JSON response is a plus.

I believe your implementation should include machine-readable metadata as a default, and human-readable metadata when requested. The human-readable metadata could be returned with every request if you like or, preferably, on-demand via a query parameter, such as include=metadata or include_metadata=true.

In your particular scenario, I would include the URI for each product with the record. This makes it easy for the API consumer to create links to the individual products. I would also set some reasonable expectations as per the limits of my paging requests. Implementing and documenting default settings for page size is an acceptable practice. For example, GitHub's API sets the default page size to 30 records with a maximum of 100, plus sets a rate limit on the number of times you can query the API. If your API has a default page size, then the query string can just specify the page index.

In the human-readable scenario, when navigating to /products?page=5&per_page=20&include=metadata, the response could be:

{
  "_metadata": 
  {
      "page": 5,
      "per_page": 20,
      "page_count": 20,
      "total_count": 521,
      "Links": [
        {"self": "/products?page=5&per_page=20"},
        {"first": "/products?page=0&per_page=20"},
        {"previous": "/products?page=4&per_page=20"},
        {"next": "/products?page=6&per_page=20"},
        {"last": "/products?page=26&per_page=20"},
      ]
  },
  "records": [
    {
      "id": 1,
      "name": "Widget #1",
      "uri": "/products/1"
    },
    {
      "id": 2,
      "name": "Widget #2",
      "uri": "/products/2"
    },
    {
      "id": 3,
      "name": "Widget #3",
      "uri": "/products/3"
    }
  ]
}

For machine-readable metadata, I would add Link headers to the response:

Link: </products?page=5&perPage=20>;rel=self,</products?page=0&perPage=20>;rel=first,</products?page=4&perPage=20>;rel=previous,</products?page=6&perPage=20>;rel=next,</products?page=26&perPage=20>;rel=last

(the Link header value should be urlencoded)

...and possibly a custom total-count response header, if you so choose:

total-count: 521

The other paging data revealed in the human-centric metadata might be superfluous for machine-centric metadata, as the link headers let me know which page I am on and the number per page, and I can quickly retrieve the number of records in the array. Therefore, I would probably only create a header for the total count. You can always change your mind later and add more metadata.

As an aside, you may notice I removed /index from your URI. A generally accepted convention is to have your ReST endpoint expose collections. Having /index at the end muddies that up slightly.

These are just a few things I like to have when consuming/creating an API. Hope that helps!

As someone who has written several libraries for consuming REST services, let me give you the client perspective on why I think wrapping the result in metadata is the way to go:

Without the total count, how can the client know that it has not yet received everything there is and should continue paging through the result set? In a UI that didn't perform look ahead to the next page, in the worst case this might be represented as a Next/More link that didn't actually fetch any more data.
Including metadata in the response allows the client to track less state. Now I don't have to match up my REST request with the response, as the response contains the metadata necessary to reconstruct the request state (in this case the cursor into the dataset).
If the state is part of the response, I can perform multiple requests into the same dataset simultaneously, and I can handle the requests in any order they happen to arrive in which is not necessarily the order I made the requests in.

And a suggestion: Like the Twitter API, you should replace the page_number with a straight index/cursor. The reason is, the API allows the client to set the page size per-request. Is the returned page_number the number of pages the client has requested so far, or the number of the page given the last used page_size (almost certainly the later, but why not avoid such ambiguity altogether)?

Pagination response payload from a RESTful API

Related

Recent Posts