fetch-paginate

Fetches multiple pages from paginated APIs with fetch (using either Link headers like GitHub, or with customizable page or offset & limit query parameters).

Also use to search a paginated API until you find your item (see Async Iterators or until option).

Supports TypeScript.
Isomorphic - works in Node and browser^*
Supports custom fetch wrappers for caching, etc.

* For environments without global fetch support, a polyfill is required. Recommend cross-fetch/polyfill, otherwise node-fetch (Node.js) or whatwg-fetch (browser). With node-fetch, add it globally:

global.fetch = require('node-fetch');

Usage

import { fetchPaginate } from "fetch-paginate";

const { items } = await fetchPaginate("https://api.example.com/foo");

Now items will be an array of items across all pages (unless you define a custom merge).

If the API returns your results array nested in the response, use a custom getItems function to select them:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  getItems: (body) => body.results,
});

If you need access to all the page bodies or entire response objects, use:

const { pages, responses } = await fetchPaginate("https://api.example.com/foo");

You can also specify the types of your objects with generics:

const { items, pages } = await fetchPaginate<MyBody, MyItem>(
  "https://api.example.com/foo"
);

// Now `items` has type `MyItem[]`,
// and `pages` has type `MyBody[]`.

fetchPaginate(url, options);

To use query parameters instead of Link headers, use the params option:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: true, // Assumes `page` parameter, where the first page is `1`.
});

You can customize the name of the page parameter, e.g., to pg, like:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: {
    page: 'pg', // Assuming our API expects `?pg=2`, for example.
  },
});

If your API uses offset and/or limit parameters instead of page parameter, you can do:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: {
    offset: true,
  },
});

To customize the name of the offset and/or limit parameters, you can do:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: {
    offset: 'start',
    limit: 'count',
  },
});

If your pages start at a different number, like 0, use the page option:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: true, // Assumes `page` parameter.
  page: 0,
});

If you want to start fetching at a later page, use the page option:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: true, // Assumes `page` parameter.
  page: 3, // Start at the 3rd page.
});

If you want to fetch with a specific number of items per page, instead of defaulting to the size of the first page (requested without limit), you can do:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: {
    offset: true,
  },
  limit: 10, // Fetch 10 items per page.
});

If you want to start fetching at a specific offset:

const { items } = await fetchPaginate("https://api.example.com/foo", {
  params: {
    offset: true,
  },
  offset: 17, // Start fetching at the 17th item.
});

Async Iterators

If you want to serially process each page, you can use fetchPaginateIterator, build on the async iterators (for await...of) API.

This also means you can use break and continue semantics - perhaps as an alterative to the until option.

import { fetchPaginateIterator } from "fetch-paginate";

const myIterator = fetchPaginateIterator("https://api.example.com/foo");

for await (const { pageItems } of myIterator) {
  console.log(pageItems);
}

For example, if you want to stop after finding a certain item:

let foundItem;
for await (const { pageItems } of myIterator) {
  foundItem = pageItems.find((item) => item.title.match(/Something/));
  if (foundItem) break;
}
console.log(foundItem);

You can also get each page body or entire response object:

for await (const { page, response } of myIterator) {
  console.log(page, response);
}

And also get the final result which has the same shape as fetchPaginate ({ items, pages, responses }):

for await (const { pageItems } of myIterator) {
  console.log(pageItems);
}

myIterator.getResult();

The iterator similarly supports TypeScript generics:

const myIterator = await fetchPaginateIterator<MyBody, MyItem>(
  "https://api.example.com/foo"
);

Custom Fetch

If you want custom fetching behavior like caching, you can provide a factory for a custom fetch-compatiable function. If you return undefined, it’ll fall back to global fetch:

await fetchPaginate("https://api.example.com/foo", {
  getFetch: ({ url, offset, page, fetchOptions, ...etc }) => async () => {
    const cached = await cache.get(url);
    if (cached) return new Response(cached.body, cached.init);
  },
});

Or you can resolve that fetch-like function asynchronously:

await fetchPaginate("https://api.example.com/foo", {
  getFetch: async ({ url }) => {
    const cached = await cache.get(url);
    if (cached) return async () => new Response(cached.body, cached.init);
  },
});

Browser

For bundled/browser use fetch-paginate/bundle (which includes dependencies, except fetch):

import "cross-fetch/polyfill";
import fetchPaginate from "fetch-paginate/bundle";

or even with the UMD global (on window):

import "cross-fetch/polyfill";
import "fetch-paginate/bundle";

const { items } = await fetchPaginate("https://api.example.com/foo");

Options

`getItems`

An optional function specifying how to get items list from a page of response body.

Defaults to identity:

(body) => body;

`merge`

An optional function specifying how to merge pages of items. Receives an array of arrays of items from each page (from getItems(await parse(response)) for each page).

Defaults to flatten arrays:

(setOfSetsOfItems) => setOfSetsOfItems.reduce((acc, v) => [...acc, ...v], []);

`parse`

An optional function specifying how to parse responses. Return a promise.

Defaults to parse JSON:

(response) =>
  response.ok && response.status !== 204 ? response.json() : response.text();

`until`

An optional function specifying when to stop paginating. Receives parsed body and whole response object. Return true to stop paginating, or a promise that resolves as such.

Defaults to always return false - to continue to consume until all pages:

({ page, pages, response, responses, items, pageItems }) => false;

`params`

boolean | ParamsObject

Optionally use these if the API paginates with query parameters (either page, or limit and offset), rather than Link headers.

If you pass params: true, it will use page as the default, instead of limit and offset.

`params.page`

string

The name of the query parameter to use for pages.

Defaults to "page".

`params.limit`

string | boolean

The name of the query parameter to use for limit per page.

If limit: true, it will indicate to use limit and offset instead of page, but use default names.

Defaults to "limit".

`params.offset`

string | boolean

The name of the query parameter to use for page offset.

If offset: true, it will indicate to use limit and offset instead of page, but use default names.

Defaults to "offset".

`page`

number

If using params with page, this indicates the page at which to start fetching.

Defaults to 1.

`offset`

number

If using params with offset and limit, this indicates the offset at which to start fetching.

Defaults to 0.

`limit`

number

If using params with offset and limit, this indicates the size of each page.

Defaults to the size of the first page fetched.

`fetchOptions`

ResponseInit (Object)

Additional options to pass to fetch.

`getFetch`

(args: FetchPaginateGetFetchArgs) => ( typeof fetch | Promise<typeof fetch> )

A factory that provides a fetch-compatible function. Use this, e.g., to define your own custom cache wrapper.