Pagination
Pagination in REST APIs refers to the process of dividing the large amount of data returned by an API into smaller chunks or pages, for better management and performance. It allows clients to retrieve a limited number of items from the API at a time, rather than retrieving the entire data set in a single request.
For example, an API may return a large number of items in a database, but the client can retrieve only a few items at a time, by specifying the page number and the page size in the API request. This way, the API can return the data in a more manageable and efficient manner, reducing the overhead on the client and the server.
Pagination is often indicated by the API using specific headers or query parameters, such as "Next-Page" or "Page-Size". Clients can use these parameters to retrieve different pages of data or to control the amount of data retrieved in a single request.
To set up pagination in the Advanced tab
Click on the Advanced tab.
Choose the Pagination option.
Enable the toggle button.
Provide a name for the pagination variable, which will be used in the URL as
http://example.com?page={{pagination_variable_name}}Select the type of pagination you prefer.
Enter the path to the array of results in the API response.
Types of pagination
There are multiple ways to paginate data in APIs. We'll look at two common methods: Auto-increment and Response-based pagination.
Auto increment
Auto-increment pagination is a common technique in REST APIs. In this method, data is returned in a series of pages. The client requests the next page by incrementing a specific parameter in the URL.
Example:
API Endpoint:
http://example.com/dataRequesting the first page (10 items per page):
http://example.com/data?page=1&page_size=10For the subsequent pages, the "page" parameter is incremented:
http://example.com/data?page=2&page_size=10This continues until all data is retrieved. The API might use headers or other methods to indicate when there's no more data available.
Note: The naming of parameters, like "page" and "page_size", can vary among APIs. Some might use "offset" or "start". Always refer to the specific API documentation for details.
Incorporation in iHub:
By adding the variable {{page}} into the URL, iHub can handle pagination in the background:http://example.com/data?page={{page}}&page_size=10
Breaking path: is the array where the data is located in the response.
Response based
In response-based pagination, the information about the next page is provided within the current page's response.
Example:
API Endpoint:
http://example.com/dataInitial request:
http://example.com/data?page_size=10The response in the execution log might look like this:
{
"data": [...],
"next_page_url": "http://example.com/data?page_size=10&page_token=abc123"
}
To fetch the next page, the client uses the "next_page_url" from the response. This continues until no "next_page_url" is present in the response, signifying all data has been fetched.
Note: The structure of the response can differ based on the API. Always refer to the specific API documentation for correct field names and structures.
Note the response data is accessed using $.data and headers using $.headers
This means that the data seen in the Execution Log on the Response tab is accessed by using $.data.next_page_url
If the API is paginated using the headers then use $.headers.next_page_url
Breaking path: is the array where the data is located in the response.
There is no need to use $.data as prefix on this because the this checks agains the data response.
Example of a API using href as next url
Scripted pagination
The custom pagination script must consistently return a value at the script's conclusion. This returned value represents the paginated data set.
Available Variables:
response: Stores the raw response data in plain text format.jsonResponse: Holds the parsed JSON object of the response. This variable remains empty if the response isn't in JSON format.getIssue(): This function retrieves the current issue from the script's context (if available).context: Provides access to context variables associated with the request.jsonContext: Offers the context variables in JSON object format, extracted from the request.YOUR_PAGINATION_VARIABLE_NAME: This variable contains the value from the previous script call, allowing you to increment the page counter within your script.path(jsonResponse, "$.path.in.json"): This JSONPath function helps extract specific values from the response's JSON structure. You can replace"$.path.in.json"with the actual path to your desired data point.
Key Points:
Ensure the script always returns a value representing the paginated data set.
Utilize the provided variables to access response data, context information, and facilitate pagination logic.
below return statement will check if the pagination variable name called page is defined, if not it starts with 0
next run it it will get assgined value 0 and therefore not undefined and will add 0+1 and return 1.
and next run it will then be 1 and do 1+1 return 2 etc.`
var defaultLink = '${oktaBaseUrl}?limit=${pageSize}';
if(response && response.headers && response.headers.link){
// Use regular expression to extract the URL with rel="next"
const match = /<([^>]+)>;\s*rel="next"/.exec(response.headers.link);
const nextUrl = match && match[1];
return nextUrl+'?limit=${pageSize}';
}