Need a Help about dynamic data paging · AI Automation Society

Need a Help about dynamic data paging

A requirement for performing data manipulation based on the request's return value (dynamic data paging):

Because the data is returned in pages, the first HTTP request sent has cursor = 0 and request_id = "". If there is more information in the returned information, hasmore = 1, along with the corresponding cursor (8, 18, or other) and request_id. This needs to be added to the HTTP request, and the loop continues until hasmore = 0, indicating no more information.

In my current process, the code1 node is used to extract the cursor, requestid, and hasmore; the splitout node is used to return the returned information. The multiple records in the information are split to facilitate insertion into the subsequent loop node for Google Sheet record insertion.

Code 2 extracts the cursor, requestID, and HasMore nodes and places them in the root node. This allows the subsequent if node to determine whether to end the request or proceed to the next one based on HasMore. The edit node also sets the required request parameters for use in the HTTPRequest node.

The current problem I'm encountering is:

The loop node continuously accumulates requests, unlike my expectation where each batch of data loops and then initiates another HTTPRequest to request new data.

How can I design a solution to meet this requirement?

0 comments