API Pagination in Copy Data pipeline activity

I am working with an API that does pagination via bookmarks. In the response JSON file there are 2 string value pairs that are used for identifying more records and the parameter to pass in the next request URL. The JSON is structured like this.

{

"Items": [

{

...

{

...

}

"Bookmark": "jobsuffixsite_ref<D><f>false</f><f>false</f><f>false</f></D><F><v> 1</v><v>0</v><v>BLGS</v></F><L><v> 783</v><v>0</v><v>BLGS</v></L>",

"MoreRowsExist": true,

"Success": true,

"Message": null

}

When setting up the Pagination rules in the Copy Data activity, how can I pass the value in the Bookmark back to the Relative URL?

My current Relative URL looks like this,

@concat('IDORequestService/ido/load/',item().source_table,'?filter=',item().FILTER,'&loadtype=NEXT&Properties=',item().cols,'&recordcap=',pipeline().parameters.recordcount,'&bookmark=')

I need to add the bookmark parameter at the very end.

In Azure Data Factory it is done via a parameter set in the Linked Service but since that doesn't exist in Fabric I'm stuck at getting this to work.

I can handle the pagination in a notebook easily, but I wanted to explore using a Data Pipeline to try a more Metadata driven approach.

10 comments