Activity
Mon
Wed
Fri
Sun
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
What is this?
Less
More

Memberships

Learn Microsoft Fabric

14.3k members โ€ข Free

2 contributions to Learn Microsoft Fabric
API Pagination in Copy Data pipeline activity
I am working with an API that does pagination via bookmarks. In the response JSON file there are 2 string value pairs that are used for identifying more records and the parameter to pass in the next request URL. The JSON is structured like this. { "Items": [ { ... }, { ... } ], "Bookmark": "<B><P><p>job</p><p>suffix</p><p>site_ref</p></P><D><f>false</f><f>false</f><f>false</f></D><F><v> 1</v><v>0</v><v>BLGS</v></F><L><v> 783</v><v>0</v><v>BLGS</v></L></B>", "MoreRowsExist": true, "Success": true, "Message": null } When setting up the Pagination rules in the Copy Data activity, how can I pass the value in the Bookmark back to the Relative URL? My current Relative URL looks like this, @concat('IDORequestService/ido/load/',item().source_table,'?filter=',item().FILTER,'&loadtype=NEXT&Properties=',item().cols,'&recordcap=',pipeline().parameters.recordcount,'&bookmark=') I need to add the bookmark parameter at the very end. In Azure Data Factory it is done via a parameter set in the Linked Service but since that doesn't exist in Fabric I'm stuck at getting this to work. I can handle the pagination in a notebook easily, but I wanted to explore using a Data Pipeline to try a more Metadata driven approach.
API Pagination in Copy Data pipeline activity
0 likes โ€ข Jul '24
@Will Needham, ok I'll try that and post back. Thank you!
0 likes โ€ข Jul '24
@Will Needham, Thanks again for pointing me in the right direction! I was able to accomplish what I needed with this pipeline
๐Ÿ‘‹ New joiner? Welcome! Start here ๐Ÿ‘‡
Welcome to all new members, here's some links and information to help you get started! ๐—ค๐˜‚๐—ถ๐—ฐ๐—ธ ๐—Ÿ๐—ถ๐—ป๐—ธ๐˜€ ๐˜๐—ผ ๐—ด๐—ฒ๐˜ ๐˜€๐˜๐—ฎ๐—ฟ๐˜๐—ฒ๐—ฑ - For an introduction to this community โ†’ Explore the Welcome Pack - New-ish to Fabric? โ†’ Check out our Fabric Foundation module - Studying for the DP-600? โ†’ Check out the DP-600 Module and the DP-600 category - Studying for the DP-700? โ†’ Check out the DP-700 Module and the DP-700 category - Want to get hands-on? โ†’ Check out Fabric Dojo ๐—›๐—ผ๐˜„ ๐˜๐—ผ ๐—ฒ๐—ป๐—ด๐—ฎ๐—ด๐—ฒ ๐˜„๐—ถ๐˜๐—ต ๐˜๐—ต๐—ฒ ๐—ฐ๐—ผ๐—บ๐—บ๐˜‚๐—ป๐—ถ๐˜๐˜†? - Share your knowledge and experience! Even if you're relatively new to Fabric, or the community, your opinion and experiences are valued here! A great way to earn your first point(s) is to introduce yourself in the thread below ๐Ÿ‘‡๐Ÿ˜€ Thank you for engaging and joining us on this exciting learning journey! ๐Ÿ™ Will
2 likes โ€ข Jun '24
Hi, I'm Bob from Billings, MT, I'm currently working to migrate into Fabric from a Snowflake environment.
1-2 of 2
Bob Samuelson
1
2points to level up
@bob-samuelson-1399
Sr. Data Engineer

Active 25d ago
Joined Jun 20, 2024
Powered by