Issue with File Download Phase in My RAG Workflow – Request for Assistance

Hello everyone,

I’m experiencing a problem with the workflow I’ve set up for my RAG (Retrieval-Augmented Generation) agent, and I’d appreciate any advice or insights from the community.

Workflow Overview:

1. File Listing Phase:

• The first agent queries my storage source (e.g., Dropbox) and generates a complete list of available files along with their metadata (name, date, size, etc.).

• This list is then filtered based on predefined criteria (e.g., files not yet processed or the most recent ones) to determine which files should be handled.

2. File Download Phase:

• The second agent is supposed to take this filtered list and automatically download each file by sending the appropriate API requests (e.g., via HTTP).

• The goal is to fully automate the process so that any new file added is detected, listed, and then downloaded without manual intervention.

The Problem:

• Although the first agent successfully lists all the files, the download phase does not trigger for certain documents, or it fails to detect new files.

• It appears that the second agent is not correctly selecting files to download. This could be due to an issue with the filtering rules, trigger configuration, or the iteration process over the file list.

What I’ve Tried:

• I have verified the trigger configuration in my orchestrator (using n8n) to ensure it’s set up to detect new files.

• I checked the filtering criteria used during the file listing (date, size, file name).

• The API connection to Dropbox seems to work correctly for the listing phase.

My Request:

Has anyone experienced a similar issue in their workflow? What adjustments or best practices would you recommend to ensure that the second agent reliably downloads all relevant files? I’m open to any suggestions on how to debug this process or improve the transition from listing to downloading.

Thank you in advance for your help and feedback!

3 comments