[TECHNICAL CHALLENGE – Chunking & SplitInBatches Loop with Pinecone in n8n] Hey everyone, I’m working on building a smart agent that processes documents in n8n. I’m doing manual chunking (not using Pinecone’s native splitter) because I want full control over chunk size, overlap, formatting, and semantic structure before sending anything to the vector store. The pipeline does a lot: custom chunk splitting, cleaning, enrichment, reformulation, and finally vectorization in Pinecone. Everything seems to be in place… but there’s a weird issue: • I’m using a SplitInBatches node (batchSize: 1) to process chunks one at a time • I feed it 72 chunks, but only the first chunk is processed, and it never moves to the next • I’ve tried returning the loop via Merge, NoOp, and other tricks — no success On top of that, sometimes a single chunk results in 10 empty items after passing through the LLM or post-processing nodes, which breaks the whole logic. I’m trying to figure out: • How do you correctly handle batch looping in n8n with SplitInBatches? • How do you avoid the LLM or code nodes creating multiple empty outputs from a single chunk? • If I do manual chunking, how do I ensure Pinecone gets only clean, usable, and logically separated chunks? If anyone’s comfortable reviewing my workflow and helping fix it, I’d really appreciate it. I can share a stripped-down workflow.json file with everything in place, just needs tuning to work properly. Thanks in advance to anyone who loves debugging split logic and complex automations. Let’s solve this challenge cleanly.

Hello everyone, I’m experiencing a problem with the workflow I’ve set up for my RAG (Retrieval-Augmented Generation) agent, and I’d appreciate any advice or insights from the community. Workflow Overview: 1. File Listing Phase: • The first agent queries my storage source (e.g., Dropbox) and generates a complete list of available files along with their metadata (name, date, size, etc.). • This list is then filtered based on predefined criteria (e.g., files not yet processed or the most recent ones) to determine which files should be handled. 2. File Download Phase: • The second agent is supposed to take this filtered list and automatically download each file by sending the appropriate API requests (e.g., via HTTP). • The goal is to fully automate the process so that any new file added is detected, listed, and then downloaded without manual intervention. The Problem: • Although the first agent successfully lists all the files, the download phase does not trigger for certain documents, or it fails to detect new files. • It appears that the second agent is not correctly selecting files to download. This could be due to an issue with the filtering rules, trigger configuration, or the iteration process over the file list. What I’ve Tried: • I have verified the trigger configuration in my orchestrator (using n8n) to ensure it’s set up to detect new files. • I checked the filtering criteria used during the file listing (date, size, file name). • The API connection to Dropbox seems to work correctly for the listing phase. My Request: Has anyone experienced a similar issue in their workflow? What adjustments or best practices would you recommend to ensure that the second agent reliably downloads all relevant files? I’m open to any suggestions on how to debug this process or improve the transition from listing to downloading. Thank you in advance for your help and feedback!