Hello Fabric Community, I’m encountering an issue with a data pipeline in Fabric where the pipeline is supposed to transform data from the Bronze Lakehouse and save it to the Silver Lakehouse. Although I have explicitly set the pipeline configuration to overwrite data, the output in the Silver Lakehouse is being appended instead of being overwritten. Details: - Pipeline Setup: - The pipeline reads data from the Bronze layer. - Data transformations are handled within a PySpark notebook. - The final DataFrame is intended to append new records to the existing data in the Silver Lakehouse. - Observed Behavior: - Despite setting the save mode to prevent appending, the data in the Silver Lakehouse is getting completely overwritten with each pipeline run. Steps I’ve Taken: 1. Checked the Notebook Save Mode: Confirmed that the save mode is set to append, not overwrite, in the PySpark code. 2. Reviewed Pipeline Settings: Ensured that the pipeline activity is configured correctly to append data to the Silver layer. 3. Validated Metadata and Schema: Verified that the schema in the Silver Lakehouse matches the DataFrame structure to prevent schema conflicts. 4. Analyzed Logs: Examined the pipeline execution logs, but no specific errors or warnings indicate why the data is being overwritten. Request: Has anyone encountered a similar issue, or can anyone suggest what might be causing the data to be overwritten instead of appended? Any insights or recommendations on how to resolve this would be greatly appreciated. Thank you in advance for your help! Best regards, Amit

Welcome to all new members, here's some links and information to help you get started! 𝗤𝘂𝗶𝗰𝗸 𝗟𝗶𝗻𝗸𝘀 𝘁𝗼 𝗴𝗲𝘁 𝘀𝘁𝗮𝗿𝘁𝗲𝗱 - For an introduction to this community → Explore the Welcome Pack - New-ish to Fabric? → Check out our Fabric Foundation module - Studying for the DP-600? → Check out the DP-600 Module and the DP-600 category - Studying for the DP-700? → Check out the DP-700 Module and the DP-700 category - Want to get hands-on? → Check out Fabric Dojo 𝗛𝗼𝘄 𝘁𝗼 𝗲𝗻𝗴𝗮𝗴𝗲 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆? - Share your knowledge and experience! Even if you're relatively new to Fabric, or the community, your opinion and experiences are valued here! A great way to earn your first point(s) is to introduce yourself in the thread below 👇😀 Thank you for engaging and joining us on this exciting learning journey! 🙏 Will