What are some CI/CD Best Practices?
Hey everyone, first time question poster here. So I have been tasked with looking into git integration and CI/CD best practices at my job. There are a couple of snafus I have run into. 1. Manuel migration from development to production. Using the development pipeline seems like a good use case, however, this is still a lot of manual interaction regarding setting up triggers and unsupported items (looking at you spark jobs). Are there any good ways to parameterize items when using this? Are there any good workarounds? 2. Underlying data access. When publishing a report with a direct lake semantic model to a business user workspace, I have noticed a bit of hassle in having to manually add access to each lake house. Giving access to the workspace that contains all the pipelines and data migration logic is also undesirable. Maybe this is just something that has to be done, but looking for ideas. 3. I would really like to be able to develop locally, test, and then push to Fabric (on the data engineering side). I know Fabric has a rest API and I am imagining setting up some logic to be able to use this, wondering if this endeavor would be worth it. Ultimately, anything I can take the time to automate would be awesome. TLDR - I need to organize multiple business departments and need to manage access to lots of data and want to incorporate Dev and production separation. Would prefer to be able to work local and want to automate as much as possible. I know that is a lot, any help with best practices or even some brainstorming would be appreciated.