Attached is what a rigorous CRO workflow looked like at my old agency a few years back. Every step had a dedicated human. Research, ideation, technical feasibility, documentation, design, coding, QA. If anyone was slow or busy, the whole thing stalled. And this is assuming the team actually had time to go through all of it — sometimes you had to cut corners just to keep the program moving. Last week @Victor Paytuvi and I went live to talk about how much this has changed. Full replay here if you missed it: How AI is Reshaping Ecomm Optimization Workflows — Live with Intelligems A few things that came up: - Automated test reporting was both of our first real AI use case (~2023). The mechanical part of writing up results doesn't need a human. That time goes back to strategy. - Prototyping with AI closed the communication gap between CRO strategists and designers. Days of back-and-forth became hours. - AI-coded variants are no longer science fiction. One practitioner at Jones Road is running his entire program solo today. We also talked about what won't change: the contextual judgment that comes from knowing a brand deeply, its customers, its history, what's keeping the founder up at night. That's still on us. Curious where this community is at. What's the first step in that process you stopped doing manually?

My take: the concern is valid, but "therefore stop testing" is the wrong conclusion. The criticism has a legitimate core. The testing industry has been slow to reckon with acquisition costs. If your test "wins" on conversion rate but the winning experience makes your CAC worse, you might have won nothing — that's a real blind spot. But every single change you introduce might affect your CACs; stopping to measure their impact is not a solution. What's actually going on in most cases: paid media and on-site testing run on completely different clocks. Meta sees a 5% CAC shift and someone acts that day. A valid on-site test usually needs weeks of traffic. They're measuring different things in different places, and when paid wobbles during a test window, it's easy to blame the test. But there's something many people miss: if your numbers hurt a little while the test runs, but you walk away knowing something you'll use for the next 6, 12, 18 months... that's not a loss. That's the cost of learning something real. A temporary CAC spike that teaches you the right price point is a bargain. What I'm curious to hear from you: - Have you seen any true correlation between running tests and CAC spikes? What did you do about it? - Has this changed how you structure your program at all? - How are you handling this conversation with stakeholders? Our gut says the CAC impact is probably overstated... but that's not always how it reads in the room. How do you navigate that?

In your opinion, what is it? We all use heatmaps, surveys, customer service reports, analytics, etc. But do you have a “secret” research method that worked wonders for a particular client, or your own brand?

What's the most impactful or interesting test you all are running currently?

Couple headline tests that have RIPPED for me lately, and kinda made me re-think my approach to testing. For the first time ever, I wrote headlines with specific intentions, deeper than just "make more money" ... On the first test here, I wrote specifically to increase CVR, the headline was geared at convincing more people on our website that our products were good enough quality and worth more to them than the price we offer it for... That's usually how I write... But now that I'm at a supplements brand, I took a swing at writing headlines that would persuade people to subscribe, rather than one-time-purchase our supps. The headline & subtext is more about how our products already fit it to their daily routines and how it can improve them. CVR and Rev Per Session were basically flat on that test, but the Sub % rate was WAYYY higher, so that's a huge win for us. Anyone else have examples of content tests designed to move specific metrics like this?