2d • General discussion
How to A/B Test Your Skool DMs
To A/B test your Skool DMs, pick one message that goes to a lot of people, change one thing about it, split new recipients evenly between the two versions, and compare reply rates after each version has gone to enough people to tell them apart. The most useful test in most communities is the first message after the welcome, because almost everyone gets it.
What to test first
Start with the message most members receive. In order of impact, test:
1. The opener. "Hey [name], quick question" versus "Welcome in, [name]."
2. Question versus statement. Ending with a question almost always gets more replies, but test what kind of question.
3. Length. Two lines versus six lines.
4. Link versus no link. A link can pull people into a lesson, or it can make the message feel like marketing.
5. Timing. Day 2 versus day 4 for the same message.
Change one of these per test. If you change the opener and the length at once, you won't know which one did the work.
Pick one metric
Use reply rate within 48 hours. It's simple to count, and a reply in a DM means a real conversation started. If your follow-up includes a link to a lesson, you can also track whether they started that lesson, but keep reply rate as the main score.
How many sends you need
Be honest about your numbers. If 20 people join a month, a test will take months to show a real difference, and only a big gap will mean anything. Rough guide, not a law:
• Under 50 sends per version: only trust large differences, like 10 replies versus 25
• 100 or more per version: you can start trusting smaller gaps
• Small community: test bold changes, not single words
Running it by hand
No tool needed to start:
• Members who join on odd days get version A, even days get version B
• Keep a simple log: date, member, version, replied yes or no
• Review after each version has gone to your target number
A testing log template
• Message: day 2 check-in
• Version A: "How's your first couple of days going?"
• Version B: "What's one thing you hoped to get from this community?"
• Metric: replies within 48 hours
• Sends so far: A 0, B 0
• Decision date: when each reaches 60
What to do with the result
Keep the winner, then test the next variable against it. Write down what you learned in one sentence, like "specific questions beat general ones." After four or five tests you'll have a short list of rules that make every message you write better, not just this one.
What not to test
• Anything sent to cancelling or upset members. Write those with care and keep them personal.
• Tiny wording changes in small communities. You'll be reading noise.
• Two versions that are both bad. Rewrite first, then test.
A worked example
Say 80 people join a month. You test a day 2 message: A asks "How's it going so far?" and B asks "What's one thing you want to have done by the end of this month?" After six weeks, each version has gone to about 60 people. A got 9 replies, B got 21. That gap is big enough to act on. B becomes the default, and you test timing next.
Testing is also a guard against robotic messaging. If a version gets fewer replies, members are telling you it sounds automated. Skool Automation Mistakes That Make Your Community Feel Robotic covers the patterns that cause that.
StickyHive's DM sequences support A/B variants inside a sequence, so the split happens automatically. Sequences start on the Growth plan at $49/month, and you can see how they're built on the Skool DM sequences page.
Which message in your onboarding would you test first?
0
0 comments
How to A/B Test Your Skool DMs
skool.com/skool-automation
Skool automation for DMs, member follow-ups, posts, AI moderation, engagement, retention, and community workflows powered by stickyhive.ai
Powered by