Playlist Pitch A/B Testing: A Practical Guide for Indie Artists

In marketing, A/B testing compares two versions of the same element to see which performs better. Hypebot's explainer on music marketing describes it as showing two versions to comparable groups, changing one element at a time, and measuring a chosen outcome such as clicks, plays, opens or views.

Applied to playlist pitching, the "groups" are curators and the outcome is a response. You send Version A of your pitch to one set of playlists and Version B to a similar set, then compare.

That sounds simple, but a playlist pitch differs from an ad in three ways:

1. Your audience is small. There are only so many curators in a given genre and mood lane.
2. Each recipient is a person. A curator who sees two similar pitches from you in one week will notice. Don't send both versions to the same person.
3. The outcome is rare. Placement rates are low, so replies are usually a better early signal than adds. For context on realistic numbers, see our breakdown of playlist submission success rates.

If you run the test sloppily, you get a confident-looking number that means nothing. If you run it carefully, you learn which angle suits your music and which kind of curator responds to it.

What you can test in a pitch

The research on independent music marketing points to a few testable levers. Tools 4 Music lists pitch angles, cover art and email subject lines among the things artists can compare. For a playlist submission, that translates into the following.

The three pitch angles from Tools 4 Music are worth borrowing: mood-focused, data-forward and curator-specific. Curator-specific pitches, where you show how the track fits that exact playlist, are the angle we'd test first. Our guide on how to personalize a pitch to curators walks through what that looks like.

What not to test

Skip anything that changes the song itself between versions, such as sending a radio edit to one group and the album cut to another. You'd be testing two songs, not two pitches. Also avoid testing the same variable across playlists of very different types. A pitch to a lo-fi study playlist and a pitch to a workout playlist aren't comparable groups. If you're unsure how playlists differ, our overview of types of curated Spotify playlists helps you match groups.

How to set up a fair test

The core rule is the same in every source we checked: change one variable and hold the rest steady. Tools 4 Music says a valid test keeps other conditions, such as targeting, budget and copy, identical. Hypebot says the same about showing versions to comparable groups.

Here is a process that fits a solo artist.

1. Write the question first

AndR Music recommends writing the test question before you create the variants, so you're testing a specific problem that has an actionable answer. For example: "Does naming a track already on the curator's playlist raise my reply rate compared with a mood-only opening?"

2. Pick one metric

Choose a primary metric before you send anything. For most artists this is the reply rate: replies divided by pitches delivered. Secondary metrics come later (see the measurement section below). Hypebot advises defining your goal in advance, and not changing the test partway through.

3. Build a comparable list

Collect a pool of playlists in the same genre and mood lane, with similar follower ranges and similar update activity. Then split the pool at random. Don't put your "best" playlists in Version A. Sort the list, flip a coin for each playlist, or alternate down the list, so neither group is quietly stronger.

Random assignment matters because it keeps the groups similar, so a difference in results is more likely caused by your change than by outside factors. Spotify's own experimentation documentation, Confidence, makes the same point about randomization at a much larger scale.

4. Keep the rest identical

Same song. Same private link or Spotify link. Same sender name. Same day of the week, and ideally the same send window. If you pitch A on Monday morning and B on Friday night, timing becomes a second variable.

5. Record everything

AndR Music suggests logging the version files and noting any deviation from the original brief. A simple spreadsheet will do. Use these columns:

• Playlist name and curator
• Version sent (A or B)
• Date and time sent
• Reply (yes or no) and reply date
• Response type (positive, decline, no answer)
• Placement (yes or no) and date added
• Notes (anything unusual)

6. Don't peek and tweak

If Version A looks bad after four sends, resist rewriting it. Changing the test mid-flight makes the earlier data useless. Let both versions finish.

How many submissions you need

This is where honesty matters. Tools 4 Music suggests aiming for at least 200 samples per variant and a difference larger than 10 percentage points before acting, and gives 20 to 30 data points as a working range for pitch-angle tests in independent playlist outreach. Those figures come from a company blog, not a statistical authority, so read them as rules of thumb.

The two numbers pull in opposite directions because curator outreach is small. Two hundred pitches per variant means 400 curator emails, which is out of reach for most one-song campaigns. Twenty to thirty per variant is feasible but gives you only a rough direction.

A practical way to handle this:

• Treat 20 to 30 pitches per variant as the minimum for a directional read. You're looking for a big gap, not a small one.
• Require a large difference. If Version A gets 8 replies from 25 pitches (32%) and Version B gets 7 (28%), that is noise. If A gets 12 (48%) and B gets 5 (20%), that's worth acting on, then confirming.
• Pool results across songs. If one angle keeps winning across two or three releases in the same genre, your confidence grows even though each single test is small.
• Retest the winner. Use it as the new Version A and challenge it with a new Version B.

Timing matters too. Tools 4 Music cites 24 to 48 hours for email tests, but curators don't check inboxes like subscribers do. Many reply days or weeks later, or never. Give a pitch test at least a week or two for replies to arrive before you count anything as a non-response, and log late replies against the version that earned them.

What to measure

This part of the topic is where we found the least checked guidance, so what follows comes from how pitching works in practice, not from a published standard. Use three layers of metrics, from fastest to slowest.

Layer 1: Response metrics

• Reply rate: replies divided by delivered pitches.
• Positive reply rate: replies that signal interest, such as "I'll add it" or "send me the next one".
• Useful decline rate: declines that explain why. These are data. Our post on turning common playlist declines into data shows how to sort them.

Layer 2: Placement metrics

• Add rate: placements divided by pitches.
• Time to add: days between send and placement.
• Playlist quality of adds: a placement on an active playlist with real listeners beats several on dormant ones. Check for bot-inflated follower counts before counting an add as a win.

Layer 3: Listener metrics

After a placement, look at what listeners do inside Spotify for Artists: streams from that playlist, saves, and whether listeners return. These numbers belong to the playlist and song, not to your pitch wording, so don't use them to judge Version A against Version B. Use them to judge which playlists are worth pitching again.

Keep the primary metric for each test to one. If you track five and one of them favors B by chance, you'll be tempted to declare B the winner. Pick the metric before the test and judge by that.

Rules to follow while testing

Testing is only useful if you stay inside the platform's rules, because a flagged song teaches you nothing.

• Editorial pitching is a separate channel. Spotify lets artists pitch one unreleased song through Spotify for Artists, and its guidance is to submit at least seven days before release. It is a form, not an email, so there is no wording to A/B test against the editors. Spend your testing effort on independent curators. For release timing, see our note on pre-release playlist pitching.
• Don't pay for guaranteed placements. Spotify's terms discourage paying for placement or boosting streams artificially. Pay for tools or services that help you find and contact curators, not for a promised add.
• Check playlist authenticity before you pitch. A placement on a playlist with fake followers can hurt your stream profile. Include a bot check in your list-building step, so both test groups contain real playlists.
• Don't split one curator across both versions. One curator, one pitch per song.

Limits of pitch A/B testing

A small test can fool you. Hypebot warns against drawing conclusions from a tiny audience or from a few hours or days of data, and Tools 4 Music notes that small samples produce noise. If a result is inconclusive, extend the test, or accept that the variable may not matter much for your audience. AndR Music adds that you should say so plainly when results are inconclusive.

A few more limits to keep in mind:

• Results expire. Spotify's Confidence documentation notes that a measured effect is valid for the time of the test and may not carry over to later periods because of seasonal variation. A pitch angle that works in January may land differently during a summer release rush.
• Genre changes the answer. A data-forward pitch may suit a curator of emerging pop, while a mood-led pitch suits ambient or lo-fi.
• The song matters more than the wording. If a track doesn't fit the playlist, no subject line rescues it. Before blaming your pitch, read our list of common playlist rejection reasons.
• Curators are not randomized users. They have moods, backlogs and inbox habits you can't see.

A worked example

Say you're releasing an indie-pop single and have built a list of 60 verified playlists in that lane, each with real listeners and recent updates.

Question: Does a curator-specific opening (naming a track on their playlist and why yours sits beside it) get more replies than a mood-led opening?

Setup: Randomly assign 30 playlists to A (mood-led) and 30 to B (curator-specific). Same song, same link, same sentence length, all sent Tuesday morning over two days. Metric: reply rate after 14 days.

Result (illustrative numbers): A gets 6 replies from 30 (20%). B gets 13 from 30 (43%).

Decision: That's a large gap, so B becomes your default. On the next release, you use B as the control and test a new variable, such as pitch length. You also note the sample was small, so you don't over-claim.

Those numbers are an example for illustration, not a benchmark.

Where Playlist Pilot fits

Manual A/B testing is slow, mostly because of the list-building and the writing. Playlist Pilot analyzes your track's audio characteristics, genre and mood, matches it to playlists run by real curators, and generates a personalized pitch for each one, so a curator-specific version doesn't mean writing 30 emails by hand. We don't charge per pitch, which matters for testing because you can send more submissions without the cost climbing with each one. Our free playlist checker also scans for bot-inflated playlists before you send.

Plans are Starter at $19.99 per month and Professional at $29.99 per month, and Enterprise is available with no published price. See the options on the Playlist Pilot plans page, or read how unlimited playlist submissions change the math on testing.

Our perspective

Most artists we talk to don't need a statistics course. They need to stop guessing. The common failure isn't running a weak test, it's never running one: the same pitch goes to every playlist, nobody tracks the replies, and a decline reads as a verdict on the song.

We think a pitch is a first message in a relationship, and the long game matters more than a single win. Curators who see you pitch relevant tracks, with a clear reason each fits, start to recognize your name. That is why we built Playlist Pilot around ongoing curator contact instead of a one-off fee per pitch. When you can pitch the same curator again for your next release, each test builds on the last one.

Our advice: test one thing at a time, keep your list clean, and let the decline messages teach you. If you want a deeper look at what moves response rates, our four-week playbook on increasing curator responses is a good next step.

— Zander

Sources

• How to use A/B Testing to Market Music more successfully (Hypebot)
• A/B Testing Your Music Marketing: What to Test and How (Tools 4 Music)
• Music Marketing Testing: Hooks, Conditions and Uncertainty (AndR Music)
• A/B Tests, Confidence Documentation (Spotify)