Table of Contents
- Create a variant
- Review and set up your test
- Start your test
- Your test in progress
- Send your blast
- Final results
- A note on meaningful results
Watch a video
Create a variant
Note: Automatic email A/B testing is only available to those on the Team plan or above.
Begin by creating an email blast. Add recipients, select a theme, and navigate to the Content tab.
The Content tab is where you will enter the content of your email blast and set up your A/B test. You can test one of two variables:
- Email sender (this is who the email is “from”)
- Subject line
Note that you will test Sender A and Sender B from the same Broadcaster. Both variants will be sent from the same email address but listed with different sender names.
1. Choose a variable to test. Click + Add variant above the appropriate text box. You can only test one variable at a time.
2. A second text box will appear. Enter content for the two test variants.
You can remove the variant by clicking “- Remove variant” above the second text box. If you remove the variant you will lose any content entered into the text boxes.
3. Enter content for the body of your email.
4. Click Save or Save and preview to save and preview your changes. The preview of your email blast will refresh and update below.
Review and set up your test
1. Review your email within the Review and send tab, both test variants will display.
2. Select the percentage of recipients you want to use for the test. Sliding the handle right or left will adjust the size of your testing groups. The remainder will receive the winning variant. Recipients are randomly divided into these three groups. You cannot select what group a recipient will fall into.
3.You can have your blast send automatically when the test completes by checking the appropriate box. Please note, that you can only auto-send if the test can calculate meaningful results. Therefore, you must have a large enough sending population, testing groups of a certain size, and let the test run the full recommended time.
The ideal total population is upwards of 21,000 — but you can test with fewer recipients, you'll just need a larger test segment and more time. For example, with 13k emails and a 72% test segment, it'll take ~6 hours to automatically select a winner. Alternatively, you can instead opt to pick and send a winner manually at a time of your choosing.
4. Select Open rate or Click rate from the dropdown menu. This will determine how the winner of your test is selected.
You will see a suggestion for how long your test should run. This is the amount of time needed to conclusively declare a winner based on data collected.
Factors that affect your test length include:
- The size of your testing groups. Smaller groups will need a longer test.
- Whether you selected open or click rate. Click rate will require a longer test.
5. You may choose to extend the length of your A/B test. Enter a number into the provided text box.
This will add a number of hours to the minimum test length. Longer tests are more accurate, so you might consider this option to extend the length of your test without increasing the size of your test groups.
The total length of your test will display in a notification below the settings.
Start your test
Begin your A/B test by clicking Send A/B test now. You may also schedule your A/B test for a future date.
1. Click Schedule A/B test.
2. Enter the date and time for the start of your test.
3. Click Schedule. Edit the date of your scheduled test by clicking on the grey date/time button.
Adjust your date and time and click Schedule. Unschedule the test by clicking Unschedule.
If you selected automatic sending, the blast will send after the scheduled test has completed.
Your test in progress
Test results will display on the Preview screen of your email blast.
You will be able to track:
- When it was sent
- Email stats
- The time remaining in the test if you selected automatic sending
Send your blast
There are three ways to send your winning blast.
1. You may elect for automatic sending. Select the appropriate checkbox while setting up your A/B test. When the test completes the winning variant will be sent to the remaining recipients. If the test is inconclusive, Variant A will send.
2. Schedule your blast for a future time and date. Once your A/B test is in progress you can manually judge which variant has the most engagement. Select the chosen variant from the dropdown menu and click Schedule.
Select the date and time for your blast to send. You may edit the date at any time by clicking on the blue link with the date and adjusting the information.
3. Manually send the blast. While the test is in progress, select the variant you want to send from the dropdown menu and click Send to remaining recipients now.
After the test is complete and the winning variant sent, the results from the test and final send will remain accessible on the Dashboard of the email. You will also be able to see the stats of the email at [Broadcaster] > Email > Blasts.
A note on meaningful results
You will notice that automatic sending is built around "meaningful" results. A note on how meaningful results are calculated:
There is a standard period of time that a typical recipient needs to make a decision about an email: to open, to click, or to delete. After looking at the aggregate historical data of all email blasts sent in NationBuilder, this standard was calculated. When you set up an A/B test, the specifics of your test are measured against this standard and a recommended running time is produced. Running your test for the full recommended time will lead to meaningful results with a 95% confidence in accuracy.
Keep in mind that you have some control over the recommended running time. The factors listed below will affect the runtime:
1. Open vs Click rate. More people will open your email then click on a link. Therefore it will take a longer period of time to collect accurate data on the people who clicked in your email.
2. Size of testing groups. Increasing the size of your testing groups will always decrease the recommended runtime because it means more data is collected before a final decision is made.
3. Size of recipient population. The larger your list, the more data you can collect in a shorter period of time. Note that A/B testing is often not meaningful for lists smaller than 21,000 recipients because of the data needed to accurately declare results.
When sending an A/B test it is important not to make a decision on the data too quickly. Statistics are often misleading and it is hard to be certain that your test has led to a meaningful conclusion. The provided runtime is meant to act as a recommendation to ensure that your results are as meaningful as possible.