Embedded Files
PACKEDANDLEFT
A social media page interpreting the intersection of travel and marketing
I decided to do a 6-week experiment to test how real social media users respond to content. While I planned to do a data analysis through a statistical platform, the Instagram page did not perform very well. I decided to continue to work on the page and my planned experiment. Although the data did not show what I would have liked, I still did an analysis and strategic insight for my future A/B testing and took what I have learned from this and applied it to the next opportunity.
The Finding
The Finding
After the 6 weeks were completed, it was found that A/B testing does not work without outreach first, shown through flat views across almost all variables tested. While the content had some outreach, the main issue was distribution, meaning there was not enough audience activity to properly test which content variables performed better.
What I Did
What I Did
An anonymous Instagram account was created with a travel marketing niche as the main content focus. Over 6 weeks and 24 posts, I tested different content variables to see which changes could improve views and engagement.
Each week, one variable was tested as an A/B pair, with impressions tracked at 48 hours. Two additional non-A/B posts were also posted each week to create more consistent content and hopefully increase overall outreach. The variables tested were hook style, opening visual, video length, thumbnail, caption style, and CTA.
The main goal of the experiment was to see whether small changes in how content was presented could make a measurable difference in performance.
The Data
The Data
The data gathered after the 6 weeks was underwhelming, with only 9 of the 24 posts receiving any views at all. Additionally, only 1 post received any engagement, with 199 views and 2 likes. Every other post had an engagement rate of 0.00%.
The analysis also showed that 15 posts received 0 views at both 24 hours and 7 days, which inferred that these posts were never pushed by the algorithm enough to create usable data.
However, in Week 6, the biggest A/B difference developed around the Call to Action variable. The specific question CTA received 120 views, while the post with no CTA received 23 views, creating a difference of more than 5x.
The other variables had much smaller differences. The opening visual had a 6-view difference, video length had a 24-view difference, and the caption variable resulted in both posts receiving 0 views.
What It Means
What It Means
After analyzing the results, it became clear that marketers cannot properly test content variables if the content is not being seen in the first place. The Week 6 CTA comparison showed a potential signal, but there was not enough evidence to say that the CTA itself caused the difference rather than it being random variation.
This showed me that distribution needs to come before optimization. If there is not enough reach, A/B testing can make it look like there is no difference between two pieces of content when the real issue is that neither piece was given enough opportunity to perform.
What I'd Do Differently
What I'd Do Differently
This experiment has shown me the importance of distribution before optimization. In the future, I would focus on a longer period of consistent outreach to increase page and content views before beginning A/B testing.
I would also establish a clearer message, branding, and voice before testing individual content variables so the account has a stronger foundation and a more defined audience.
Finally, I would want significantly more views before making decisions from A/B testing data. As a starting benchmark, I would aim for 300+ views within 24 hours before using the results to make decisions about content performance.
Skills Demonstrated
Skills Demonstrated
A/B testing
Metric tracking and logging data
Interpreting null results
Finding signals in low-volume data
Adjusting the strategy for future A/B testing
Google Sites
Report abuse