9x ROAS using hook swapping: Insights from 100k+ videos tested using AI actors

In content creation, akin to genetics, minor adjustments such as hook swapping or altering background music can significantly enhance a video's engagement and performance, embodying the idea that small mutations can drive substantial evolutions. Despite the lack of a precise method for predicting success, as even the most random adjustments can result in high-performing videos, we've identified four factors that significantly influence outcomes during testing: 

‍

5 key Video Variables to focus on

‍

Since we had no accurate way to predict what will perform, we looked for the variable changes that on average lead to the biggest performance volatility (volatility defined as better performance, or worse performance, for experiments with $20k+ of spend per experiment). We found the following key variables were the most critical ones to focus on, in order to limit the number of possible variables:

1. Hook Swapping: the first 3 seconds of the video can 9x ROAS using hook swapping
   Example: Changing the first sentence of the video from “to all the ladies out there” to “to all the Mrs. out there” yielded a 2.6x improvement in ROAS, and this particular video reached $50k of (profitable) spend.
   
   
2. Base story: the story of the video, what the video will talk about [6x]*
   Example: If an app is in its early stages and there's a need to discover what appeals to users, crafting three varied scripts, each showcasing a unique value proposition, and testing them in an ad group is a practical approach. Observing the click-through rate (CTR) and total spend can reveal which value proposition stands out. 

3. Actor Selection: who appears in the video [4x]*
   Example: To determine which actor connects best with an audience, test six different actors (varying by gender, ethnicity, and age) using the same foundational creatives. Within each ad group, alter the hook while keeping the actor consistent. Consider cost per install (CPI) and total budget spent as metrics to identify the most effective actor.
   
   
4. Thumbnail Optimization: the very first image of the video, typically the background + scene [2.7x]*
   Example: The initial frame is crucial in a TikTok video, making it valuable to experiment with various vibrant and dynamic thumbnails while maintaining consistency in the rest of the creative elements. This strategy helps identify what captures your audience's interest. Evaluating the cost per acquisition (CPA) and total expenditure will guide you in pinpointing the most engaging thumbnail.

5. Music Selection: not the sound effects but the actual song being played in the background [2.4x]*
   Example: Typically, TikTok has 5-10 songs trending at any given moment. To discover what resonates with your audience, test each trending song, ensuring the rest of the creative content remains unchanged. Alternatively, experimenting with various styles of background music, instead of just trending tracks, might yield better results. Use cost per acquisition (CPA) and total budget spent as measures to determine the most successful approach.

‍

*The above items are ranked in order, from most important to least important. On a batch of 100 videos produced (with the only difference between the videos being the hook, keeping the remainder of the video identical in the 100 videos), the ROI on the best performer was 9x higher than the lowest. This is what the [X] means next to each section: the performance delta between best to worst.

Rapid Iterations on Top Performers vs. Producing New Concepts

‍

Why does A/B testing supersede churning out fresh, new creatives? Our top performer video (lifetime) was deployed in July of 2022…and is still running with millions of $ spend on it. This isn’t just another successful ad, it’s a unicorn ad– a standout performer that has not only stood the test of time but has also thrived under continuous refinement.

‍

After honing in on our A/B testing infrastructure, we found this winning video in July, but maintained its status as a “winner” by iterating on small tweaks of the content of the video, and other minor variables. We’re now at 1k+ iterations on this video, and this is how we’ve unlocked sustained growth without the constant need to seek out new concepts. A unicorn ad is a superior investment of time and resources than a top performer. It’s an ad that gets to live on far more than any other creative and outperforms other creatives by a significant margin. Looking at our dataset, it took close to 100 iterations to generate a unicorn ad, and its unmatched value presented the need to double down on our A/B testing infrastructure to produce new videos within a given framework. Our approach has been clear: Allocate approximately 85%+ of our spend to refining proven winners through iterative refinement, and reserve the remaining 15% for exploring new concepts.

‍

We realized it’s the same allocation we have for existing UA channels vs new tests. We encourage anyone to try this at home: take a good performer and generate 10 variations (ask the content creator or AI actor to produce 10 variations of the hook using the exact same video). 

‍

Learn More: How we test 3k+ videos weekly without blowing up our UA budget

‍

TikTok is the king to churn out bad creatives and surface good creatives

‍

We’ve invested over $10m+ on our ads, and discovered through spending on other channels for the past decade that TikTok was the best platform to churn out bad creatives and surface good creatives. 

Our insights and methodologies, including our testing approach, will be detailed in an upcoming piece on our LinkedIn account, following our announcement of  poolday.ai to the public! Our process is straightforward yet tried and true: We test 16 creatives within a Tiktok Ad Group, with a specified budget at the ad group level. The highest performing ad then advances to our BAU (Business As Usual) campaigns, and to extend across other platforms.

This methodology allows us to control the total budget spent on testing without cannibalizing our main performance campaign, despite encountering both false positives (creatives that perform great early on but don’t back out well post-install) and false negatives (creatives that ended up being top performers even though the early performance was bad).

‍

This approach has proven to be the most efficient way for us to surface creatives that are high potential before they are moved to BAU campaigns. Successful ads on TikTok often perform well on other platforms, such as Meta and Applovin, though the converse is not always true. This method underlines the unique advantage and insights gained from our TikTok tests, guiding our broader advertising strategy.

‍

Learn More: Popular TikTok Trends UA: Horror story of our video blowing up through a trend

‍

The Surprising Power of Intentional Typos

There’s a lot of “growth hacking” strategies out there that require resources and time, but we’ve found one that requires no money and less than 1 second … spelling mistakes. We voluntarily add intentional typos in videos or make mistakes that are outrageous, prompting swift and eager corrections from the TikTok community. While that specific audience is not relevant for us, we still found the organic reach is expanded. 

‍

In our next posts we’ll discuss:

1. Trends: should you focus a part of your testing methodology on the flywheel of viral concepts?
2. Proximity score between TikTok and other UA channels (likelihood for a great TikTok video to perform well on other channels)
3. The science behind producing top performers (yes, you can run “experiments” by developing hypothesis)

‍