How we test 3k+ videos weekly without blowing up our UA budget

Intro

Testing UA videos implies spending money. Not all spending is equal and more importantly, not all platforms are built similarly when it comes to creative testing. At Poolday, we use a channel waterfall approach to test creatives, meaning we initiate testing on 1 channel - TikTok, and then the best creatives surfaced by TikTok would flow into Meta, followed by Applovin, Unity, and subsequently, additional channels.

‍

It's a common misconception that successful user acquisition demands extensive, multi-platform testing, months of commitment, and substantial financial resources. However, our journey through producing and testing over 100,000 videos has led us to a streamlined strategy. Here’s how you can pinpoint high potential ads with just a couple thousand dollars and an adaptive testing philosophy: 

‍

Why you should use Tiktok for testing?

Ever upload multiple creatives to TikTok and notice only ONE grabs most of the spend? This is exactly why we love TikTok: winner takes all. 

‍

With over 100,000 videos produced in the past year, we required a platform capable of quickly determining the success of a creative. When we upload 16 creatives to an ad group on TikTok, the platform almost instantly zeroes in on high potential hits, with a single creative eating 90%+ of the spend in a “winner takes all” showdown. There are often false positives and false negatives, but the insights are golden: 

1. The video can spend on TikTok (which in itself is a win)
2. The video got something right
3. We can try to deploy this video in the main campaigns (BAU campaigns), and either kill it or scale it. Whatever happens, it’s a learning opportunity for the next batch of videos

‍

What budget to test with?

For an ad group with 16 videos, we typically spend $100-300/day. If you do the math on a per creative basis, it’ll be a tiny number. However since TikTok has a winner takes all approach, we end up spending 90% of that budget on 1 video which gives us initial statistical significance in the experiment. What’s ironic is that we need to produce a lot of videos for TikTok to find a winner, but then all of the other videos are thrown away after the experiment and we focus on the top performer to generate iterations for the next batch.

Learn More: Overcoming Tiktok Creative Fatigue: Strategies for Maintaining Spend and Performance High

Here is a sample of an ad group containing four creatives, to which a daily budget of $100 was assigned. It's evident that one ad monopolized the entire budget, utilizing nearly all of it at 98$. Despite each creative in the ad group having a comparable click-through rate (CTR), only one ad accounted for the total budget consumption. (To maintain the confidentiality of the app involved, all images and IDs have been obscured.)

‍

‍

How long to test for?

It depends on the goal of the campaign. Ideally, we aim to review at least 4-7 days of data, or even up to 10 days, to ensure we’re making informed decisions. If the goal is a down-funnel event like a purchase action, we target achieving 100-500 instances of the target event before calling the campaign optimized on TikTok. More on the number of required events from Facebook here.

Learn More: 9x ROAS using hook swapping: Insights from 100k+ videos tested using AI actors

What about testing other channels?

We deploy to the other channels (Meta, Applovin etc) after testing on Tik Tok. We find that in 60% of the time, top performers from TikTok perform well on Meta & Applovin. We then focus additional UA efforts on Meta to drive post-install optimization tests (to understand if our creatives back out from an ROI perspective on that channel). This is when we consider the creative a unicorn ad: it scaled on TikTok, then it proved successful on Meta using post-install KPI as a success metric.