Most CTV platforms are running global feature flips and calling the before/after comparison a test. Winston Francois builds cohort-based and geo-holdout experiments designed around what your platforms actually let you measure – the 10-foot UI, ad load and frequency capping, and trial-to-paid conversion – and around the certification calendar that decides when a test can even go live.
Why Growth Experiments Break Down in CTV
You can't split traffic like a mobile app
Roku, Fire TV, Samsung Tizen, LG webOS, and Vizio SmartCast don't give you the same user-level randomization and remote-config SDKs that iOS and Android do, and several of them restrict how much of the first-run and store-listing experience you can change without a new build. Teams without a workaround either run no experiments at all, or ship a global change and analyze the before/after as if it were a controlled test. That produces decisions built on noise, not signal, and it's the single most common failure mode in CTV growth teams.
App store certification turns every UI test into a multi-week cycle
A row-placement, thumbnail, or navigation change has to clear Roku, Amazon, Samsung, LG, and Vizio certification before it reaches a single living-room screen, and each platform runs its own review queue on its own timeline. A change that would ship same-day on web can take one to three weeks just to go live on one platform, and the test clock doesn't start until it does.
Ad load tests get confused with churn noise
Raising frequency caps or ad load on an AVOD or FAST tier moves ad revenue per hour immediately, but the churn cost – people who quietly don't come back next week – takes two to four weeks to surface, and it's easy to misattribute that drop to a content licensing window ending or a competitor launch happening in the same period.
Trial-to-paid experiments get diluted by multi-device signup
A viewer who starts a free trial on a phone and converts on the living-room TV counts as two separate sessions to reconcile, and most CTV analytics stacks weren't built to unify identity across those two contexts the way a single-app mobile funnel is. Teams routinely run onboarding tests device by device and never look at the blended cross-device conversion number – which is the number that actually determines revenue, and often tells a different story than either device alone.
How We Help
We start every growth experimentation engagement with an audit of what you can actually test, given your specific platform mix.
From that audit we build an experiment roadmap organized around cohort-based and geo-holdout designs instead of individual randomization, because most CTV platforms simply can't split traffic at the user level the way a mobile app can.
For 10-foot UI experiments, that looks like matched-market or matched-device-population holdouts – the same content row in position two for cohort A and position four for cohort B, or the same title with a close-up-face thumbnail against an ensemble-shot thumbnail split by geography – with submission timed so every variant clears certification on the same cycle instead of trickling out staggered and uncomparable.
For ad load and frequency capping, we run true holdout cohorts kept at the current setting through the entire churn measurement window, not just the first week of revenue impact, and we tie the read against ACR and engagement data where it's available so ad-experience churn doesn't get blamed on content or vice versa.
For onboarding and trial-to-paid flows, we run device-level cohorts first, then reconcile to a blended cross-device conversion number before calling a winner – so a result that only holds up on one device type doesn't get shipped as a platform-wide win.
What makes this different from a typical growth consultancy is the fractional, embedded model. We don't hand over a slide deck of test ideas and disappear.
Every test we run ships with a pre-registered success metric and a stop, scale, or kill call defined before launch – not a narrative fit to whatever number happened to move afterward. That discipline is what separates an experimentation program from a series of launches with opinions attached.
What we deliver
- Platform-by-platform audit of what can be tested and how, across Roku, Fire TV, Samsung, LG, and Vizio
- Cohort-based and geo-holdout test designs for 10-foot UI row placement and thumbnail/artwork variants
- Ad load and frequency-capping holdout tests measured against churn, not just immediate revenue
- Onboarding and trial-to-paid flow tests with cross-device reconciliation
- Certification-calendar-aligned experiment roadmap sequenced against the upfront and NewFronts cycle
- Pre-registered success metrics and stop/scale/kill criteria for every test
On CTV, a test isn't done when the code ships – it's done when it clears certification on every platform you're measuring, and the clock for a real result only starts after that.
Our Methodology
Days 1 through 30 are the audit and roadmap. We inventory what's actually testable on each platform you run, identify which current "wins" were really just global rollouts with no comparison group, and build the cohort and holdout framework your team will use going forward. We prioritize three to five tests against the certification calendar so the roadmap reflects real lead times instead of an internal quarterly deadline that certification can't hit.
Days 31 through 60 launch the first wave, typically one row-placement or thumbnail test in the 10-foot UI, one ad-load or frequency-cap test with a proper holdout cohort, and one onboarding or trial-flow test. Measurement gets instrumented before launch, including the churn observation window for the ad-load test and the cross-device reconciliation logic for the trial test, so nothing gets bolted on after the fact.
Days 61 through 90 read out the first wave against the pre-registered metrics, feed what we learned into the next roadmap cycle, and hand the internal team a repeatable framework – cohort design templates, holdout logic, and a certification-aware test calendar – they can run on their own once the engagement scales down or ends.
How We Work
A fractional lead embeds directly with your growth, product, or data team for roughly 15 to 20 hours a week, working from your test backlog rather than a generic playbook. This isn't an account manager checking in – it's someone who writes the test plans, sits in on the same standups your team runs, and has direct access to the raw experiment data rather than a summarized readout.
Cadence is built around what the platforms allow: weekly working sessions to keep test design and instrumentation moving, and readouts timed to whatever review cycle certification actually permits rather than an arbitrary biweekly slot that a pending Roku or Samsung review can't meet. Day-to-day coordination runs async in Slack so nothing waits for a scheduled call.
Clients should expect a named person, not a rotating team, and should expect to be asked hard questions about what their current experiments are actually measuring – a lot of the first 30 days is surfacing tests that were never really controlled in the first place.
After the initial 90-day sprint, engagements either extend to cover the next wave of tests or wind down into a handoff, with full documentation of the cohort and holdout methodology so the internal team owns the framework going forward rather than depending on us to run every test.
- Faster, more reliable reads on which 10-foot UI changes actually move engagement
- Ad load and frequency-cap decisions backed by isolated churn data instead of blended revenue noise
- Trial-to-paid conversion calls based on blended cross-device numbers, not a single-device result
- A certification-aware test calendar that stops UI experiments from missing the upfront and NewFronts cycle
- A repeatable experiment framework the internal team can run without an outside vendor
If your ctv / connected tv company needs growth experimentation leadership, we should talk.
Expand your marketing team output with our experts
Let us take a custom approach to your growth goals by assembling and leading the best-in-class marketing team to support your next stage.
Frequently asked questions
What does a growth experimentation engagement with Winston Francois cost?
Engagements typically run $15K-$35K per month depending on how many platforms you're testing across and how much of the measurement infrastructure already exists. A company running experiments on two or three CTV platforms with existing ACR access costs less to support than one starting from a global-rollout-only baseline with five OS targets.
How long does it take to see results from a CTV growth experiment?
Plan on certification alone adding one to three weeks per platform before a UI test even goes live, and then the measurement window on top of that – two to four weeks for a churn read on an ad-load test, less for a straightforward conversion comparison. The 90-day engagement is built to get one full wave of tests through certification, live, and read out, which is why the roadmap is sequenced against the certification calendar rather than an internal quarterly deadline.
Who do we actually work with day to day?
A single fractional lead embeds with your growth, product, or data team for the length of the engagement, working from your existing backlog and standups rather than a separate outside process. That person writes the test plans, defines the holdout logic, and stays through the read-out – there's no rotating account team or handoff between a strategist and an execution team.
How is this different from hiring a growth agency or a data analyst?
A generic growth agency tends to bring mobile and web A/B testing assumptions that don't hold on CTV platforms that can't split traffic at the user level, and a single data analyst usually isn't positioned to also design the cohort structure, negotiate certification timing with your platform teams, and translate the result into a ship decision. We do the experiment design, the measurement plan, and the decision framework together, embedded, not as three separate handoffs.
How do you measure ROI on experiments that can't use standard A/B testing?
Every test ships with a pre-registered success metric and a defined minimum detectable effect before launch, so the ROI conversation isn't retrospective. For a row-placement test that's typically engagement or watch-time lift on the affected cohort versus the holdout; for an ad-load test it's net revenue after accounting for the churn cost measured in the holdout window; for a trial-flow test it's the blended cross-device conversion delta.
What kind of CTV company is the best fit for this?
This works best for Series A through growth-stage CTV and streaming platforms in the $5M-$100M ARR range that are already live on at least two of the major smart TV platforms and have some baseline analytics or ACR access, even if it's not fully utilized yet. Companies pre-launch or still deciding on their platform footprint are better served starting with strategy work before committing to an experimentation program built around certification timelines they haven't hit yet.
