AdTech teams ship pricing changes, onboarding flows, and optimization features on conviction, then argue about whether they worked. We install a growth experimentation program – prioritized hypotheses, clean A/B design, and incrementality measurement that survives signal loss – so the team knows which changes actually move revenue.
The Problem
Big changes ship on conviction and nobody can prove the impact
In AdTech the highest-stakes changes – take-rate adjustments on the SSP, bidding algorithm tweaks, publisher onboarding redesigns – get shipped because a leader believes in them, not because they were tested. When revenue moves, three teams claim credit and nobody can separate the change from seasonality or a market shift. The company keeps shipping on conviction because it has never built the discipline to test, and it cannot learn from its own decisions.
Your A/B tests are statistically broken and you do not know it
Most AdTech growth teams run something they call A/B testing, but the assignment is leaky, the sample sizes are too small for the effect they are chasing, and they call wins on day three of a two-week test. With the noise inherent in auction-based revenue, an underpowered test will produce false positives constantly. Teams then build a roadmap on results that would not replicate, scaling changes that do nothing and killing changes that worked. Bad experiment design is worse than no experiments because it manufactures false confidence.
Signal loss makes conversion-based experiments untrustworthy
Cookie deprecation, ATT, and platform signal loss have degraded the conversion data most AdTech experiments measure against. When 30 to 60 percent of conversions are under-counted and the loss is not random across surfaces, a conversion-based A/B test can show a difference that is an artifact of measurement, not behavior. Teams that have not adapted their experiment methodology to the post-cookie reality are testing against a moving, biased target and do not realize their results are noise.
There is no prioritization, so the team tests whatever is easy
Without a structured backlog, growth experiments default to whatever is quick to build rather than what would move the number most. The team runs ten button-color tests and never tests the pricing model, the activation flow, or the publisher value proposition – the things that actually drive AdTech revenue. Experiment effort scatters across low-impact ideas, the win rate looks fine on tiny effects, and the metrics that matter never get touched. Effort is high and learning is low.
How We Help
We start by auditing how the team makes decisions today. In the first 30 days we review the last two quarters of shipped changes, ask how each was decided and measured, and find where conviction is standing in for evidence. For an AdTech company this surfaces the high-stakes, never-tested decisions – pricing and take rate, bidding logic, onboarding – and the broken or underpowered tests that produced false confidence. We come out with a clear picture of where experimentation would change outcomes most.
Strategy is building the experiment backlog and the prioritization model. We replace test-whatever-is-easy with a ranked queue scored by expected impact, confidence, and effort, weighted toward the levers that actually move AdTech revenue. We define the primary metrics each experiment will move and, critically, we design the measurement approach up front – because in AdTech the hardest part is not running the test, it is measuring it honestly when signal is degraded. The backlog becomes the growth roadmap, and it is ordered by what will teach the most.
Execution installs the experimentation discipline. We set up or fix the A/B infrastructure – proper randomized assignment, power analysis so tests are sized for the effect they chase, and pre-registered success criteria so nobody calls a win on day three. For the decisions that cannot be cleanly A/B tested at the user level – many AdTech revenue changes – we design geo-based or holdout-based incrementality tests instead. We run the first wave of experiments alongside the team so the methodology sticks rather than living in a doc.
Measurement is built to survive signal loss. We model conversions where deterministic signal is missing, run holdout-based incrementality for revenue changes that conversion tracking cannot measure cleanly, and we are explicit about confidence intervals so the team stops treating noise as signal. We report on experiment velocity, win rate, and the cumulative measured lift from shipped experiments. A growth experimentation program works when the team kills its own bad ideas quickly and scales the proven ones with confidence.
What makes this different from an agency that sells A/B testing tools is that we install the decision-making discipline, not just the tooling. We embed for a quarter, run real experiments on real revenue levers, and train the team to design and read tests correctly. The deliverable is a team that defaults to testing instead of arguing.
What we deliver
- Prioritized experiment backlog scored by impact, confidence, and effort
- A/B infrastructure with randomized assignment, power analysis, and pre-registered criteria
- Incrementality test designs (geo and holdout) for revenue levers that cannot be cleanly A/B tested
- Signal-loss-resilient measurement approach with modeled conversions and explicit confidence
- Experiment program dashboard tracking velocity, win rate, and cumulative shipped-experiment lift
In auction-based AdTech revenue, an underpowered A/B test does not give you no answer – it gives you a wrong answer that feels like a win. Most teams are scaling false positives and never finding out.
Our Methodology
Our experimentation build runs as a 90-day program installation. Phase one audits how the team decides today: we review two quarters of shipped changes, find the high-stakes decisions made on conviction, and identify the broken or underpowered tests that manufactured false confidence. We map where honest experimentation would change AdTech outcomes most.
Phase two builds the prioritized backlog and the measurement design. We score every candidate experiment by impact, confidence, and effort, weighted toward the levers that move AdTech revenue – pricing, bidding logic, activation. For each one we design the measurement up front, choosing user-level A/B where it works and geo or holdout incrementality where signal loss or revenue mechanics make user-level testing dishonest.
Phase three installs the discipline and hands off. We stand up or fix the A/B infrastructure with proper assignment, power analysis, and pre-registered success criteria, then run the first wave of experiments alongside the team. Unlike a tooling vendor that sells you a testing platform and leaves you to misuse it, we build the decision-making muscle – the team learns to size, run, and read experiments correctly, and to trust the result even when signal is degraded.
How We Work
Initial engagements run 3 to 4 months. The first 30 days are the decision audit and backlog construction with your growth and product leads. Days 31 to 60 stand up or repair the A/B infrastructure, design the incrementality tests for revenue levers, and launch the first experiments. Days 61 to 90 run the first full wave, read the results together, and train the team to operate the program independently.
Our team includes an experimentation lead who owns the program and methodology, a growth strategist who builds and prioritizes the backlog, and an analyst who handles power analysis, incrementality design, and readouts. From your side we need a growth PM who will own the backlog after handoff, engineering time to ship the experiments, and access to the conversion and revenue data. We design the tests and read them; your team builds and ships them.
Weekly experiment reviews track what is live, what is reading out, and what the results mean. Monthly business reviews tie the program to outcomes: experiment velocity, win rate, and the cumulative measured lift from experiments the team shipped. Most AdTech companies are running statistically sound experiments within 45 days and have a self-sustaining experiment cadence by day 90, with the compounding learning advantage building over the following quarters.
- More high-stakes decisions made by test instead of conviction
- Fewer false-positive wins scaled because tests are now properly powered
- Trustworthy experiment results despite cookie and ATT signal loss
- Higher cumulative lift as the team kills bad ideas fast and scales proven ones
If your adtech company needs growth experimentation leadership, we should talk.
Expand your marketing team output with our experts
Let us take a custom approach to your growth goals by assembling and leading the best-in-class marketing team to support your next stage.
Frequently asked questions
How much does a growth experimentation engagement cost for AdTech companies?
Most AdTech experimentation engagements run between $20K and $50K per month depending on the maturity of your existing testing infrastructure and how much measurement work the revenue levers require. That is less than hiring a dedicated experimentation lead and analyst full time, and you get a running program plus a trained team rather than a hiring ramp.
How long before we see results from a growth experimentation program?
Statistically sound experiments are typically running within 45 days, so the team starts generating trustworthy results in the second month. A self-sustaining cadence with a prioritized backlog is in place by day 90.
How does the experimentation team integrate with our growth and engineering staff?
We design and read the experiments while your engineering team builds and ships them, and we pair closely with your growth PM who will own the backlog after we leave. We embed in your existing growth cadence rather than running a parallel process.
What makes Winston Francois different from a traditional experimentation or CRO agency?
CRO agencies sell A/B tests on landing pages and report on conversion-rate lifts that often will not replicate. We install the decision-making discipline across the whole growth program, including the high-stakes revenue levers – pricing, bidding, activation – that agencies never touch.
How do you measure ROI from a growth experimentation engagement?
We track experiment velocity, win rate, and the cumulative measured lift from shipped experiments, plus the share of major decisions now made by test instead of conviction. The headline is how much proven revenue impact the program produces and how much waste it prevents by killing bad ideas before they scale. Most AdTech companies see clear program ROI within two quarters as proven wins accumulate and false positives stop reaching production.
What type of AdTech company is the right fit for this service?
Companies with enough traffic and revenue volume to power experiments – a DSP, SSP, exchange, or measurement platform – that currently ship major changes on conviction and cannot prove what worked. Series A through growth-stage companies with a product and engineering org but no real experimentation discipline see the strongest fit. The first step is a decision audit that reviews recent shipped changes and shows where testing would have changed the outcome.
