Your ROAS proves nothing: running incrementality tests on a LATAM budget

Add up the ROAS that Meta, Google and TikTok each report for the month and the total will almost certainly exceed the revenue in your bank account. That is not a glitch. It is what happens when every platform claims credit for every sale it touched. The question that actually pays salaries is not how much the dashboard says you sold, but what would have happened if you had not advertised at all. No dashboard answers that. Only an experiment does.

Platform ROAS is an attendance sheet

Attribution answers a comfortable question: which ads did buyers see? Incrementality answers the uncomfortable one: how many of those purchases would not have happened without the ad? They are different measurements, and only the second one shows up in the bank.

The cleanest evidence is still eBay. Blake, Nosko and Tadelis switched off branded search advertising across entire US markets and published what happened: traffic and sales held essentially flat, because someone typing "ebay" into Google arrives through the organic result anyway. For years that line item had been reported as wildly profitable. Much of it was a toll paid on traffic the company already owned.

Uber reached the same conclusion the painful way. When it audited its app-install spending in 2017, during the dispute with its media agency, it shut off tens of millions of dollars of digital advertising and installs did not drop the way the dashboards predicted. The lesson is not that advertising does not work. It is that nobody knew which part worked, because nothing had ever been measured against a no-ads scenario.

Why the official lift tests rarely fit

Meta has Conversion Lift and Google has geo experiments. They are legitimate tools, built for accounts with volume: they need enough conversions per cell for the result to rise above noise, and that threshold is high. At 40 or 60 conversions a month, which describes a great many accounts in Guatemala, Panama or Venezuela, the test returns a confidence interval wide enough to support whatever conclusion you walked in with.

That is usually where teams give up and go back to platform ROAS, which at least produces a flattering number. That is the mistake. You do not need the platform test. You need a cruder, cheaper experiment that belongs to you.

The do-it-yourself geo holdout

The method is old and it holds up: turn the advertising off in part of the country, leave it running in the rest, and compare total business revenue rather than platform-reported revenue.

• Group markets, not individual cities. Metro Guatemala City against the western highlands. Two or three Mexican states against two or three with a similar profile. Whole regions in Chile.
• Match on history, not instinct. Pull 8 to 12 weeks of regional revenue and build the groups so the curves track each other. Groups that did not move together before will not compare afterwards.
• Go fully dark in the control. Halving the budget is not a holdout. It is a different campaign, and it measures nothing.
• Run it for three or four weeks. Shorter does not cover the purchase cycle. Longer starts costing you real revenue.
• Measure in your source of truth. The ERP, Shopify, the sales team spreadsheet, the WhatsApp order log. Never Ads Manager.

What you want is a subtraction: revenue per thousand residents in the advertised group minus the dark group. Divide that by the spend and you have incremental ROAS. It nearly always lands below the reported figure, sometimes far below, and it is the only version you can defend in a board meeting.

One thing almost nobody says out loud: an ambiguous result is not a failed test. It tells you the effect is smaller than your ability to detect it, which is itself useful information about how much that line item deserves.

Where the waste usually hides

Campaigns do not all overstate themselves equally. The usual suspects, in order of likelihood:

• Branded search. If you already hold the top organic result, you had most of that paid click for free. Switch it off by region for a week and watch total sessions, not paid ones.
• Hot-cart retargeting. Someone who abandoned a cart two hours ago was probably coming back. The ad appears, claims the sale, and you pay for intent that already existed.
• Recent-buyer audiences. Existing customers sitting inside acquisition campaigns, counted as new.
• Catalog campaigns on autopilot. The ones nobody has touched in eight months that always post the best ROAS in the account. That record is exactly what makes them suspect.

Why this matters more now

Two things have changed in the last three years. Modeling now accounts for a growing share of what dashboards report: with less cookie signal and more estimated conversions, the number on screen is increasingly a projection rather than a count. And media costs in the region climb every peak season, so paying a higher price for traffic that was already yours scales the error instead of fixing it.

The right reflex is not to cut. It is to reallocate. Every quetzal, peso or dollar a holdout proves inert is budget freed up for prospecting, for new creative, or for the markets where people genuinely have not heard of you yet.

What to do this week

Do not build a measurement program. Run one test. Take the campaign you are proudest of, the one reporting 8x that nobody dares touch, split the country into two groups matched on your own history, go dark in one of them and watch ERP revenue for three weeks.

The worst outcome is confirming that it works, which tells you how hard you can scale it. The best outcome is freeing up money that has spent months buying customers you already had.