Honest systems, different numbers

Part 3 of 4 (8 minute read)| By Aarez Osman, Research Fellow

Key takeaways

We searched government reports, legislative reviews, earnings commentary, and court filings for one pattern: two legitimate systems in the same institution measuring the same reality and producing different numbers. The pattern is common, and it sorts into three types.

Source seams: systems draw on different raw material. The US measures its economy twice, and the two measures have differed by as much as 3.5 percent of GDP.

Definitional seams: systems apply different rules to the same data. Each ad platform's attribution is internally consistent; together, they have been documented claiming 150 to 200 percent of actual orders.

Causal seams: systems agree with each other and are wrong together. Uber turned off $100 million of $150 million in annual ad spend and saw no change in app installs. No consistency check could have found this. Only intervention did.

Each type needs a different instrument. For CMOs, CROs, and COOs, the third is the most expensive and the least visible.

At Acme, the hypothetical B2B software company followed through this series, three systems all agree that branded paid search drives about 31 percent of demo requests. The search platform says so. The attribution model says so. The CRM's lead-source field says so. The CMO is preparing to increase branded search spend by $1.2 million.

Before she does, the team pauses branded search for four weeks in two of Acme's eight sales regions. Demo requests in those regions fall by 6 percent.

The company is invented. The pattern is real, and it is the most important of the three we found. To see why, it helps to start with the two kinds of disagreement that are easier to see.

Source seams: same reality, different raw material

The United States measures its economy twice. Gross domestic product adds up spending. Gross domestic income adds up the incomes earned producing that output. By accounting identity they are the same quantity. In practice they never match, because they are built from largely independent source data (BEA).

The gap, officially the statistical discrepancy, is substantial. It has averaged 0.8 percent of GDP since 1947 and has been as large as 2.7 percent (BLS Monthly Labor Review). In 2014, measured GDI exceeded GDP by about $200 billion (Council of Economic Advisers). In the first quarter of 2022, Treasury reported the gap at 3.5 percent, the largest on record, at a moment when the two measures told different stories about the recovery (U.S. Treasury).

Even the size of the disagreement moves. In mid-2024 the discrepancy stood at 2.7 percent of GDP; when BEA incorporated new IRS data on corporate profits that September, GDI was revised up and the gap fell to 0.3 percent (Council of Economic Advisers). The disagreement itself has vintages, and any system that tracks it has to remember which version a decision was made on.

The institutional response is instructive. Since 2015, BEA has published the average of GDP and GDI, and its own analysis finds the average revises less than either input (BEA Survey of Current Business). The Philadelphia Fed publishes a statistically weighted alternative (Federal Reserve). These are fusion steps: a better single number from two honest ones. What blending cannot do is say where the disagreement lives. BEA notes the average "cannot be distributed among the major components" (BEA). The BLS review cites William Nordhaus's "two map problem": both measures will always differ from each other and from a true level of output that is unknowable (BLS).

Inside a company, the same seam is smaller and costs more. Acme's CRM shows $11.5 million closed-won for the quarter. The ERP shows $10.1 million booked. Both are correct: the CRM records a deal when the rep marks it won, the ERP when a signed order form and purchase order arrive. The $1.4 million difference is mostly three enterprise deals stuck in customer procurement. Averaging the two, as national accountants do, gives $10.8 million, a number that matches nothing in either system and hides the fact that the entire gap sits in three accounts the CRO could name. For the COO, who staffs onboarding against bookings, those three accounts are the forecast risk.

Definitional seams: same data, different rules

On December 22, 2020, Austin Public Health reported 672 new COVID-19 cases in Travis County. The Texas Department of State Health Services reported 537 for the same county (Texas Senate HHS Interim Report).

The Texas Senate's interim review traced the discrepancies to timing (some counties required hospitals to report to them directly as well as to the state), to calculation choices (the time window, whether confirmed and probable cases were combined, how jurisdiction was assigned, which data feeds were used), and to routing errors that delayed cases on their way to the right office. Older cases arriving in backlogged batches were added to statewide totals but excluded from new-case counts: on December 21, the state reported 8,107 new confirmed cases alongside 902 older ones (Texas Senate).

Every item on that list is a restriction map. Neither number was wrong.

Advertising measurement offers a sharper version, because the definition was buried inside a single metric. From 2015 to 2016, Facebook calculated average video watch time while disregarding views under three seconds, which pushed the reported averages up (Fortune). Advertisers alleged the averages were inflated by 150 to 900 percent, and Facebook, which denied wrongdoing, agreed to a $40 million settlement that received final approval in June 2020 (MediaPost, Cohen Milstein). One choice about the denominator, invisible on the dashboard, moved the number by multiples.

The multi-platform version is the clearest case of local truth adding up to global fiction. Each ad platform applies its own attribution window and its own definition of a touchpoint, so one purchase touched by search, social, display, and email can be claimed four times. The measurement firm C3 Metrics, which sells deduplicated attribution and so has a commercial interest in the point, puts it precisely: "Each of these rules is internally consistent. Collectively, they guarantee overlap." It reports platform-claimed conversions documented at 150 to 200 percent of actual orders (C3 Metrics). In sheaf terms, every local section is valid, and their sum is not a global section of anything.

Acme has this problem in miniature. Its paid social platform reports 410 demo requests for the quarter, search 380, webinars 290, email 240: 1,320 in total. The CRM shows 720 demo requests from every source combined. The CMO's channel report overstates reality by 83 percent, and when the CRO asks why marketing claims more demos than sales has ever seen, both are telling the truth.

Figure 3. Acme (hypothetical). Left: a definitional seam, where each platform's claim is valid by its own rule but the sum exceeds actual demo requests by 83 percent. Right: a causal seam, where three systems agree on branded search's contribution and a holdout measures a fraction of it.

Definitional seams are the most common in business and the most tractable: once the maps are written down, the disagreement can be explained line by line.

Causal seams: when the systems agree and are wrong

The third type is different in kind.

In 2017, Uber's head of performance marketing, Kevin Frisch, began switching off ad networks after Uber ads appeared on a site the company had blacklisted. He cut about 10 percent of rider-acquisition spend, and acquisition did not fall. He kept going. "We turned off two-thirds of our spend," he later said. "We turned off 100 million of annual spend out of 150, and basically saw no change" in rider app installs. Installs that had been credited to paid channels simply showed up as organic (Adotat). The investigation that followed found evidence of attribution fraud, including an app with roughly a thousand monthly active users credited with about 350,000 Uber installs (Anura).

The same year, Procter & Gamble cut more than $200 million in digital advertising after viewability data showed, among other things, an average view time of 1.7 seconds for mobile feed ads (Reuters). P&G's CFO told analysts the cuts did not produce "a reduction in the growth rate," showing the spending had been "largely ineffective" (MediaPost).

Look at what Uber's measurement looked like before the shutdown. Paid installs plus organic installs equaled total installs. The attribution system, the ad network reports, and the install counts were mutually consistent. Any consistency check would have passed. The error lived inside a shared restriction map, the rule deciding which installs counted as paid. The system had reached a global section that did not correspond to reality.

Acme's branded search is the everyday version. Three systems agreed on 31 percent because they shared one map: whoever clicked last gets the credit. Most people clicking the ad were already searching for Acme by name. The holdout showed 6 percent. The CMO redirects the $1.2 million. The CRO, who had been counting on those demos in his pipeline coverage model, revises his coverage ratio ahead of the next forecast call.

This is the formal version of Part 1's warning. Consistency across systems shows that they agree. It cannot show that they are right. The only thing that exposed the Uber error was turning spend off and watching whether the world changed.

Each seam needs a different instrument

Source seams call for measurement that preserves the residual, so persistent structural disagreement can be tracked and attributed.

Definitional seams call for an explicit record of the maps: every definition that translates a number across a boundary, with its owner and history, so each disagreement can be traced to a specific definition.

Causal seams call for something sheaf consistency cannot provide on its own: a standing practice of intervention. Holdouts, spend pauses, and natural experiments are the only way to test whether agreement has been earned. The most mature version we found is at Amazon, whose multi-touch attribution system calibrates machine-learning models against a database of "hundreds of thousands" of randomized controlled trials. Its authors state the trade-off directly: observational models scale but may be biased, while experiments are unbiased but noisy (Lewis et al., Amazon Ads Multi-Touch Attribution). Neither is treated as the whole truth. One disciplines the other.

A note on the examples

The public cases come from the sources cited. Several are secondary or come from interested parties: the Uber figures come from public interviews and trade coverage, and the C3 Metrics range is vendor-reported. Acme is hypothetical, and its figures are illustrative throughout.

None of these cases involved anyone being dishonest. The disagreement was what happens by default when several honest systems measure the same reality through different definitions, and nobody checks whether those definitions agree at the seams or whether their agreement holds up when the world is pushed. That raises the obvious next question: if the pattern is this common and the mathematics exists, has someone already done this work?

Previous: Part 2, "The math came first" · Next: Part 4, "Where the gap actually is"

References

Bureau of Economic Analysis. Summary of Results, 2024 Annual Update. bea.gov

BLS Monthly Labor Review. GDP, GDI, and GDO: an evaluation of output measures for productivity analysis, 2026. bls.gov

BEA Survey of Current Business. Revisions to GDP, GDI, and their major components, 2021. apps.bea.gov

Council of Economic Advisers. Gross Domestic Output (GDO) issue brief, 2015. obamawhitehouse.archives.gov

U.S. Treasury. Measuring the Strength of the Recovery. home.treasury.gov

Federal Reserve. Do national account statistics underestimate US real output growth? federalreserve.gov

Council of Economic Advisers. Revisions Show US Economy Grew Faster, 2021–23, 2024. bidenwhitehouse.archives.gov

Texas Senate Committee on Health and Human Services. Interim Report to the 87th Legislature. senate.texas.gov

Fortune. Facebook to pay $40 million to settle lawsuit over inflated video data, 2019. fortune.com

MediaPost. Facebook To Pay $40 Million To Settle Lawsuit Over Video Metrics. mediapost.com

Cohen Milstein. LLE One, LLC v. Facebook. cohenmilstein.com

C3 Metrics. Every Platform Claims Every Conversion. c3metrics.com

Adotat. Uber Chief Claims Most Performance Marketing is Pure Fraud, 2021. adotat.com

Anura. The Uber Ad Fraud Story. anura.io

Reuters. P&G says cut digital ad spend by $200 million in 2017. reuters.com

MediaPost. P&G Makes Its Numbers, Slashes Ad Expenses. mediapost.com

Lewis, R., Zettelmeyer, F., Gordon, B. R., et al. Amazon Ads Multi-Touch Attribution, 2025. arxiv.org