When a Survey Repeats Itself, What Is It Actually Comparing

Gallup published two loneliness figures within three weeks of each other in October 2023. One, from the Gallup-Meta State of Social Connections survey, reported that 24% of people worldwide feel very or fairly lonely. The other, drawn from the same underlying dataset but analysed separately in a follow-up piece, described connectedness patterns that do not simply mirror the loneliness figure — Gallup’s own analysis notes that connectedness and loneliness are not straightforward inverses of one another. A reader skimming both could be forgiven for wondering whether Gallup had changed its mind about the state of the world in three weeks. It had not. It had asked, in effect, two different questions and was reporting on both.

This is the ordinary condition of survey research on loneliness, and it is worth examining on its own terms, because the instinct when comparing numbers over time is to treat a repeated survey brand as if it were a single, stable ruler. It usually is not.

The same brand, three different bases

Gallup’s loneliness reporting since 2023 illustrates the problem cleanly because it comes from one organisation and is therefore about as favorable a case as exists for comparability. The October 2023 Gallup-Meta survey covered roughly 1,000 people aged 15 and older in each of 142 countries, fielded between June 2022 and February 2023, and excluded China — about 77% of the world’s adult population. It found 24% reporting loneliness “very” or “fairly” often, with adults 65 and older at 17% and young adults 19-29 at 27%.

A year later, Gallup’s October 2024 update reported that more than one in five adults worldwide feel lonely a lot of the previous day. That is not the same question. “Very or fairly lonely,” an evaluative frequency judgment, is not the same construct as “lonely a lot of the previous day,” a reference to a specific recent experience. Both are legitimate ways to ask about loneliness. Neither is wrong. But treating the shift from 24% to “more than one in five” as evidence of a one-point decline mistakes an artifact of question wording for a trend in the world.

The same caution applies to Gallup’s separate workplace-specific tracking. The 2024 State of the Global Workplace report found that one in five employees worldwide report loneliness a lot of the previous day, with fully remote workers at 25%, hybrid at 21%, and on-site at 16%. That figure is drawn from a working population using a day-referenced item, distinct from the general population survey using an evaluative-frequency item. Comparing the workplace figure to the general-population figure to conclude that work is somehow less lonely than life generally would compound the same error twice over: different population, different question, different year.

What “up 7 points” actually meant

Cigna’s 2020 loneliness report offers an instructive case in the other direction. It reported that 61% of U.S. adults sometimes or always feel lonely, up seven percentage points from the prior year’s Cigna survey. Within Cigna’s own series, using the same instrument on comparable samples year over year, a seven-point movement is a real and reportable trend. The mistake would be exporting that number outside its own series — comparing Cigna’s 61% “sometimes or always lonely” to, say, the Surgeon General’s 2023 estimate that approximately half of U.S. adults report experiencing loneliness. These are not competing measurements of the same underlying rate that disagree by eleven points. They are different instruments, asked at different reference frequencies, of different samples, in different years. The Surgeon General’s advisory draws on a range of underlying data rather than a single repeated series, which makes it useful for establishing that loneliness is common and consequential but poorly suited to answering whether the rate moved.

AARP’s 2018 survey of adults 45 and older is worth flagging as a comparative exception, not because it is perfectly comparable to others but because it names its own comparability strength: it used the 20-item UCLA Loneliness Scale, the standard academic instrument, rather than a proprietary question. That single choice makes its finding — one in three adults 45 and older report loneliness — more portable across the academic literature than most bespoke survey items, even though it still cannot be compared directly across time within its own series, since AARP did not repeat this exact survey wave.

The gap between when data was collected and when it was published

A separate comparability problem sits inside single reports rather than between them. The CDC’s Morbidity and Mortality Weekly Report on loneliness, published in June 2024, presents estimates drawn from 2022 survey data. Reading it as a 2024 snapshot rather than a description of conditions two years prior changes what it can support. This is not a flaw in the report — federal surveillance data routinely takes a year or more to process and publish — but it means a claim like “loneliness in the United States, as of this year” cannot rest on that report alone without noting the two-year lag.

A 2023 methods review in BMC Public Health flagged exactly this class of problem as a structural weakness of the field: inconsistent measurement across studies makes it difficult to compare findings even when the underlying construct is nominally the same. That review’s argument was not that any individual survey is unreliable, but that the field lacks a shared, repeated, validated instrument fielded consistently enough to support the kind of trend claims that headlines routinely draw from it.

What would actually support a trend claim

A defensible over-time comparison needs three things held constant: the same instrument (identical item wording and response options), the same reference period (a lifetime evaluation is not a “yesterday” report), and a comparable sampling frame across waves. Where an organisation changes any of the three — as Gallup did between its 2023 and 2024 releases, and as is common across the field generally — the resulting numbers describe two separate snapshots, not two points on a single line.

None of this means the underlying surveys are poor instruments. It means the responsible use of them is narrower than the headlines built on them usually suggest: cite a figure for what it measured, in the year it was collected, against the population it sampled, and resist the pull toward a trend line the data was never built to support.