Resources for evaluating statistics stories
Every month we highlight some of the statistics and survey results buried in press releases from the ResponseSource Press Release Wire. Polls are a controversial source for stories: we’ve all read the small print on TV cosmetics ads and realised that 82% of users who would recommend a moisturiser isn’t quite so conclusive when the sample size is 38.
But a survey with a representative sample of a relevant audience, with details about the methodology and background to the research, can form a news story in itself as well as supporting a feature or providing an interesting snippet for your social media. The stories we highlight from the Press Release Wire might fall under “fun facts” as often as they do under serious statistics, so we’ve pulled together some further links for evaluating surveys and polls.
Sample sizes
When a brand quotes its own
consumer surveys, what makes a fair sample size and what conclusions can
they legitimately draw? The Advertising Standards Authority’s guidance on
substantiating surveys and sample claims is here.
Courses and books
The Poynter Institute offers a free self-directed course on understanding and
interpreting polls here
or if you prefer a UK perspective, try the Royal Statistical Society’s (RSS) free
Statistics for Journalists course here.
For even more depth, the European Journalism Centre’s Datajournalism.com offers
free video courses and two comprehensive handbooks here.
Industry associations
In November 2019 IMPRESS and The Market Research Society (MRS) brought out
a handy 11-page guide “to provide journalists with the tools they require to
report on statistical data responsibly”. You can download it here.
The MRS and RSS also joined forces with the PR industry’s CIPR to produce a best practice guide for public relations professionals here.
Polling
The British Polling Council, which promotes professional standards in
public opinion polling and aims to increase understanding of how polls are
conducted, provides specific guidance for journalists here.
Journalist’s Resource has created a list of questions you should ask about every poll here.
The FAQs from YouGov here give plenty of background in to their panel and methodology – for other polling organisations see the British Polling Council’s members list here.
Fact-checkers
Fact-checking organisations will often be investigating the same claims that
catch your eye and offer good insight in to how to evaluate surveys and other
statistical stories. Most of them have blogs or newsletters you can follow but
for examples see Full Fact’s
analysis of a story about GP waiting times or FactCheck
NI’s examination of clams about assaults on school staff.
Margins of error
Journalist’s Resource has created a useful guide here
to help you understand the margin of error in stats and why you should be
cautious with results (and interpretations).
Media policy
Major media outlets tend to have policies on surveys, polls and statistics. The most comprehensive we’ve seen is from the BBC here and more advice on reporting statistics here.
ResponseSource’s top three tips when working with stats:
– Determine the author of the report and interrogate their methodology. Is the sample representative of a population and is the sample size significant? A UK sample typically needs to be above 2,000 to be nationally representative. As well as the resources above, a useful overview can be found here.
– Do your own calculations. It can be worth asking to see the raw data behind the research; not only can you satisfy yourself that the original story is accurate but there may be more: Datajournalism.com’s Data Journalism Handbook points out “Data as such is unwieldy…It needs experienced journalists, who have the stamina to look at often confusing, often boring raw data and “see” the hidden stories in there.”
– Even if you’re confident about your own expertise, an independent point of view can add to your authority. Industry organisations in any sector can often connect you with experts via their media centre and universities will normally provide full details about the experience and specialisms of their academic staff. If you’re short on time, our Journalist Enquiry Service community can connect you with thousands of sources business, finance, technology, retail, health and any other sector – it’s still up to you to evaluate what they say – send a request on https://responsesource.com/send
