My report, “Bias at the Beeb? A Quantitative Analysis of Slant in BBC Online Reporting” is published this week. The goal of this report was to try to bring a more dispassionate approach to a debate that has become extremely heated in recent years.
In this blog post I’ll quickly outline the results of the two main approaches I use and then respond to some of the most common questions/critiques.
Results on Quantity of Coverage
The first approach I use compares the quantity of coverage given to a set of think-tanks by the BBC news website with the amount of coverage these same think-tanks received in the Guardian and Telegraph newspapers. The goal is to examine whether the BBC’s coverage is “more similar”, in a statistical sense, to one newspaper or the other. The key results are as follows:
These results are marginal effects: the question they attempt to answer is “what would be the effect on the level of the BBC’s coverage of a given think tank if one of the newspapers we look at increased its coverage?” The results we find then suggest that BBC coverage is twice as sensitive to Guardian coverage as Telegraph coverage. This is not equivalent to saying that right-wing think tanks receive twice as much coverage as left-wing ones. Unfortunately the discussion of the results in my article in the Sunday Times was not sufficiently clear on this point and even more misleading descriptions of the results have appeared in one or two places since.
3. The regression results shows that, once we control for the number of hits a think tank received in the Guardian during the time period, the number of hits it received in the Telegraph does not have any statistically significant effect on coverage by the BBC.
The important thing to note about this first set of results is that they require very few subjective judgements on our part: we are not making assumptions about the ideologies of the think tanks or even reading the content of the BBC’s articles beyond checking that they are actually about the think-tank of interest.
Results on Tone of Coverage
The second set of results looks at the way the think-tanks in our sample are introduced by the BBC, through examining whether they are given “health warnings” by being labelled with an existing view on a subject (e.g. “which opposes the…”), an ideological position (right-of-centre, left-of-centre, conservative etc.) or a link with a party political figure (e.g. “founded by Margaret Thatcher”, “whose director is a former Labour cabinet minister” etc.).
This approach requires some more in-depth reading of the BBC’s articles to find any instances of health warnings and so I restrict the sample to five right-of-centre think-tanks (the Centre for Social Justice, the Institute of Economic Affairs, the Centre for Policy Studies, Policy Exchange and the Adam Smith Institute), four left-of-centre think-tanks (the Social Market Foundation, Demos, the New Economics Foundation and the Institute of Public Policy Research) and CentreForum, which has links with the Liberal Democrats.
Responses to Common Critiques/Questions
1. “So What?”
Firstly, I think that the results are interesting in their own right: they provide objective, quantitative evidence of slant in both the amount and tone of the BBC’s coverage of think-tanks. Secondly, the results reinforce the existing anecdotal allegations of bias at the organization and so suggest that these views should be taken more seriously.
2. “Isn’t the CPS even more biased than the BBC? How can we trust you?”
It is inevitable that, if an organization like the CPS criticises the BBC, it will be accused of ulterior motives. That is why I tried to use methods with clear (and I hope) uncontroversial assumptions. By removing much of the subjective element from measuring slant, my hope is that people will read the report, assess the assumptions and then draw their own conclusions.
Questions About the Comparison between the BBC and the Telegraph/Guardian
1. “The BBC’s coverage should be more similar to the Guardian than the Telegraph, one is a reliable news source, the other is outrageously right-wing.”
I think that my choice of reference newspapers is reasonable. The Guardian is the most prominent quality newspaper with an openly left-of-centre editorial line and the Telegraph is the most prominent quality newspaper with an openly right-of-centre editorial line. I would also point to the fact that the Telegraph has a circulation three times that of the Guardian as evidence that it is hardly a fringe publication.
I would also ask anyone who has the reaction “You’ve convinced me that the BBC’s coverage is more similar to the Guardian’s, but that’s how it should be” to acknowledge that this probably means that both the Guardian and the BBC are producing output that fits with their worldview. This may be okay from their point of view, but surely they should respect that that those of a different political persuasion might think differently.
2. “You generate a ranking of think-tanks using their relative coverage in the Guardian/Telegraph. However, some of the think-tanks aren’t where I expect them to be, why should I trust your results?”
We rank the think-tanks in our sample by the relative coverage they receive in the Guardian versus the Telegraph. We do this to check whether the resulting ranking looks plausible so as to test whether it really seems to be the case that newspapers give more coverage to sources they agree with. There are some outliers (in particular, IPPR appears rather right-wing by this measure). However, given that this ranking required no assumptions on our part about the think-tank’s ideological persuasion, we think that it does a pretty good job.
Also, even if the ranking of think-tanks was completely mad, the results would still suggest a similarity between the BBC and the Guardian citations, which would be interesting in itself.
Questions About Results on Health Warnings
1. “How can you equate the Social Market Foundation to some of the right-of-centre think-tanks? The SMF are far less extreme.”
The treatment of the SMF has been the biggest source of contention so far. I would like to first point out that the classification of think-tanks as left-of-centre/right-of-centre is only necessary for our results on health warnings and has no bearing for the first set of results looking at the level of coverage a think-tank receives from the BBC.
Even if one disagrees with the classification of the SMF, and I can see how people might, I think that the other choices about which think-tanks to classify as left-of-centre are substantially less contentious. Most would agree, for example, that the New Economics Foundation is more left-wing than Policy Exchange is right-wing. The broad picture, that right-of-centre think-tanks get more health warnings, remains the same when we exclude the SMF. The results are robust to their exclusion.
This debate also underlines one of the points I make in my article for last week’s Sunday Times: when trying to measure the extent of bias, subjective judgments are best avoided wherever possible in favour of a more value-neutral approach.
2. “Maybe everyone treats right-of-centre sources in this way and so you’re singling the BBC out unfairly? How do the newspapers treat the think-tanks in your sample?”
We took Demos and the CPS as representative left-of-centre and right-of-centre sources and did a quick check of the Guardian and Telegraph’s use of health warnings. Demos received a health warning 4% of the time in the Guardian and 20% of the time in the Telegraph. In contrast, the CPS received a health warning 71% of the time in the Guardian and 22% of the time in the Telegraph. It would be foolish to take too much from this observation, but it is fair to say that it does not support the idea that the BBC’s asymmetric treatment of think-tanks sources is something that is common to all news sources.
3. “The right-of-centre CSJ was referred to as “independent” more often than any other think-tank. Doesn’t this suggest that the BBC is being fair?”
The CSJ was a tricky think-tank to deal with because the BBC nearly always referred to its foundation by Iain Duncan Smith, but then often combined this with another health warning like “right of centre”, or a reference to its “independence”. This led to introductions like “The CSJ, an independent think-tank founded by Iain Duncan Smith”. The result was that it received both more “independents” and more health warnings than anyone else. References to its independence were probably undermined somewhat by the accompanying health warning, but it is impossible to know by how much.