A new study of e-cigarettes’ efficacy in quitting smoking has not only pitted some of vaping’s most outspoken scientific supporters against among its fiercest academic critics, but additionally illustrates many of the pitfalls facing researchers on the topic and those – including policy-makers – who must interpret their work.
The furore has erupted spanning a paper published in The Lancet Respiratory Medicine and co-authored by Stanton Glantz, director from the Center for Tobacco Control Research and Education at the University of California, San Francisco, plus a former colleague – Sara Kalkhoran, now of Harvard Medical School, who is actually named as first author but does not enjoy Glantz’s fame (or notoriety) in tobacco control and vaping circles.
Their research sought to compare the success rates in quitting combustible cigarettes of smokers who vape and smokers who don’t: put simply, to discover whether use of e-cigs is correlated with success in quitting, that might well mean that vaping can help you stop trying smoking. To do this they performed a meta-analysis of 20 previously published papers. That is certainly, they didn’t conduct any new information directly on actual smokers or vapers, but rather tried to blend the final results of existing studies to see if they converge over a likely answer. It is a common and well-accepted strategy to extracting truth from statistics in lots of fields, although – as we’ll see – it’s one fraught with challenges.
Their headline finding, promoted by Glantz himself online in addition to by the university, is the fact that vapers are 28% more unlikely to prevent smoking than non-vapers – a conclusion which may suggest that vaping is not only ineffective in smoking cessation, in fact counterproductive.
The result has, predictably, been uproar from the supporters of Free E Cig Trial within the scientific and public health community, specifically in Britain. Among the gravest charges are those levelled by Peter Hajek, the psychologist who directs the Tobacco Dependence Research Unit at Queen Mary University of London, calling the Kalkhoran/Glantz paper “grossly misleading”, and also by Carl V. Phillips, scientific director from the pro-vaping Consumer Advocates for Smoke-Free Alternatives Association (CASAA) inside the Usa, who wrote “it is clear that Glantz was misinterpreting the info willfully, rather than accidentally”.
Robert West, another British psychologist and also the director of tobacco studies with a centre run by University College London, said “publication with this study represents an important failure in the peer review system within this journal”. Linda Bauld, professor of health policy in the University of Stirling, suggested the “conclusions are tentative and sometimes incorrect”. Ann McNeill, professor of tobacco addiction inside the National Addiction Centre at King’s College London, said “this review is not really scientific” and added that “the information included about two studies which i co-authored is either inaccurate or misleading”.
But what, precisely, are definitely the problems these eminent critics discover in the Kalkhoran/Glantz paper? To answer a number of that question, it’s essential to go beneath the sensational 28%, and look at that which was studied and just how.
Meta-analysis is a seductive idea. If (say) you might have 100 separate studies, every one of 1000 individuals, why not combine them to create – ultimately – a single study of 100,000 people, the results that ought to be significantly less vunerable to any distortions that might have crept into someone investigation?
(This might happen, as an example, by inadvertently selecting participants having a greater or lesser propensity to give up smoking because of some factor not considered through the researchers – a case of “selection bias”.)
Obviously, the statistical side of a meta-analysis is pretty more sophisticated than simply averaging out your totals, but that’s the typical concept. And also from that simplistic outline, it’s immediately apparent where problems can arise.
If its results should be meaningful, the meta-analysis has to somehow take account of variations in the appearance of the patient studies (they could define “smoking cessation” differently, for example). If this ignores those variations, and attempts to shoehorn all results into a model that a number of them don’t fit, it’s introducing its own distortions.
Moreover, in the event the studies it’s according to are inherently flawed in any way, the meta-analysis – however painstakingly conducted – will inherit those same flaws.
It is a charge made by the Truth Initiative, a U.S. anti-smoking nonprofit which normally takes an unwelcoming view of e-cigarettes, regarding a previous Glantz meta-analysis which will come to similar conclusions to the Kalkhoran/Glantz study.
In a submission last year to the U.S. Food and Drug Administration (FDA), addressing that federal agency’s require comments on its proposed electronic cigarette regulation, the Truth Initiative noted it had reviewed many studies of e-cigs’ role in cessation and concluded that they were “marred by poor measurement of exposures and unmeasured confounders”. Yet, it said, “many of these happen to be included in a meta-analysis [Glantz’s] that claims to reveal that smokers who use e-cigarettes are not as likely to quit smoking compared to people who do not. This meta- analysis simply lumps together the errors of inference from the correlations.”
Additionally, it added that “quantitatively synthesizing heterogeneous studies is scientifically inappropriate and the findings of such meta-analyses are therefore invalid”. Put bluntly, don’t mix apples with oranges and be prepared to receive an apple pie.
Such doubts about meta-analyses are far from rare. Steven L. Bernstein, professor of health policy at Yale, echoed the Truth Initiative’s points as he wrote within the Lancet Respiratory Medicine – the identical journal that published this year’s Kalkhoran/Glantz work – that this studies a part of their meta-analysis were “mostly observational, often without control group, with tobacco use status assessed in widely disparate ways” though he added that “this is no fault of [Kalkhoran and Glantz]; abundant, published, methodologically rigorous studies just do not exist yet”.
So a meta-analysis could only be as effective as the study it aggregates, and drawing conclusions from this is just valid in the event the studies it’s based on are constructed in similar ways to the other person – or, a minimum of, if any differences are carefully compensated for. Obviously, such drawbacks also pertain to meta-analyses which can be favourable to e-cigarettes, such as the famous Cochrane Review from late 2014.
Other criticisms of the Kalkhoran/Glantz work exceed the drawbacks of meta-analyses generally speaking, and focus on the specific questions caused from the San Francisco researchers as well as the ways they made an effort to answer them.
One frequently-expressed concern continues to be that Kalkhoran and Glantz were studying a bad people, skewing their analysis by not accurately reflecting the true number of e-cig-assisted quitters.
As CASAA’s Phillips highlights, the e-cigarette users in the two scholars’ number-crunching were all current smokers who had already tried e-cigarettes once the studies on their quit attempts started. Thus, the analysis by its nature excluded people who had started vaping and quickly abandoned smoking; if these people exist in large numbers, counting them could have made e-cigarettes seem an infinitely more successful route to quitting smoking.
Another question was raised by Yale’s Bernstein, who observed that does not all vapers who smoke are attempting to quit combustibles. Naturally, people who aren’t wanting to quit won’t quit, and Bernstein observed that whenever these people kndnkt excluded from your data, it suggested “no effect of e-cigarettes, not too e-cigarette users were more unlikely to quit”.
Excluding some who did manage to quit – and then including individuals who have no goal of quitting anyway – would most likely manage to impact the outcome of a study purporting to measure successful quit attempts, even though Kalkhoran and Glantz debate that their “conclusion was insensitive to a variety of study design factors, including whether the study population consisted only of smokers considering smoking cessation, or all smokers”.
But additionally there is a further slightly cloudy area which affects much science – not just meta-analyses, and not simply these particular researchers’ work – and, importantly, is frequently overlooked in media reporting, as well as by institutions’ publicity departments.