>Subject: Meta-analyses >From: Dr John Marks, jma...@ertnet.demon.co.uk >Date: Sat, 1 Apr 1995 09:57:00 GMT >>I want to find out something about meta-analyses?
>Are they just another name for review articles?
>Or are they something which is, statistically speaking, more >sophisticated?
>Any help or useful references gratefully received.
Dear Dr Marks:
Meta-analysis is a set of statistical procedures designed to accumulate experimental and correlational results across independent studies that address a related set of research questions. Unlike traditional research methods, meta-analysis uses the summary statistics from individual studies as the data points. A key assumption of this analysis is that each study provides a differing estimate of the underlying relationship within the population (rho). By accumulating results across studies, one can gain a more accurate representation of the population relationship than is provided by the individual study estimators.
There are a variety of different procedures for conducting a meta-analysis involving the accumulation of individual study statistics and converting them to correlations (r), standardized differences between mean scores (d), p values, or Z-scores (Glass, 1976, 1977; Hunter et al.,1982; Hunter and Schmidt, 1990; Rosenthal, 1991; Smith and Glass, 1977; Smith, Glass and Miller, 1980; Wolf, 1986).
Schmidt, Hunter, and their colleagues (Schmidt and Hunter, 1977; Hunter, et. al., 1982; Hunter and Schmidt, 1990) developed one method of meta-analysis that does not rely on the combination of Z-scores or probability values as the common metric. This procedure uses either r or d as the combinatorial statistic. It progressively corrects the mean r or d and their obtained variances for sampling error and then measurement error and range restriction.
The following references may be helpful:
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd Ed). Hillsdale, NJ: Lawrence Erlbaum Associates. Glass, G. (1976). Primary, secondary and meta-analysis of research. Educational Researcher, 5, 3-8. Glass, G. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379. Huffcutt A., Arthur, W., and Bennett W. (1993). Conducting meta-analysis using the PROC MEANS procedure in SAS. Educational and Psychological Measurement, 53, 119-131. Hunter, J. and Schmidt, F. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Beverly Hills CA: Sage. Hunter, J., Schmidt, F., and Jackson, G. (1982). Meta-Analysis: Cumulating research findings across studies. Beverly Hills CA: Sage. McDaniel, M. (1986). Computer programs for calculating meta-analysis statistics. Educational and Psychological Measurement, 46, 175-177. Mullen, B. and Rosenthal R. (1986). BASIC meta-analysis: Procedures and programs. Hillsdale, NJ: Lawrence Erlbaum Associates. Mullen, B. (1990). Advanced BASIC meta-analysis. Hillsdale, NJ: Lawrence Erlbaum Associates. Rosenthal, R. (1991a). Meta-analytic procedures for social research (rev ed). Beverly Hills CA: Sage. Rosenthal R. (1991b) Meta-analysis: A review. Psychosomatic Medicine, 53, 247-271. Schmidt, F. L. and Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540. Smith, M., and Glass, G. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32,752-760. Smith, M. Glass, G. and Miller, T. (1980). The benefits of psychotherapy. Baltimore: The Johns Hopkins University Press. Wolf, F. (1986). Meta-Analysis: Quantitative methods for research synthesis. Beverly Hills, CA: Sage.
I did not include material by Olkin and Hedges etc (mainly because I have not got around to reading them), but otherwise this covers a fair amount of the field. Also if you have access to a Mac, I've developed a hypercard based meta-analysis program that may be useful. If you are interested, it is available at ftp.stolaf.edu in directory /pub/macpsych. the file is called MetaAnalysis_Stack.sea.hqx.
Regards,
Larry C. Lyons | Dept of Psychology email: Solo...@vt.edu | Virginia Tech, | Blacksburg, VA 24061 - 0436 Tel: (703) 231 - 6581 | Fax: (703) 231 - 3652
My opinion alone, no one else will take responsibility for it! ======================================================== Life is Complex. It has both real and imaginary parts. ========================================================
Dr John Marks (jma...@ertnet.demon.co.uk) wrote: : I want to find out something about meta-analyses? ..
: Any help or useful references gratefully received.
Here are some references which I found useful:
-D.R. Jones: Meta-analysis of observational epidemiological studies: a review. In: Journal of the Royal Society of Medicine Vol 85 March 1992, 165-168. -Malcolm Maclure: Demonstration of deductive meta-analysis: ethanol intake and risk of myocardial infarction. In: Epidemiologic Reviews 1993, 15:2, 328-351. -Larry V. Hedges, Ingram Olkin: Statistical methods for meta-analysis. Academic Press, Boston
THey are review articles that some one with good statistical INSIGHT would carry out.
for Randomized Trial see
O'Rourke K Detsky AS Meta-analysis in medical research: strong encouragement for higher quality in individual research efforts. Journ of Clin Epi Vol 42 No 10 1989 pg 1021-25
for observational studies
Shapiro S. Meta-analysis/Shmeta-analysis. Amer Journ of Epi Vol 140 No 9 1994
In article <D6Coz2....@demon.co.uk>, Dr John Marks <jma...@ertnet.demon.co.uk> writes:
> I want to find out something about meta-analyses?
> Are they just another name for review articles?
No, most researcher respect review articles.
> Or are they something which is, statistically speaking, more > sophisticated?
There are two *major* problems with meta analyses cited by most critics: (1) publication bias (2) the varying quality of the studies used in the analysis.
I always include (3) meta analysis is *always* used when problems (1) and (2) are likely to cause the *most* damage!
> Any help or useful references gratefully received.
Read Begg & Berlin JRSSA, 151(1988), 419-463. This will put any foolish notions out of your head. :-)
BTW, yes I *would* pay some regard to an analysis based on a registry. So, there!
In article <1995Apr4.115...@hnrc.tufts.edu> Jerry Dallal,
je...@hnrc.tufts.edu writes: >There are two *major* problems with meta analyses cited by most >critics: >(1) publication bias >(2) the varying quality of the studies used in the analysis.
>I always include >(3) meta analysis is *always* used when problems (1) and (2) are likely to >cause the *most* damage!
I have to disagree with you about points 1 and 2. There are a number of ways of ensuring that publication bias is not an issue. Generally this issue is called the "File Drawer Problem." It suggests that there is a bias in using only journal articles in a meta analysis, rather than dissertations, studies reported in books, or papers presented at meetings and symposia. There is some justification to this criticism. No matter how exhaustive a literature search is, there will always be some data that will never be included because of a variety of reasons, usually because these excluded studies are languishing in the back of a file drawer in some lab. This raises the possibility that the inclusion of this data into the analysis could drastically alter the results, typically by reducing a significant relationship to not significant. However, my own work has shown that any differences between published and unpublished material is typically fairly trivialtrivial.
One method of dealing with this file drawer problem is to use the Fail Safe N (FSN; Rosenthal, 1991). This measure provides an estimate of the possible number of unretrieved studies with null results which would be required to reduce the statistical significance of an average effect size in a meta-analysis to not significant.
Fail Safe N = (k/2.706)*((k*(mean z)^2)-2.706)
Where k = the number of studies in the analysis, and mean z is the mean effect size expressed as a z score. If the resulting number is equal to or smaller than 5k +10, then you have a problem, otherwise your results are not susceptible to the File Drawer Problem.
In the last meta-analysis I conducted this included no only going over the usual sources (psychlit and ERIC) but also Dissertation Abstracts International, looking at numerous conference proceedings, sending out requests for additional studies via the InterNet and Snail Mail. After collecting all this information and calculating the FSN, the number of "undiscovered"studies that were needed to invalidate my results were in the magnitude of about 9,000. Several of the other meta-analyses I've done had similar results.
The second criticism involves including studies of doubtful methodological quality. Inclusion of studies of dubious methodological quality affects the overall results of the meta-analysis, causing a possible attenuation of the population estimates. Problems of including studies of differing methodological quality can be examined analytically, by creating a coding procedure based on methodological and internal and external validity concerns. This procedure involves having several experts in methodology rate each study using several indices of study quality If methodological quality is related to the effect size, then studies of dubious quality can be justifiably eliminated from the analysis. If methodological quality is found not to influence effect size (as is commonly the case), then there is no problem including these studies in the analysis.
Hence, if these procedures are done correctly, where's the problem. If you pay attention to many of the concerns, and follow proper methodological procedures, your results can be valid. The results become prolematic when some bozo decides to conduct a meta-analysis without having a proper understanding of the assumptions and cautions involved with the methodology.
Regards,
Larry C. Lyons | Dept of Psychology email: Solo...@vt.edu | Virginia Tech, | Blacksburg, VA 24061 - 0436 Tel: (703) 231 - 6581 | Fax: (703) 231 - 3652
My opinion alone, no one else will take responsibility for it! ======================================================== Life is Complex. It has both real and imaginary parts. ========================================================
Larry C. Lyons (solo...@vt.edu) wrote, expressing what seemed to me like optimism: ... : Hence, if these procedures are done correctly, where's the problem. If : you pay attention to many of the concerns, and follow proper : methodological procedures, your results can be valid.
Maybe he has seen a different set of `typical' meta-analyses, than the ones that have been brought to my attention. The comment which came next seemed more germane:
: The results become : prolematic when some bozo decides to conduct a meta-analysis without : having a proper understanding of the assumptions and cautions involved : with the methodology.
I particularly like to note the Caution, that you cannot get an useful or meaningful `average effect' when you average two results which are strongly significant in OPPOSITE directions. That is, `Plus one' and `Minus one' do not average out as `zero' but rather, as `Inconsistent results yet to be explained,' and probably NOT suitable material for a Meta-analysis. -- This seems to me to be basic, essential, common sense, but people get published without it.
For more explanation, I point to an analogy with correlations; there is Fisher's transformation available for accurately averaging Pearson product-moment correlations, but there is also Bartlett's test on heterogeneity, which may tell you that those correlations are varied enough that you should not try to describe them with any average.
Rich Ulrich, biostatistician Internet: wpi...@vms.cis.pitt.edu Western Psychiatric Inst. and Clinic Univ. of Pittsburgh
>There are two *major* problems with meta analyses cited by most >critics: >(1) publication bias
Perhaps you could explain why publication bias is a problem specifically of meta-analysis, and not of empirical research in general? My point is that if you believe that the group of studies whose results are to be combined represents a biased sample, then any single one of those studies represents a biased estimator just as much as does the mean of the collected group. Hence, every time you read a journal article that reports a statistical result, you are, at least potentially, affected by publication bias.
Now when you are only looking at the one study, you have no tools to help determine whether publication bias is present. On the other hand, when you are looking at a collection of studies, there is a great deal of evidence available from comparison of the empirical sampling distribution with the theoretical sampling distribution. To give a ridiculously simple example, suppose you are looking at the distribution of 50 Fisher's-Z-transformed correlation coefficients. If they were drawn from a population with a mean of, say, 0.2, then the theoretical sampling dist'n is N( 0.2, 1/(n_i-3)). Given typical sample sizes, some of the tail of that distribution would be expected to cross zero. If the actual distribution in your sample looks more like a truncated normal distribution, with a paucity of negative results, then you might reasonably conclude that a bias against publication of negative results was present, and adjust your estimate accordingly. On the other hand, if you merely looked at one study, you would have no evidence of its bias.
In practice, the best techniques for addressing publication bias are more formal and sophisticated than what I've just described (see, for example, recent work by Colin Begg, or by Larry Hedges and me). The point is, though, that the problem of publication bias is potentially a strength of meta-analysis, not a weakness.
<`Plus one' and `Minus one' do not average out as `zero' but rather, <as `Inconsistent results yet to be explained,' and probably NOT <suitable material for a Meta-analysis.
The most important role of Meta-analysis may be to explicitely point out that individual study result are not consistent (or interpretable).
That detecting common effects should be called Meta-analysis and detecting heterogenious effects should be called something else leaves undefined what you are doing when you first start out.
for a more coherent? discussion
O'Rourke K Detsky AS Meta-analysis in medical research: strong encouragement for higher quality in individual research efforts. Journal of Clin Epi Vol 42 No 10 1989 pg 1021-1025
In article <D6Coz2....@demon.co.uk> Dr John Marks <jma...@ertnet.demon.co.uk> writes:
>I want to find out something about meta-analyses? >Are they just another name for review articles? >Or are they something which is, statistically speaking, more >sophisticated? >Any help or useful references gratefully received.
After reading all of these replies, I would add what Glass said when he coined the phrase meta-analysis:
The approach to research integration referred to as "meta-analysis" is nothing more than an attitude of data analysis applied to quantitative summaries of individual experiments. By recording the properties of studies and their findings in quantitative terms, the meta-analysis of research invites one who would integrate numerous and diverse findings to apply the full power of statistical methods to the task. Thus, it is not a technique; rather it is a perspective that uses many techniques of measurement and statistical analysis.
Keep this in mind and avoid the mathematical trivialities generated by people looking for unimportant publications. If you want to know more about the subject and what it really is you can contact me at:
<In article <D6IznB....@utstat.toronto.edu>, orou...@utstat.toronto.edu (Keith O'
<Rourke) writes:
<> <> Would you agree that it is unethical to fund individual studies unless <> there is a registry first set up?
<From: je...@hnrc.tufts.edu (Jerry Dallal) <No. I would be merely willing to put aside some of my distrust of combining <results through meta-analysis if the meta-analysis were based on studies <contained in a registry. Sorry if my post was unclear on that point.
Your reply will allow me to make my view more clear.
I too distrust COMBINING studies through meta-analysis. I attempt to INTERPRET individual studies through meta-analysis. I distrust INTERPRETING individual studies in isolation.
So I do think it is unethical to fund individual studies unless
> there is a registry first set up (or some tracking).
Keith O'Rourke <orou...@utstat.toronto.edu> wrote: ><In article <D6IznB....@utstat.toronto.edu>, orou...@utstat.toronto.edu (Keith O' ><Rourke) writes: ><> Would you agree that it is unethical to fund individual studies unless ><> there is a registry first set up?
It is not just a matter of funding; to do meta-analysis the crude way it is usually done, the result of every experimenter on every subject would have to be easily accessed in such a registry, and even its existence should be easily found by a researcher. This includes all studies where it was decided that the results were not worth writing up.
><From: je...@hnrc.tufts.edu (Jerry Dallal) ><No. I would be merely willing to put aside some of my distrust of combining ><results through meta-analysis if the meta-analysis were based on studies ><contained in a registry. Sorry if my post was unclear on that point.
I have no problems in combining studies. No Bayesian has such problems. BUT, what is done in typical meta-analysis can give quite erroreous results.
Once there is selection in what is published, including being placed in a registry, the results are almost certain to be biased. An example I hear about 25 years ago occurred in physics; a particular result found in one situation was extrapolated to another, where that had been sought. The theory gave a maximum of 9% for the effect, but the pooled published data gave 15%. The theory is correct, and there was nothing wrong with the data. -- Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399 Phone: (317)494-6054 hru...@stat.purdue.edu (Internet, bitnet) {purdue,pur-ee}!a.stat!hrubin(UUCP)
<Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399
<Once there is selection in what is published, including being placed <in a registry, the results are almost certain to be biased. An example <I hear about 25 years ago occurred in physics; a particular result found <in one situation was extrapolated to another, where that had been sought. <The theory gave a maximum of 9% for the effect, but the pooled published <data gave 15%. The theory is correct, and there was nothing wrong with <the data.
I did not intend to imply that there was any hope that meta-analysis of published studies would be unbiased.
I don't even have much hope that full access to all the raw data will help.
Meta-analyses of randomized studies are "more like" observational studies than randomized studies ... and a few years ago we made the "cute claim" that they were just "cost/benifit" analyses of whether to do a(nother) randomized study.
Keith O'Rourke The Toronto Hosp.
p.s. infinite cost covers the case of it being unethical to do a randomized study.
p.s2. I thought Meta-analysis would be an ideal stage for the introduction of Bayesian methods for clinical trials but a C.R. Roa argued in Statistical Science back in 88-89 "the prior would dominate the posterior in most cases (usually less than a dozen RCTs on most subjects)
|
