1. What is the basic research question, or problem Try to state
it in just one sentence.

2. Is the purpose of the study explanatory, evaluative,
exploratory, or descriptive? Did the study have more than one purpose?

3. Was a theoretical framework presented? What was it? Did it
seem appropriate for the research question addressed? Can you think of
a different theoretical perspective that might have been addressed?

4. What prior literature was reviewed? Was it relevant to the
research problem? To the theoretical framework? Does the literature
review appear to be adequate? Are you aware of (or can you locate) any
important omitted studies?

5. How well did the study live up to the guidelines for science?
Do you need additional information in any areas to evaluate the study?
To replicate it?

6. Did the study seem consistent with current ethical standards?
Were any tradeoffs made between different ethical guidelines? Was an
appropriate balance struck between adherence to ethical standards and
use of the most rigorous scientific practices?

7. What were the major concepts in the research? How, and how
clearly, were they defined? Were some concepts treated as
unidimensional that you think might be best thought of as
multidimensional?

8. Were any hypotheses stated? Were these hypotheses justified
adequately in terms of the theoretical framework? In terms of prior
research?

9. What were the independent and dependent variables in the
hypothesis or hypotheses? Did these variables reflect the theoretical
concepts as intended? What direction of association was hypothesized?
Were any other variables identified as potentially important?

10. Did the instruments used, the measures of the variables, seem
valid and reliable? How did the authors attempt to establish this?
Could any more have been done in the study to establish measurement
validity?

11. What were the units of analysis? Were they appropriate for the
research question? If some groups were the units of analysis, were any
statements made at any point that are open to the ecological fallacy?
If individuals were the units of analysis, were any statements made at
any point that suggest reductionist reasoning?

12. Was the study design cross-sectional or longitudinal, or did it
use both types of data? If the design was longitudinal, what type of
longitudinal design was it? Could the longitudinal design have been
improved in any way, as by collecting panel data rather than trend data,
or by decreasing the dropout rate in a panel design? If cross-sectional
data were used, could the research question have been addressed more
adequately with longitudinal data?

13. Were any causal assertions made or implied in the hypotheses or
in subsequent discussions? What approach was used to demonstrate the
existence of causal effects? Were all four criteria for establishing
causal relations addressed? What, if any, variables were controlled in
the analysis to reduce the risk of spurious relationships? Should any
other variables have been measured and controlled? How satisfied are
you with the internal validity of the conclusions?

14. Was a sample or the entire population of elements used in the
study? What type of sample was selected? Was a probability sampling
method used? Did the authors think the sample was generally
representative of the population from which it was drawn? Do you? How
would you evaluate the likely generalizability of the findings to other
populations?

15. Was the response rate or participation rate reported? Does it
appear likely that those who did not respond or participate were
markedly different from those who did participate? Why or why not? Did
the author(s) adequately discuss this issue?

16. Was an experimental, survey, participant observation, or some
other research design used? How well was this design suited to the
research question posed and the specific hypotheses tested, if any? Why
do you suppose the author(s) chose this particular design? How was the
design modified in response to research constraints? How was it
modified in order to take advantage of research opportunities?

17. Was a historical comparative design used? Which type was it?
Were problems due to using historical and/or cross-national data
addressed?

18. Were multiple methods used? Were findings obtained with
different methods complementary?

19. Was any attention given to social context? To biological
processes? If so, what did this add? If not, would it have improved
the study? Explain?

20. Summarize the findings. How clearly were statistical and/or
qualitative data presented and discussed? Were the results
substantively important?

21. Did the author(s) adequately represent the findings in the
discussion and/or conclusions sections? Were conclusions well grounded
in the findings? Are any other interpretations possible?

22. Compare the study to others addressing the same research
question. Did the study yield additional insights? In what ways was
the study design more or less adequate than the design or previous
research?

23. What additional research questions and hypotheses are suggested
by the study's results? What light did the study shed on the
theoretical framework used? On social policy questions?


From Investigating the Social World, 2nd edition by Russell K. Schutt. This book also has an example article in which Schutt applies these 23 points to an article from American Sociological Review. It is well worth reading.