PP279: RESEARCH DESIGN AND DATA COLLECTION

FOR PUBLIC POLICY ANALYSIS

Professor Robert MacCoun

Goldman School of Public Policy, 2607 Hearst Ave.

642-7518, maccoun@ berkeley.edu

Fall 2006: Tuesdays and Thursdays, 2-3:30, 105 GSPP (the new bldg.)

(Office Hours: by appt.)

 

The online version of this syllabus is at http://socrates.berkeley.edu/~maccoun/PP279_f06.html 
Please see the online version for the most up-to-date version; I will announce any revisions in class. 

COURSE DESCRIPTION

Empirical arguments and counterarguments play a central role in policy debates, thus public policy analysis requires a sophisticated understanding of a variety of types and sources of data. Quantitative analysis courses teach you how to analyze data; this course will introduce you to strategies of data collection and principles for critically evaluating data collected by others. Topics include measurement reliability and validity, questionnaire design, sampling, experimental and quasi-experimental program evaluation designs, qualitative research methods, and the politics of data in public policy.

 

READINGS

Carmines, E. G., & Zeller, R. A. (1979). Reliability and Validity Assessment. Quantitative Applications in the Social Sciences #17. Newbury Park, CA: Sage Publications.

Converse, J., & Presser, S. (1986). Survey Questions. Quantitative Applications in the Social Sciences #63. Newbury Park, CA: Sage.

Kalton, G. (1983). Introduction to Survey Sampling. Quantitative Applications in the Social Sciences #35. Newbury Park, CA: Sage.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and Quasi-Experimental Designs For Generalized Causal Inference. Houghton Mifflin.  [If you are interested, my review of this book appears here: http://socrates.berkeley.edu/~maccoun/JPAM_2003_BookRev.pdf.]

 

ASSIGNMENTS (see due dates in Schedule at end of syllabus)

  • Two draft research proposals:  Each worth 40%. Like most research proposals, these will be developed in small groups of 2-3 people (size will depend on class enrollment). Teams will give in-class briefings on the second proposals at the end of the term.

1) Survey proposal (http://socrates.berkeley.edu/~maccoun/PP279_proposal1.html)

2) Program evaluation proposal/group briefing:  http://socrates.berkeley.edu/~maccoun/PP279_proposal2.html

READINGS - Schedule appears on last page

NOTE: The readings are NOT in a reader; they are online at http://socrates.berkeley.edu/~maccoun/PP279.html

The Philosophy of Science and the Politics of Data (First Look)

MacCoun, R. (1998). Biases in the interpretation and use of research results, Annual Review of Psychology, 49, 259-287. http://socrates.berkeley.edu/~maccoun/MacCoun_AnnualReview98.pdf

MacCoun, R. J. (2001). American distortion of Dutch drug statistics. Society, 38, 23-26.  [Note that there was a "distortion" introduced in typesetting: Second-to-last sentence should read "Accuracy won't invariably breed consensus," rather than "Accuracy will invariably breed consensus."]
http://socrates.berkeley.edu/~maccoun/Society2001_DistortionDutchDrugStats.pdf
 

Describing the World: Surveys and Other Measures

Asking Questions

Converse & Presser’s Sage booklet

Krosnick, Jon A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.50.1.537 


Schaeffer, Nora Cate, & Presser, Stanley (2003).  The science of asking questions.  Annual Review of Sociology Aug 2003, Vol. 29: 65-88.

http://socrates.berkeley.edu/~maccoun/PP279_SchaefferPresser2004.pdf

 

OPTIONAL READING The following article is somewhat difficult, but it makes an important theoretical point about attitude strength and also illustrates the multi-trait/multi-method approach discussed next week: 

Krosnick, J. et al., (1993).  Attitude strength: One construct or many related constructs? Journal of Personality & Social Psychology. 65(6) 1132-1151.

http://socrates.berkeley.edu/~maccoun/PP279_Krosnick.pdf

 

Reliability and Validity (basic psychometrics)

Rob’s memo on coefficient alpha: 

http://socrates.berkeley.edu/~maccoun/CoefAlpha.pdf

 

Rob’s memo on how low reliability weakens the ability to detect relationships (e.g., program effects)

http://socrates.berkeley.edu/~maccoun/PredictiveValidity.pdf

Carmines & Zeller’s Sage booklet

Neisser, U., et al. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101.

http://socrates.berkeley.edu/~maccoun/PP279_Neisser1.pdf

 

Neisser’s reply to critics of that article are at:  http://socrates.berkeley.edu/~maccoun/PP279_Neisser2.html

Flynn, James R., Searching for justice: The discovery of IQ gains over time. American Psychologist. 1999 Jan Vol 54(1) 5-20.

http://socrates.berkeley.edu/~maccoun/PP279_Flynn.pdf

 

Schmitt, Neal, Uses and abuses of coefficient alpha. Psychological Assessment. 1996 Dec Vol 8(4) 350-353.
http://socrates.berkeley.edu/~maccoun/PP279_Schmitt.pdf

 

OPTIONAL:  Kuncel, N. R., Hezlett, S. A., Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin. 127(1) 162-181.
http://socrates.berkeley.edu/~maccoun/PP279_GRE.pdf


OPTIONAL: Lubinski, D. (2000). Scientific and social significance of assessing individual differences: "Sinking shafts at a few critical points." Annual Review of Psychology, 51, 405-444.   

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.51.1.405

 

Survey Sampling

Kalton's Sage booklet.

Rob’s memo on computing sample sizes.  http://socrates.berkeley.edu/~maccoun/pp279_samplesize.pdf

Tourangeau, Roger (2004).  Survey research and societal change.  Annual Review of Psychology, 55, 775-801.

http://socrates.berkeley.edu/~maccoun/PP279_Tourangeau2004.pdf

 

Magnani, R., Sabin, K., Saidel, T., & Heckathorn, D. (2005). Review of sampling hard-to-reach and hidden populations for HIV surveillance.  AIDS 2005, 19, S67-S72.  http://socrates.berkeley.edu/~maccoun/PP279_Magnani.pdf

 

OPTIONAL BUT RECOMMENDED:  Birnbaum, Michael H. (2004).  Human research and data collection via the internet.  Annual Review of Psychology, 55, 803-832.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.55.090902.141601

 

Marshall, G.N., Burnam, M. A., Koegel, P., Sullivan, G., & Benjamin B. (1996).  Objective Life Circumstances and Life Satisfaction: Results from the Course of Homelessness Study.  Journal of Health and Social Behavior, Vol. 37, No. 1. (Mar., 1996), pp. 44-58.  [Note: This paper has a very interesting sampling strategy for a difficult to sample population; it also has a nice example of multiple-indicator measurement of latent constructs.  And two of the authors attended my wedding!]

http://socrates.berkeley.edu/~maccoun/PP279_Marshall.pdf

 

Inferring Cause and Effect: Experimental and Quasi-Experimental Design

 

Dealing with Threats to Internal Validity

 

Shadish, Cook, & Campbell text, Chapter 1, pp. 53-62 of Chapter 2, and Chapter 8.

 

OPTIONAL:  Sander Greenland and Hal Morgenstern (2001). Confounding in health research. Annual Review of Public Health, 22, 189-212.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.22.1.189

 

OPTIONAL:  A. G. Barnett, J. C. Van der Pols, & A. J. Dobson (2005).  Regression to the mean: What it is and how to deal with it.  Int. J. Epidemiology, 34, 215-220.  http://socrates.berkeley.edu/~maccoun/PP279_Barnett.pdf

 

Quasi-Experiments

Shadish, Cook, & Campbell text, Chapters 4-7

OPTIONAL: Mark W. Lipsey and David S. Cordray (2000). Evaluation Methods for Social Intervention.  Annual Review of Psychology, 51, 345-375. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.51.1.345

OPTIONAL: Joseph P. Newhouse and Mark McClellan (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health, 19, 17-34.   http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.19.1.17

 

Dealing with Threats to Statistical Conclusion Validity

Rosnow, R. L., and Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.  PP279_Rosnow.pdf


Cohen, Jacob  The earth is round (p < .05). American Psychologist. 1994 Dec Vol 49(12) 997-1003

PP279_Cohen1.pdf

 

Cohen, J. (1992b). A power primer. Psychological Bulletin, 112, 155-159.

PP279_Cohen2.pdf

 

OPTIONAL:  Lenth R. (2001).  Some practical guidelines for effective sample size determination.  The American Statistician, 55, 187-193.

http://socrates.berkeley.edu/~maccoun/PP279_Lenth.pdf  [NOTE: LENTH AND COHEN DISAGREE ABOUT SOME ISSUES; WE CAN DISCUSS THE POINTS OF DISAGREEMENT IN CLASS]

OPTIONAL: Christopher Winship and Stephen L. Morgan (1999).  The estimation of causal effects from observational data Annu. Rev. Sociol., 25, 659-706.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.25.1.659

OPTIONAL: Roderick J. Little and Donald B. Rubin (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health, 21, 121-145.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.21.1.121

Here are some useful web-based power calculators.  I recommend you verify your results using more than one calculator.  I also recommend that you create a table and solve for N under a variety of assumptions about alpha and effect size.

 

http://www.dssresearch.com/toolkit/spcalc/power.asp

 

http://home.clara.net/sisa/power.htm

 

http://www.stat.uiowa.edu/~rlenth/Power/index.html

 

http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/

 

http://statpages.org/

 

Dealing with Threats to External Validity

Shadish, Cook, & Campbell text, Chapters 13 and 14

 

Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.  PP279_Schmidt.pdf

 

Hunter, John E.; Schmidt, Frank L., Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, & Law. 1996 Jun Vol 2(2) 324-347 PP279_Hunter.pdf

OPTIONAL: R. Rosenthal and M. R. DiMatteo (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.52.1.59

 

 

Qualitative Methods

 

Shadish, W. R. (1995).  Philosophy of science and the quantitative-qualitative debates: Thirteen common errors.  Evaluation and Program Planning, 18, 63-75.

http://socrates.berkeley.edu/~maccoun/PP279_Shadish.pdf

 

Morgan, G., & Smircich, L. (1980).  The case for qualitative research. The Academy of Management Review, 5, 491-500.

http://socrates.berkeley.edu/~maccoun/PP279_Morgan.pdf

 

David L. Morgan (1996). Focus groups. Annual Review of Sociology, 22, 129-152.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.22.1.129

 

PP279 SCHEDULE OVERVIEW – FALL 2006

Week

Day

Date

Topic

 

1

Tues

Aug 29

Course overview

 

 

Thurs

Aug 31

Philosophy vs. politics

 

2

Tues

Sep 5

Asking questions

 

 

Thurs

Sep 7

Asking questions

 

3

Tues

Sep 12

Asking questions

 

 

Thurs

Sep 14

Intro to psychometrics

 

4

Tues

Sep 19

Measurement reliability

 

 

Thurs

Sep 21

Measurement validity

 

5

Tues

Sep 26

Survey sampling

 

 

Thurs

Sep 28

NO CLASS*

HW due Fri Sep 29, 5pm

6

Tues

Oct 3

Survey sampling

 

 

Thurs

Oct 5

Survey sampling

PR#1 preposal due

7

Tues

Oct 10

Special populations

 

 

Thurs

Oct 12

Threats to internal validity

 

8

Tues

Oct 17

Threats to internal validity

 

 

Thurs

Oct 19

Experimentation

 

9

Tues

Oct 24

Experimentation

 

 

Thurs

Oct 26

Quasi-experimentation

PR#1 due Fri Oct 27, 5pm

10

Tues

Oct 31

Quasi-experimentation

 

 

Thurs

Nov 2

NO CLASS*

 

11

Tues

Nov 7

Stat conclusion validity

 

 

Thurs

Nov 9

Stat conclusion validity

PR#2 preposal due

12

Tues

Nov 14

Threats to external validity

 

 

Thurs

Nov 16

Threats to external validity

 

13

Tues

Nov 21

Qualitative methods

 

 

Thurs

Nov 23

THANKSGIVING

 

14

Tues

Nov 28

Group briefings

 

 

Thurs

Nov 30

Group briefings

 

15

Tues

Dec 5

Group briefings

 

 

Thurs

Dec 7

Group briefings

 

FinalsWk

 

 

 

PR#2 due Thu Dec 14, 5pm

 

 

 

 

* We will schedule a special evening makeup session (pizza?) if needed.


Last revised on 5/3/07