PP279: RESEARCH DESIGN AND DATA COLLECTION

FOR PUBLIC POLICY ANALYSIS

Professor Robert MacCoun

Goldman School of Public Policy, 2607 Hearst Ave.

642-7518, maccoun@ berkeley.edu

Fall 2005: Tuesdays and Thursdays, 12:30-2:00, 105 GSPP

(Office Hours: by appt.)

 

The online version of this syllabus is at http://socrates.berkeley.edu/~maccoun/PP279.html 
Please see the online version for the most up-to-date version; I will announce any revisions in class. 

COURSE DESCRIPTION

Empirical arguments and counterarguments play a central role in policy debates, thus public policy analysis requires a sophisticated understanding of a variety of types and sources of data. Quantitative analysis courses teach you how to analyze data; this course will introduce you to strategies of data collection and principles for critically evaluating data collected by others. Topics include measurement reliability and validity, questionnaire design, sampling, experimental and quasi-experimental program evaluation designs, qualitative research methods, and the politics of data in public policy.

 

READINGS

Carmines, E. G., & Zeller, R. A. (1979). Reliability and Validity Assessment. Quantitative Applications in the Social Sciences #17. Newbury Park, CA: Sage Publications.

Converse, J., & Presser, S. (1986). Survey Questions. Quantitative Applications in the Social Sciences #63. Newbury Park, CA: Sage.

Kalton, G. (1983). Introduction to Survey Sampling. Quantitative Applications in the Social Sciences #35. Newbury Park, CA: Sage.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and Quasi-Experimental Designs For Generalized Causal Inference. Houghton Mifflin.  [If you are interested, my review of this book appears here: http://socrates.berkeley.edu/~maccoun/JPAM_2003_BookRev.pdf.]

 

ASSIGNMENTS (see due dates in Schedule at end of syllabus)

  • Two draft research proposals:  Each worth 40%. Like most research proposals, these will be developed in small groups of 2-3 people (size will depend on class enrollment). Teams will give in-class briefings on the second proposals at the end of the term.

1) Survey proposal (http://socrates.berkeley.edu/~maccoun/PP279_proposal1.html)

2) Program evaluation proposal/group briefing:  http://socrates.berkeley.edu/~maccoun/PP279_proposal2.html

READINGS - Schedule appears on last page

NOTE: The readings are NOT in a reader; they are online at http://socrates.berkeley.edu/~maccoun/PP279.html

The Philosophy of Science and the Politics of Data (First Look)

" MacCoun, R. (1998). Biases in the interpretation and use of research results, Annual Review of Psychology, 49, 259-287. http://socrates.berkeley.edu/~maccoun/MacCoun_AnnualReview98.pdf

MacCoun, R. J. (2001). American distortion of Dutch drug statistics. Society, 38, 23-26.  [Note that there was a "distortion" introduced in typesetting: Second-to-last sentence should read "Accuracy won't invariably breed consensus," rather than "Accuracy will invariably breed consensus."]
http://socrates.berkeley.edu/~maccoun/Society2001_DistortionDutchDrugStats.pdf
 

Describing the World: Surveys and Other Measures

Asking Questions

Converse & Presser’s Sage booklet

Krosnick, Jon A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.50.1.537 


Schaeffer, Nora Cate, & Presser, Stanley (2003).  The science of asking questions.  Annual Review of Sociology Aug 2003, Vol. 29: 65-88.

http://socrates.berkeley.edu/~maccoun/SchaefferPresser2004.pdf

 

OPTIONAL READING The following article is somewhat difficult, but it makes an important theoretical point about attitude strength and also illustrates the multi-trait/multi-method approach discussed next week: 

Krosnick, J. et al., (1993).  Attitude strength: One construct or many related constructs? Journal of Personality & Social Psychology. 65(6) 1132-1151.

http://socrates.berkeley.edu/~maccoun/PP279_Krosnick.pdf

 

Reliability and Validity (basic psychometrics)

Rob’s memo on coefficient alpha: 

http://socrates.berkeley.edu/~maccoun/CoefAlpha.pdf

 

Rob’s memo on how low reliability weakens the ability to detect relationships (e.g., program effects)

http://socrates.berkeley.edu/~maccoun/PredictiveValidity.pdf

Carmines & Zeller’s Sage booklet

Neisser, U., et al. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101.

http://socrates.berkeley.edu/~maccoun/PP279_Neisser1.pdf

 

Neisser’s reply to critics of that article are at:  http://socrates.berkeley.edu/~maccoun/PP279_Neisser2.html

Flynn, James R., Searching for justice: The discovery of IQ gains over time. American Psychologist. 1999 Jan Vol 54(1) 5-20.

http://socrates.berkeley.edu/~maccoun/PP279_Flynn.pdf

 

OPTIONAL:  Kuncel, N. R., Hezlett, S. A., Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin. 127(1) 162-181.
http://socrates.berkeley.edu/~maccoun/PP279_GRE.pdf


Schmitt, Neal, Uses and abuses of coefficient alpha. Psychological Assessment. 1996 Dec Vol 8(4) 350-353.
http://socrates.berkeley.edu/~maccoun/PP279_Schmitt.pdf


OPTIONAL: Lubinski, D. (2000). Scientific and social significance of assessing individual differences: "Sinking shafts at a few critical points." Annual Review of Psychology, 51, 405-444.   

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.51.1.405

 

Survey Sampling

Kalton's Sage booklet.

Tourangeau, Roger (2004).  Survey research and societal change.  Annual Review of Psychology, 55, 775-801.

http://socrates.berkeley.edu/~maccoun/Tourangeau2004.pdf

 

OPTIONAL BUT RECOMMENDED:  Birnbaum, Michael H. (2004).  Human research and data collection via the internet.  Annual Review of Psychology, 55, 803-832.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.55.090902.141601

 

Marshall, G.N., Burnam, M. A., Koegel, P., Sullivan, G., & Benjamin B. (1996).  Objective Life Circumstances and Life Satisfaction: Results from the Course of Homelessness Study.  Journal of Health and Social Behavior, Vol. 37, No. 1. (Mar., 1996), pp. 44-58.  [Note: This paper has a very interesting sampling strategy for a difficult to sample population; it also has a nice example of multiple-indicator measurement of latent constructs.  And two of the authors attended my wedding!]

http://socrates.berkeley.edu/~maccoun/PP279_Marshall.pdf

 

Inferring Cause and Effect: Experimental and Quasi-Experimental Design

 

Dealing with Threats to Internal Validity

 

Shadish, Cook, & Campbell text, Chapter 1, pp. 53-62 of Chapter 2, and Chapter 8.

 

OPTIONAL:

Sander Greenland and Hal Morgenstern (2001). Confounding in health research. Annual Review of Public Health, 22, 189-212.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.22.1.189

 

 

Quasi-Experiments

Shadish, Cook, & Campbell text, Chapters 4-7

OPTIONAL: Mark W. Lipsey and David S. Cordray (2000). Evaluation Methods for Social Intervention.  Annual Review of Psychology, 51, 345-375. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.51.1.345

OPTIONAL: Joseph P. Newhouse and Mark McClellan (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health, 19, 17-34.   http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.19.1.17

 

Dealing with Threats to Statistical Conclusion Validity

Rosnow, R. L., and Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.  PP279_Rosnow.pdf


Cohen, Jacob  The earth is round (p < .05). American Psychologist. 1994 Dec Vol 49(12) 997-1003

PP279_Cohen1.pdf

 

Cohen, J. (1992b). A power primer. Psychological Bulletin, 112, 155-159.

PP279_Cohen2.pdf

OPTIONAL: Christopher Winship and Stephen L. Morgan (1999).  The estimation of causal effects from observational data Annu. Rev. Sociol., 25, 659-706.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.25.1.659

OPTIONAL: Roderick J. Little and Donald B. Rubin (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health, 21, 121-145.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.21.1.121

Dealing with Threats to External Validity

Shadish, Cook, & Campbell text, Chapters 13 and 14

 

Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.  PP279_Schmidt.pdf

 

Hunter, John E.; Schmidt, Frank L., Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, & Law. 1996 Jun Vol 2(2) 324-347 PP279_Hunter.pdf

OPTIONAL: R. Rosenthal and M. R. DiMatteo (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.52.1.59

 

 

Qualitative Methods

 

Shadish, W. R. (1995).  Philosophy of science and the quantitative-qualitative debates: Thirteen common errors.  Evaluation and Program Planning, 18, 63-75.

http://socrates.berkeley.edu/~maccoun/PP279_Shadish.pdf

 

Morgan, G., & Smircich, L. (1980).  The case for qualitative research. The Academy of Management Review, 5, 491-500.

http://socrates.berkeley.edu/~maccoun/PP279_Morgan.pdf

 

David L. Morgan (1996). Focus groups. Annual Review of Sociology, 22, 129-152.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.22.1.129

 

 

PP279 SCHEDULE OVERVIEW – FALL 2005

Week

Day

Date

Topic

 

1

Tues

Aug 30

Course overview

 

 

Thurs

Sep 1

Philosophy vs. politics

 

2

Tues

Sep 6

Asking questions

 

 

Thurs

Sep 8

Asking questions

 

3

Tues

Sep 13

Asking questions

 

 

Thurs

Sep 15

Intro to psychometrics

 

4

Tues

Sep 20

Measurement reliability

 

 

Thurs

Sep 22

Measurement validity

 

5

Tues

Sep 27

Survey sampling

 

 

Thurs

Sep 29

Survey sampling

 HW due Fri Sep 30, 5pm

6

Tues

Oct 4

Survey sampling

 

 

Thurs

Oct 6

Special populations

 

7

Tues

Oct 11

Threats to internal validity

 

 

Thurs

Oct 13

Threats to internal validity

 

8

Tues

Oct 18

Experimentation

 

 

Thurs

Oct 20

Experimentation

 

9

Tues

Oct 25

Quasi-experimentation

 

 

Thurs

Oct 27

Quasi-experimentation

PR#1 due Fri Oct 27, 5pm

10

Tues

Nov 1

Stat conclusion validity

 

 

Thurs

Nov 3

NO CLASS*

 

11

Tues

Nov 8

Stat conclusion validity

 

 

Thurs

Nov 10

Threats to external validity

 

12

Tues

Nov 15

Threats to external validity

 

 

Thurs

Nov 17

Qualitative methods

 

13

Tues

Nov 22

NO CLASS*

 

 

Thurs

Nov 24

THANKSGIVING

 

14

Tues

Dec 1

Group briefings

 

 

Thurs

Dec 3

Group briefings

 

15

Tues

Dec 6

Group briefings

 

 

Thurs

Dec 8

Group briefings

 

FinalsWk

 

 

 

PR#2 due Wed Dec 14, 5pm

 

 

 

 

* We will schedule a special evening makeup session (pizza?) if needed.


Last revised on 10/2705