PP279/MacCoun

PROGRAM EVALUATION PROPOSAL ASSIGNMENT

DUE DATES FOR THE PREPROPOSAL AND THE PROPOSAL ARE LISTED ON THE COURSE WEB PAGE.

The Preproposal

Prior to any serious work on the proposal, your team should submit a 1-page letter of intent--concisely describing the central policy issue you are addressing, and the specific research questions you plan to answer. (You needn’t explain your methodology here.) You can evaluate an existing program, a program that will soon exist, or a hypothetical program you’d like to exist. 

The Proposal

The proposal must be submitted as an email attachment (maccoun@berkeley.edu) in MS Word format. This will allow me to embed my comments directly in the file and send it back to all of you.

  1. EACH TEAM MEMBER'S EMAIL ADDRESS MUST BE ON THE CC: LINE OF THE EMAIL. 
  2. TO HELP ME TRACK THE FILES ON MY COMPUTER, PLEASE USE THE FOLLOWING FORMAT FOR NAMING THE FILE:  ‘279_F06_P1_NameNameName.DOC’ (where NameNameName lists your last names).
  3. Submit only a single file; if you have spreadsheets and appendices, merge them into a single Word file. 
  4. Make sure there are no team comments or revision marks left in the file.  I have repeatedly received files with students’ comments to each other plainly visible on my screen; needless to say, the contents of these comments have not cast the teams in the best possible light.

Your proposal should include the sections described below; very approximate space-and-a-half page guidelines appear in brackets. If you wrote the maximum number of pages for each section, your proposal would be roughly 30 pages plus bibliography (we’ll skip the vitae); at minimum, you’ll probably need at least 20 pages.  These pages are GUIDELINES; not strict rules.  Use your own judgment about what your topic requires.  When in doubt about anything, the fundamental deciding principle isn’t “what does Rob want” but rather “what would encourage a funder to decide it is worth risking a large sum of money on our organization?”

  1. Proposal abstract

[1 page]: Brief summary of issues, questions, and general strategy

  1. Policy context

[2-3 pages]: Explain the policy issues that motivate your proposal. Assume the reader is a non-expert who is fairly conversant in current policy issues. Briefly foreshadow the program you plan to evaluate.

  1. Review of relevant existing research

[4-6 pages]: What do and don’t we know about your research questions? Be critical in your evaluation of the existing literature; draw upon your vast and deep knowledge of experimental and quasi-experimental methodology obtained in this course, including threats to internal, external, construct, and statistical conclusion validity (e.g., power). Assume that while your reader may not be an expert on the policy context, he or she (more likely, they--sometimes 10-15 different reviewers) knows enough about methodology that you don’t have to ‘start from scratch’ in explaining your critique.

  1. Precise statement of research questions and objectives

[1-2 pages]

  1. More detailed description of program/intervention you plan to evaluate

[2-4 pages]. Please describe it in enough detail that the reader understands what it is you will be evaluating. (a) Who will be the treatment provider? (b) What exactly is the treatment/ intervention? (c) What is the target population from which you’ll draw your sample? Please be sensitive to ethical issues, and don’t propose anything that (in your estimate) wouldn’t get passed a human subjects protection committee.

  1. Design #1: Experimental design

[4-6 pages]. (a) Under optimistic conditions of full agency cooperation and minimal political controversy, describe the experimental design you propose to use. (Use or adapt the Campbell-Stanley notation, but you aren’t limited to their exact designs.) Be clear about your unit of analysis. (b) Describe your planned sample size per condition, and justify it. (c) Describe the different conditions and how you plan to determine who gets in which condition. (d) Briefly list your major dependent measures, but this time you don’t have to produce an actual questionnaire. Be sure to include manipulation checks among your measures. (e) I’d like to see at least a few sentences about how you plan to analyze the data, but I won’t hold you accountable for any methods you have been trained to use yet. (f) Assess the design with respect to threats to validity (all types of validity)--be specific with respect to your particular topic and setting.

  1. Design #2: Quasi-experimental design

[4-6 pages]: (a) Identify possible reasons why Design #1 might become infeasible. (b) Describe the quasi-experimental design you plan to use as a "fall-back" option. (c) Explain what threats to validity you anticipate as a result of this fall-back option, and evaluate their potential consequences for your policy conclusions. Mention any additional dependent measures you might use to strengthen your inferences about the design. (d) In explaining your choice of quasi-experimental design, feel free to mention other alternatives you rejected. (Use or adapt the Campbell-Stanley notation, but you aren’t limited to their exact designs.) (e) Describe your planned sample size per condition, and justify it. (f) Again, I’d like to see at least a few sentences about how you plan to analyze the data, but I won’t hold you accountable for any methods you have been trained to use yet.

  1. Concluding paragraph

These are often redundant and/or corny, but better than simply ending with purely technical material.

  1. Bibliography

[whatever pages you need; probably 2-4?]

Same rules as before: (1) It is fair game to email me questions about your proposal. (2) It is also fair game to discuss this (unlike the homework) with others. Obviously, my hope is that you’ll discuss it with your teammates! In addition, you can discuss it with members of other teams. But I encourage you to see how far you can get without consulting experts (e.g., professors, survey professionals).