HEA 625-01 (3 credits), Community Health Research and Evaluation

Department of Public Health Education

Course Syllabus, Fall 2005

 

Instructor: Vincent T Francisco, PhD

Department of Public Health Education

The University of North Carolina at Greensboro

437 HHP Building, P.O. Box 26170

Greensboro, NC 27402-6170

Voice: 336.334.5520

Fax: 336.334.3238

Email: Vincent_Francisco@uncg.edu

 

Meeting Time: Thursday, 6:00-8:50 PM

Meeting Place: Room 351, HHP Building

Office Hours: Monday and Wednesday, 230pm to 430pm

________________________________________________________________________

"If you are going to work with small groups and your aim is to change society, and you know that you need masses of people to accomplish that, you have to work with those people who can multiply what you do. It's a matter of having a concept of education that is yeasty, one that will multiply itself."

                                 --Myles Horton

                                   Founder of the Highlander Folk School

________________________________________________________________________

 

Prerequisites:             HEA 601, 602, and completion of, or concurrent enrollment in 603, or permission of instructor.

For whom planned:    This course is part of the required core course sequence for graduate students pursuing their MPH degree.

Course Description:

Catalog Description: Issues, problems and techniques involved in evaluation of community health education programs.

Expanded Description: The course establishes a framework, rational, and the basic concepts essential to planning, designing, and conducting an evaluation of health promotion programming. Course content will cover the essential components of program implementation, process, impact and outcome evaluations. Students will develop skills in the identification of data sources, the collection of primary data, and in the development of measurable goals, objectives, and performance measures. Students learn quantitative, qualitative, and mixed-method approaches. This course uses conceptual models and examples from domestic and international health promotion programs. Students critique evaluation designs and prepare their own evaluation design of a health promotion program. Course consists of lectures, case studies, group discussions, and poster presentations. Student evaluation is based on class participation during lectures, group discussions; written exercises; and final project.

Student Learning Outcomes:

At the end of this course, you should be able to:

1. Discuss the principles and theories needed to strategically plan, design, implement and evaluate health promotion programs.

2. Critically evaluate data sources used in needs assessments, process, outcome, and impact evaluations.

3. Recognize and translate the needs of communities, program staff, and funding agencies into useable planning and evaluation tools.

4. Define the planning and organizational steps needed to successfully plan and carry out needs assessments, process, impact and outcome evaluations.

 

Required Textbooks:

Chen, H. T. (2005). Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousand Oaks: Sage Publications.

Girden, E. R. (2001). Evaluating Research Articles from Start to Finish (2nd Edition). Thousand Oaks: Sage Publications.

Optional textbooks:

Coley SM & Scheinberg CA (2000). Proposal Writing, 2nd Edition. Thousand Oaks, CA: Sage Publications. ISBN:0-7619-1960-0. http://www.sagepub.com

 

Readings:

Additional course readings (except textbook materials) will be made available through UNCG’s library system, or posted on BlackBoard. Students are expected to have completed the required reading assignments before each class session.

 

 

COURSE REQUIREMENTS

Teaching Strategies:

This course will combine the lecture, class discussion, group process, and student presentation formats. The success of this course depends on how much you, your classmates, and I put into our work here together. This syllabus is the beginning of a learning contract we will share. The outline of topics to be covered and readings to be done - as well as the course objectives - lay out part of my commitment to you in what learning I will facilitate. The course requirements and the assignments listed below lay out part of your commitment to you and to your classmates to what learning you will undertake and facilitate.

 

Academic Honor Code:

Students are responsible for familiarizing themselves with UNCG’s policy on issues such as cheating, plagiarism, misuse of academic resources, falsification and facilitation of dishonest conduct. Procedures and penalties related to these and other violations of the Academic Honor Policy are found in the undergraduate bulletin as well as via the following web site. http://saf.dept.uncg.edu/studiscp/Honor.html Any violation of the Honor Policy may result in failure of the assignment and subsequently the entire course.

 

Students are required to sign the Academic Integrity Pledge given below on ALL major work submitted to an instructor. Please note that “a student's work need not be graded until he/she has signed the statement. Exceptions to the requirement of signing the statement may be specified by appropriate persons or offices, as, for example, on theses and dissertations. In the absence of such exceptions, students who do not sign the pledge may be assumed to have violated the Academic Integrity Policy. In signing the pledge, the student indicates his/her knowledge that the Academic Integrity Policy governs his/her academic activities at the University”.

 

               Academic Integrity Pledge:

 

               I HAVE ABIDED BY THE ACADEMIC INTEGRITY POLICY ON THIS

               ASSIGNMENT.

 

               Signature___________________________________________________

               Date__________________

 

Grading:

Students will be graded on 1) submission of three to five short written assignments (20%); 2) weekly journal discussions (10%); 3) a review paper (30%); 4) small group development of a program evaluation (30%): and 5) participation in class discussions, including written responses to questions and informal writings about the readings (10%).

Philosophy:

This course is designed to train students in the fundamentals of evaluating health promotion programs. It is recognized that many students in the class bring with them experiences from field and policy settings. These experiences are a valuable part of the course and it is intended that students will participate actively in classes by elaborating upon lectures, asking questions, and exchanging ideas with each other and the lecturers. Our learning process depends on your participation and commitment to learning. Before-class ways that you will demonstrate this commitment include reading and thinking about the readings and completing your assignments. In class, you will demonstrate this commitment by: listening, sharing your insights about the readings and topics at hand, asking questions, and participating in activities. To ensure that everyone has opportunities to contribute, I will call upon those who are not participating to help us out, but the primary responsibility for contributing falls on you.

 

Grading:

1) Critical review of evaluation case study or article (10%) Students will be assigned short written assignments that will be a critical review of an article or case studies. Students must attend and participate in the discussion session on the article/case study to receive all the potential credit for the assignment. Reviews should be no more than 3 pages (12 pt font, double spaced) and should clearly assess the ideas and conclusions stated in the article/case study. Directions for any component assignment of an evaluation plan will be given with the assignment. (SLOs # 1-4) DUE LAST DAY OF CLASS

2) Weekly Journal Reflections (15%) You will be responsible for presenting a summary and critique of selected assigned reading through BlackBoard. Each week, you must provide 1 original posting related to the readings of the week, and respond to 3 postings of your class peers. For this assignment you should be prepared to lead a discussion of the main points. You will be expected to focus the discussion relative to the topic and readings of that week. All students are required to read the articles and be prepared to contribute to the discussion (See class participation below). (SLOs # 1-4) DUE EVERY WEEK BEFORE CLASS STARTS

3) Review Paper (30%) The final product of the class will be a paper that will cover a topic within evaluation of your choosing. At various points throughout the semester, specific pieces of the paper will be due to the instructor for review on progress. These pieces include a question or topic of the paper, abstract or summary, outline of the paper, reference list, and the entire paper itself. We will cover expectations for each piece prior to the due date. The body of the paper will be 7 pages, with 1 inch margins, and double spaced (25 lines per page). Follow the current guidelines of the American Psychological Association, or the Council on Biology Editors, style guides for references and other relevant standards. (SLOs # 1-4) DUE LAST DAY OF CLASS

4) Group Design of a Program Evaluation Plan (25%) In small groups, we will be developing an evaluation of a program of your choice. This can be a fictional program, or a program of which you have access. Students will be assessed not only on their participation, but also on the quality of their work relative to the evaluation. You should expect to be working independently and as part of a group for various phases of this process. The goal here is to implement the concepts being presented in the classroom to a real evaluation plan. It is hoped that the final product of this group class activity is an evaluation report for the program being evaluated. (SLOs # 1-4) FINAL PRODUCT DUE LAST DAY OF CLASS (weekly in-class exercises due the Sunday following class)

5) Summary of Lessons Learned (5%) Additionally, due at the end of the semester is a 2 to 3-page summary of lessons learned through the semester. This is most commonly taken from the BlackBoard discussion forums, but can focus on the main points of what you learned and how you hope to be able to use the knowledge and skills acquired during the semester. The format is flexible, but single-spaced with 1-inch margins might be best. (SLOs # 1-4) DUE LAST DAY OF CLASS

6) Rating of individual contribution to group Grant Application (5%) DUE LAST DAY OF CLASS

7) Attendance (5%) Attendance will be recorded using a sign-in sheet every class. Given the participatory nature of the course and course activities, attendance is very important.

8) Class Participation (10%) Students are expected to contribute to classes based on their analysis of the readings and their own experience, when applicable. Students are expected to attend all classes; if you cannot attend, please notify the instructor and arrange to get copies of notes or handouts. Class participation includes online communications through BlackBoard, and other written communications. (SLOs # 1-4)

The course assignment point values follow:

 

(1) Critical reviews of evaluation case studies

10%

60 pts. (4 case studies X 15 pts. each)

(2) Weekly Journal/Online Discussion Forum

15%

90 pts. (15 weeks X 6 pts. each week)

(3) Review Paper

30%

180 pts.

(4) Group design of an evaluation plan

25%

150 pts.

(5) Summary Lessons Learned

5%

30 pts.

(6) Rating of individual contribution to group exercises

5%

30 pts.

(7) Class Attendance

5%

30 pts. (15 weeks X 2 pts. each week)

(8) Class Participation

5%

30 pts.

Total

100%

600 pts.

 

Final grades will be computed using standard percentages authorized by the University.

 

Policies Regarding Assignments:

1) All assignments should be submitted to the instructor prior to the beginning of the class on the day they are due. Assignments received after that time will be subject to an automatic 10% deduction in points for each day late, unless prior arrangements have been made with the course instructor. Requests to turn in assignments at times other than the due date must be submitted to the course instructor in writing prior to the due date. The instructor will notify the student in writing whether or not the request is approved.

The general standard followed by the instructor is fairness across students. If everyone cannot benefit from an exception, then it must accompany extraordinary circumstances. No extra credit is allowed.

2) Although students are encouraged to discuss course content and to share information about data sources, it is expected that students will work independently in preparing the written assignments, unless team or group projects have been assigned.

3) Only electronic versions emailed directly from the student to the instructor, or original hard copies of written assignments, will be accepted. The instructor prefers electronic communications over paper printouts.

Internet:

The course instructor set up a course homepage utilizing the Blackboard Course Management System for intra-class communications. Blackboard may be used to inform students of any changes in the assignments, room or scheduling changes, etc. It may also be used by students to ask questions of classmates and the instructor, or to further discussions outside of the classroom. Students must login at http://blackboard.uncg.edu/. Blackboard email cannot be changed to use an email address other than uncg.edu (except through email forwarding), so be sure to check your UNCG email account at least weekly.

If you have any problems doing either of these tasks contact the Student Computing Help Center in the library’s SuperLab or call 6-8324 (or off campus at 256-8324). It is your responsibility to make sure you set up your email accounts in order to receive important messages that will be sent through Blackboard.

 

Syllabus:

Learning is an organic process. Learning needs to be tailored and re-tailored to both meet the needs of the participants (that’s you, in this case) and to meet the objectives of the program (PHE). This syllabus may change. You will need to keep up with these changes, and you will receive information on any changes and/or a new syllabus (with a new date in the upper right corner) as necessary. You are expected to keep up with these changes both so that you complete the right assignments and so that you are informed of any changes. Please bring your syllabus to each session.

 

Week

Date

Topic

Detailed Class Content;

Readings and Assignments

1

Aug 18

Course Overview

Syllabus, ground rules

Student and Instructor expectations for the course

Foundations of effective (good) program evaluation; What you will learn in this course (and what you may not learn)

Proof “versus” Plausibility

Forms of error in reasoning, and how they can be avoided

Goals and purposes of evaluation (and research)

 

2

Aug 25

Evaluation Frameworks and Approaches

Applied research, Action research, Participatory action research, Community-based participatory research, Behavioral Community Technologies, Formative and Summative evaluation, Empowerment evaluation, Every other form and school of evaluation imaginable

Readings:

  • Chen—Part I: Introduction, pages 1-70.

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Due Sunday: List of team members, intervention topic, draft logic model

In Class:

  • Choose a team, develop a topic, and begin developing a theory of action and change

 

3

Sept 1

Reading and Understanding Scientific and Evaluation Reports

Definitions of science; Science and scientism; Internal and external validity; Experimental designs; Validity or Reliability? (or Both, or Neither); Reproducibility of results; Discovery

Readings:

  • Girden—Chapter 1: Introduction, pages 1-22.
  • Campbell and Stanley – Validity Tables (on Blackboard)
  • Handout – Theories of Causation and List of Threats to Validity (adapted from Cook, Campbell, & Peracchio, L. (1990). Chapter 9: Quasi-experimentation. In M. D. Dunnette & L. M. Hough, eds., Handbook of Industrial and Organizational Psychology, 2nd Edition. Palo Alto, CA: Consulting Psychologists Press, Inc.

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Topic/Question and brief description of review paper due at the end of the course.
  • Due Sunday from group: vision and mission, revised theory of action/change

In Class:

  • Group work: develop a vision and mission statement, continue working on theory of action/change

 

4

Sept 8

Evaluation Context and Describing the Program

Program components and elements; Implicit and explicit logic models; Context is everything--Why findings don’t always lead to solutions!

Readings:

  • Chen -- Part II, pages 71 through 128

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Due Sunday from group: program components and elements table, refined logic model

In Class:

  • Group work: program components and elements table, refined logic model

 

5

Sept 15

Stakeholder Identification and Evaluation Questions

Understanding Community Leadership, Funding Agents, and Evaluators: What are there interests? Developing evaluation questions.

Readings:

  • Chen – Part III, pages 129 through 178.
  • Community Tool Box (http://ctb.ku.edu/), Chapter 36, Section 3 – Understanding Community Leadership
  • Community Tool Box (http://ctb.ku.edu/), Chapter 36, Section 5—Developing an Evaluation Plan
  • Community Tool Box (http://ctb.ku.edu/), Chapter 1, Section 4--Evaluating Comprehensive Community Initiatives

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Reference list (draft) along with topic/question for review paper.
  • Due Sunday: products of group work from this week.

In Class:

  • Group work: develop list of stakeholders and their specific interests, translate these interests into evaluation questions.

 

6

Sept 22

Data Collection Methods I

Quantitative techniques: Direct observation, surveys, logs and self-reports, etc.

Readings: Girden – Chapters 4, 5, 6, 12

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Due Sunday: products of group work from this week.

In Class:

  • Group work: list evaluation questions, potential data collection methods and experimental designs

 

7

Sept 29

Data Collection Methods II

Qualitative techniques: semi-structured interviews, ethnography, historical analysis (historiography), case study, comparative case study, etc.

Readings:

  • Girden – Chapters 2 and 3
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 8 – Conducting Interviews with Key Participants

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Outline (draft) of review paper, along with updated reference list and topic/question.
  • Due Sunday: products of group work from this week.

In Class:

  • Group work: draft evaluation timeline

 

8

Oct 6

Process Evaluation

Evaluation Case study

Satisfaction surveys, implementation logs, diaries and records, “implementation checks”, social validation of goals and implementation

Readings:

  • Chen – Part IV, pages 179 through 230
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 2--Gathering Information: Monitoring Your Progress
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 3 – Rating Community Goals
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 4--Rating Member Satisfaction
  • Evaluation case study: TBA

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Critical review of case study.
  • Due Sunday: products of group work from this week.

In Class:

  • Group work: draft process evaluation tools

 

9

Oct 13

Outcome Evaluation

Evaluation Case study

Goal attainment scaling, self-reported behavior change surveys, direct observation of behavior

Readings:

  • Chen – Part IV, pages 231 through 266.
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 5 – Constituent Survey of Outcomes
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 6 – Reaching your goals: The Goal Attainment Report
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 7 – Behavioral Surveys
  • Evaluation case study: TBA

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Critical review of case study.
  • Due Sunday: products of group work from this week.

In Class:

  • Group work: draft outcome evaluation tools

 

10

Oct 20

Impact Evaluation

Evaluation Case study

Community-level indicators, social validation of effects

Readings:

  • Girden – Chapters 9, 10, 11
  • Community Tool Box (http://ctb.ku.edu/), Chapter 38, Section 9 – Gathering and using community level indicators
  • Evaluation case study: TBA

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Critical review of case study.
  • Due Sunday: products of group work from this week.

In Class:

  • Group work: draft impact evaluation tools

 

11

Oct 27

Answering the Evaluation Questions

Evaluation Case Study

Experimental designs, internal and external validity revisited; Visual Analysis of Data; Statistical Analysis of Data; Ongoing feedback and interaction effects; Generality of findings and generalizability of behavior

Readings:

  • TBA
  • Evaluation case study: TBA

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Critical review of evaluation case study.
  • Due Sunday: products of group work from this week.

In Class:

  • Group work: draft example data results from your program evaluation

 

12

Nov 3

SOPHE Annual Meeting

Guest lecture??

Readings: TBA

Assignments: TBA

 

13

Nov 10

Writing the Evaluation Report

 

Modes of communication: written report, group presentation; Working with evaluation clients

Readings:

  • Community Tool Box (http://ctb.ku.edu/), Chapter 39, Section 1--Using the Evaluation System to Answer Key Questions About the Initiative
  • Community Tool Box (http://ctb.ku.edu/), Chapter 39, Section 2--Providing Feedback to Improve the Initiative
  • Community Tool Box (http://ctb.ku.edu/), Chapter 39, Section 4--Communicating Information About the Initiative to Gain Support From Key Audiences
  • Evaluation case study: TBA

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.
  • Critical review of case study.
  • Due Sunday: products of group work from this week.

In Class

  • Group work: draft your evaluation presentation

 

14

Nov 17

Evaluation Tools and Resources

Evaluation Standards, Values and Ethics

 

Expectations of evaluators and evaluations; Choosing program evaluators, budgeting for program evaluation

American Evaluation Association, Email reflectors, SAGE Pubs, journals, web sites, popular evaluation workshops, etc.

Stuffelbeam et alia, AEA standards project

Readings:

  • Chen – Chapter 11, pages 267 through 272.
  • Community Tool Box (http://ctb.ku.edu/), Chapter 36, Section 4 – Choosing Evaluators

Assignments:

  • Blackboard – 1 original post, and 3 responses to other’s postings.

 

15

Nov 24

Thanksgiving

 

16

Dec 1

Putting it all together

Lessons learned

Future of Evaluation

Is evaluation good science? Systematic (component) and non-systematic (direct) replication—Where’s the beef? Can evaluation lead to generalizable knowledge? Does it matter? Future directions in evaluation.

Readings: TBA

Assignments:

  1. Review paper due from individuals.
  2. Package of evaluation plan, materials generated from group work during semester, and PPT evaluation report presentation due from groups.
  3. Course evaluations.

 

17

Dec 8

Final Exam Day

No exams.